My real question is much more general: **How do you evaluate hires for roles you do not know how to perform?** There are a couple hard versions of this problem: - It's 2012 and you want to start an illegal taxi company. You would prefer not to end up in jail. You'll probably want to hire an expert and get their opinion on whether breaking this law will make you a felon or a billionaire. Choose your expert wisely! - You're willing [to spend $60m](https://web.archive.org/web/20140203005049/http://www.forbes.com/sites/scottmendelson/2014/01/28/the-lego-movie-can-revive-warner-bros-animation/) on making [the next hit movie](https://en.wikipedia.org/wiki/The_Lego_Movie) but it will only be worth it if the movie ends up wildly popular. - You are launching a new protocol / smart contract to the world and entrusting it with large amounts of money. If a vulnerability is found it is likely that [large amounts of money will be stolen](https://www.rekt.news/leaderboard/). The team you hire should be better at writing secure software than attackers are at exploiting vulnerable software, and you only get one chance. How do you find the right experts? --- So far I've found a number of different solutions: - **Word of mouth**. For problems that many people have tried to answer it is often sufficient to ask your friends for recommendations. This works because they have also grappled with this question! Every tax season I see a lot of this happen. - **Certification**. If you have a situation you know is simple then it's probably sufficient to hire anyone who is a member of the local bar. This is, indirectly, a strategy we use when deciding where to eat out. Health codes in most cities are so stringent that you don't need any knowledge of how to safely prepare food, you can trust that only safe food is being served. - **Social proof**. Someone at a pharmacy stuck a needle into my arm and I'm now much more willing to be in dense social situations. I have no tangible proof that I'm now immunized! However, I know that enough people have monitored and are monitoring the process that if it turned out the vaccination I received didn't work I would definitely find out. - **Second opinions**. If you have the opportunity to ask your question to multiple professionals who you can be pretty sure aren't colluding you can then compare their answers. Every time you're given the same answer increases your confidence in that answer. This works for problems which require little human judgement. e.g. If some belt in your engine has snapped that's the answer you're always going to get. If your question is about some novel and complicated legal strategy or tax loophole then even before you ask for the second opinion you can probably expect to get differing answers. - **Proof-of-success**. If your personal trainer looks strong there's a decent chance they know how to make you look strong. The tutor of a famous grandmaster is likely a good tutor. The auditor of a smart contract which lost a lot of money to a hack is likely not a good auditor. This strategy works when success is legible. - **Guess-and-check**. (sometimes called [Babble-and-Prune](https://www.lesswrong.com/posts/i42Dfoh4HtsCAfXxL/babble)). If the cost to failure is low /and/ success is legible then you can hire multiple people and try them all. This roughly describes the strategy of [r-selection](https://en.wikipedia.org/wiki/R/K_selection_theory#r-selection), or building a sales team or [troll farm](https://www.buzzfeednews.com/article/janelytvynenko/job-ads-for-russian-troll-factory) or [pod shop](https://capitalgains.thediff.co/p/multimanagerpodhedge-fund-101). - **Tests**. If you happen to have a similar problem with a known solution you can evaluate candidates by giving them that problem and verifying that they give you the correct answer. This strategy only works when you already have a solved problem. - **Peer-review**. e.g. the NSF has this problem when deciding who to give money to. Who will do useful research? They've decided that the best way to answer this question is to ask [other researchers](https://www.nsf.gov/bfa/dias/policy/merit_review/reviewer.jsp#1). This is the heuristic you use when you trust an article in Nature over a pre-print on biorxiv. The Nature article isn't necessarily correct but a few experts have looked at it and decided it's not obviously wrong. There are also more abstract strategies. I've never heard of this strategy being used to hire a professional but something like it might work: - Software Engineers occasionally struggle with [the problem of knowing whether some solution is correct](https://en.wikipedia.org/wiki/Test_oracle) and have found a solution, [property-based testing](https://www.hillelwayne.com/post/property-testing-complex-inputs/). Say you've written a novel and you're evaluating translators. Have your candidate translator translate just one passage. Then, find yourself some native speakers and give them the passage and ask some questions. "How did Mary feel about the proposal?" "What kind of person does Mark sound like?" if details you think important made it through then the translator did well! Are there more strategies? See also: Lesswrong has had [some thoughts](https://www.lesswrong.com/posts/YABJKJ3v97k9sbxwg/what-money-cannot-buy) on [this problem](https://www.lesswrong.com/posts/wEebEiPpEwjYvnyqq/when-money-is-abundant-knowledge-is-the-real-wealth).