A simulator of what happens when you use nonsense to decide whether you hire someone. It doesn't go well.
The multi-stage word-play in the headline came out poorly, but it’s worth saying, you can’t hire the best AND the brightest unless the phrase is redundant and the best are necessarily the brightest. Because you can only sort people by one attribute. And in populations as large as those that apply for jobs at the FBI, if they aren’t exactly the same, their sample size will turn up a counter-example.
Which means you have to choose what you care about when you hire people. If you filter on something unrelated to outcomes you care about, it comes at a cost. And if you decide to start firing people whose work is good because of a fitness test, obviously you’ll only be left with employees that have a solid 1.5 mile time, but the work will be done more and more poorly with every person you fire. Every single hiring or firing must be important, otherwise you’re strapping on concrete shoes.
So I’ve created a tool for the machismos at the FBI (and, of course, everyone else who hires professionals), so they can get a sense of how many murderers, terrorists, bank robbers, and arsonists are walking the streets free of criminal prosecution because they’ve decided looking good in a calendar is more important than their directive to enforce the laws of the United States.
Now that’s slightly unfair of me, because it’s not quite that simple – but then again, the FBI director says his goal is to have people look upon FBI agents with national pride, knowing with only a glance that “That’s an FBI special agent”. This bizarre goal makes it hard to avoid pointing out: this will negatively impact their undercover operations. “Identifiable as FBI at a glance” is not state-of-the-art undercover work.
And, of course, it’s terrifying how few major crimes in America are actually solved, so any improvement would be a tremendous boon.
So, I don’t recommend trying to simulate the many, many thousands of people who apply to the FBI. I have done it for you! If they eliminate the physical fitness test (based on numbers I grabbed while googling about who drops out at which stages, the number of people who apply, and the positions available) the FBI could noticeably improve it’s hiring. Using a very dumb model, I estimate they could up their clearance rate by about 4% (relative increase). Not bad! And that’s assuming (1) the rest of the hiring criteria are best-in-class, and (2) they don’t care at all about the PFT results except if it’s a failing score. These appear to not be true, so the gains could be even larger.
It’s also instructive to see that if you filter very rarely, the feedback isn’t strong. Run a couple simulations and you’ll see – for filters that only subtract a small number of people, the costs wash out in the statistical noise. On one hand, this means it’s financially justifiable to avoid hiring amoral people (as there are so few) but that other small groups (which, according to my measurements, include groups as large as 10% of the population, like homosexuals) are vulnerable to the same lack of cost. If we want a society free of discrimination, this is a fair reason why they may need specific protection. The lack of an obvious financial punishment for filtering out very small groups is a possible explanation for why redlining is an actual phenomenon instead of something that destroys companies.