Structured Doesn't Mean Fair

For years the American SAT tested analogies, and one of the famous ones asked students to complete the pair RUNNER is to MARATHON as OARSMAN is to what. The intended answer was regatta. Every student in the country saw the identical item, scored on the identical scale by the same machine. And the question still sorted them by who had spent summers around boats.

The test was structured in every sense the word usually carries. Same item, same scoring, no human judgment to play favourites. What it could not standardize was the content of the question. Regatta measures who grew up around the water, not who can reason.

We have built the same thing in hiring, and we are proud of it. Somewhere in the last decade structured interviewing went from a niche practice to the default answer for fairness. A team notices its debriefs are chaos, three interviewers reaching three different verdicts on one candidate for reasons none of them can quite defend. So they standardize. Same questions for everyone, a rubric with anchored levels, a calibration session at the end. The chaos drops. The scores align. And the team reports, honestly, that they made the process fairer.

They are right that they removed something. They removed the part of bias that comes from interviewers disagreeing. Whether that is the same as fairness is the question we stopped asking, because the answer felt obvious.

You can see the two come apart in a search most of us run. A Director of Engineering, internal panel, a structured loop with anchored scoring on things like strategic clarity and stakeholder influence. Two finalists. One leads with the conclusion, names three drivers, takes the panel through it in clean layers. The other has run a forty-person engineering org through a platform migration and answers in the specifics of it, the vendor that slipped, the staff engineer who quit halfway, the quarter they lost and how they got it back. Across three interviewers the first candidate scores a level higher on strategic clarity, consistently, because the top anchor on the rubric describes the shape of a consultant's answer and not an operator's. Nobody on the panel decided to reward the performance. The scorecard did it for them, the same way, every time.

Sociologist Lauren Rivera found the same thing when she studied how elite law firms, banks and consulting houses hire. Many ran genuinely structured case interviews, same prompt, same criteria, scored against a defined standard, and the structure still advantaged candidates from high-status backgrounds, because a top score required mastering what she called elaborate insider rituals, codes and styles. The format was identical for everyone. The content of a strong answer assumed exposure you bought through prep courses or borrowed from a friend already on the inside.

This is the structured question bias paradox. Standardizing the form of an evaluation removes inconsistency between evaluators.

Both halves are well supported. Structured interviews predict performance better than unstructured ones and narrow the gaps between demographic groups, a result that industrial psychologists have replicated for thirty years. Structure works, and nobody serious is telling you to go back to the gut-feel interview. The other half is just as solid. A standardized instrument reproduces, faithfully, whatever bias is built into its content.

So there are two different biases in any interview, and we have been using one word for both. One is the bias between interviewers, the noise of different people weighting things differently. The other is the bias inside the answer key, the question of what the process treats as a good answer in the first place. Structure was built for the first kind and does nothing about the second.

When the right answer encodes access, a particular reasoning ritual, a narrative shape, a vocabulary you only pick up on the inside, structure does not soften that filter. It applies it to every candidate, the same way, every time. A biased human interviewer is at least unreliable. He is harsh on Monday and generous on Friday, he falls for one candidate's story and forgets to hold the next one to it, and in all that noise a few people who do not match his template slip through anyway. A biased rubric never has an off day. What structure gives you, consistency, is what makes a content filter airtight.

Once a process is called structured, it gets called objective, and once it is objective nobody audits what it rewards. Leadership announces that the new interview removed bias. The announcement is sincere, and it is also a door closing, because the claim shields the process from the one examination that would find the bias still in it. Content audit never gets scheduled. The filter keeps running, now formally approved. Nobody in the room is favouring anyone. The scores are honest, but the rubric is the problem.

The fix starts with a question you can run on your own scorecard this afternoon. Take any single criterion and ask whether a candidate who can do the job could score low on it because they never learned the code. Could a strong analyst fail your case because the format is full of consulting idiom her real work will never touch. If the honest answer is yes, that line is measuring access, and you can rewrite it around the reasoning you care about and accept any route that arrives there.

For a role whose output you can sample directly, the deeper move is to stop testing rituals and start testing the work. A timed verbal case stuffed with framework names is a stylized performance. A take-home built from real, anonymised data the role handles is the job itself, and the job has no insider dialect to be fluent in. Where the task is a direct sample of the work, the gap between performing the format and doing the job closes, and that gap is where content bias was hiding.

Name the limit honestly, because for most of us it is the whole job. You cannot hand a take-home to a VP of a business line or a Director of Engineering. There is no clean sample of running a function, and the more senior the role, the more the interview is the only instrument you have. At that level no better test is coming to rescue you. The fix is the audit itself, reading your own rubric for the places where it rewards the shape of an answer over its substance.

Once a process is called objective, nobody questions it. Schedule the rubric review in the same quarter you introduce the structure, while questioning it still feels normal. A year in, nobody will want to look at it again.

Structure bought us consistency. Fairness is a separate claim, and it rests on checking what we reward against what the job needs.

There is a real case where structured does mean fair. When the task has a right answer the job itself can verify, the whole problem dissolves. Translation is accurate or it is not. A financial model balances or it breaks. There the content of the answer key is the work, with no cultural dialect sitting between the candidate and the score, and a structured version of that test is both more consistent and more fair. The paradox only exists when the test and the job are different things. When they are the same, it disappears.

None of this argues against structure. The pull, once you see the content problem, is to throw the rubric out and trust your read of the room again. That brings back the larger bias structure removed, the one where every interviewer ran a private answer key nobody could inspect. Structure is necessary. It just is not the whole of fairness, and the comfort of the word has let us treat it as if it were.

Which brings me back to the oarsman and the regatta. That item could not tell a student who could reason from a student who simply knew about boats. It gave both the same score and called that fair. Your scorecard is making the same call right now, on every candidate, with perfect reliability. The only question worth asking is whether the box you are scoring rewards the rowing, or just the knowing about it.

The Recruiting Lattice takes mental models from fields like behavioral science, sociology, and decision theory and turns them into practical tools for talent acquisition.