Wicked Learning Environments

She's hired dozens of senior leaders over twelve years. Ten minutes into a conversation, she can tell whether someone has "it." The last three VP hires all performed well. She trusts her read. And she has almost no evidence that her read is any good.

In Thinking, Fast and Slow, Kahneman borrows a distinction from a psychologist named Robin Hogarth. Some learning environments are kind. Chess, tennis, firefighting. You make a move and the world responds quickly and clearly enough to teach you something. Others are wicked. The feedback arrives months later, shaped by dozens of variables you didn't control, and filtered through your own memory of what you predicted.

Hogarth's example was a doctor in the early 1900s who grew more confident in his treatments over a long career. He prescribed treatments, patients recovered, and he concluded his methods worked. His patients had self-limiting diseases. They recovered despite his treatments, not because of them. Confidence grew. Accuracy didn't.

Hiring is one of the wickedest learning environments in professional life. You interview someone in March, they start in May, they hit their stride (or don't) by November, and meaningful performance data trickles in over the following year. By the time the signal arrives, you've interviewed forty more candidates and can barely remember what you predicted about this one. And that's the optimistic version. Plenty of hiring managers never see structured performance data on their hires at all. Some roles generate faster feedback (quota-carrying sales reps give you ramp-to-quota data within months), but for most positions that land on a recruiter's desk, especially senior and leadership hires, the signal arrives late and tangled.

The problem gets worse when you consider the people you never hired. A hiring manager who trusts her gut on leadership potential will hire candidates who match a particular profile. Confident, articulate, visionary in conversation. Some of those hires succeed. The manager remembers the successes, credits the profile, and doubles down. What she never encounters is the counterfactual, the candidates she rejected who would have been excellent. Among those she hired, some succeeded because of a strong team, favorable timing, or a mentor, not because of whatever she detected in the interview. The research on this is consistent. Hiring managers with ten years of experience are no more accurate in their predictions than those with two. The extra eight years produced confidence, not calibration.

Culture fit hiring runs on the same broken loop. A team hires for alignment, sees low turnover, and credits the approach. But low turnover in a stable market tells you almost nothing about whether the screening improved outcomes. The team is the physician, seeing recovery and crediting the treatment when the outcome would have happened anyway.

None of this is unfixable. But fixing it requires building feedback loops that don't currently exist. The simplest version is to record a specific prediction at the point of hire. A sentence or two in the ATS, written before the candidate starts. "I expect this person to hit full productivity by month four and struggle with getting buy-in across teams." Twelve months later, pull those predictions and compare them to what actually happened. Most teams skip this because it feels like overhead. Without a written record of what you predicted, your memory will quietly revise the prediction to match the outcome.

Every six months, pull outcomes for the cohort hired 12 to 18 months ago. Show each interviewer their calibration data. When they said "strong hire," what percentage actually were? This kind of outcome tracking, done consistently, is one of the few things that actually improves prediction accuracy. The first round of data is usually humbling enough to change how debriefs go.

And before the team agrees on a hire, ask one question. "What would have to be true about this candidate for them to fail in this role?" That forces the group to think about failure instead of matching the candidate to a familiar profile. It pulls out the disagreements that politeness keeps buried.

That hiring manager who can tell within ten minutes whether someone has 'it' has been in a wicked environment for twelve years. That's exactly long enough to feel certain about a pattern that was never there.

The Recruiting Lattice takes mental models from fields like behavioral science, sociology, and decision theory and turns them into practical tools for talent acquisition.