Kahneman: The Scatter of Noise in Your Decisions
Why Daniel Kahneman’s Last Major Work Reveals That Your Organisation’s Decisions Are Less Consistent Than You Think
The Learning phase article on Kahneman introduced System 1 and System 2 as the cognitive architecture beneath organisational resistance to change. Bias was the headline: the systematic errors that make leaders overconfident, anchored to first impressions, and blind to what they do not know. That was a learning problem. This is the deciding problem that Kahneman spent his final years working on, and it is worse than bias: your organisation’s decisions are not just systematically wrong. They are randomly wrong. Different people, facing the same facts, on different days, reach different conclusions. And nobody notices.
Kahneman, with Olivier Sibony and Cass Sunstein, called this noise.
1. Noise Is Not Bias
Bias is a systematic deviation from the correct answer. If every underwriter in your insurance business overestimates risk for applicants from certain postcodes, that is bias: the error points in the same direction every time. Bias is visible in aggregate. You can measure it, name it, and design interventions to correct it.
Noise is the variability in judgments that should be identical. Two underwriters, given the same file, on the same day, produce different risk assessments. The same underwriter, given the same file on a different day, produces a different risk assessment. The error does not point in a consistent direction. It scatters. And because it scatters, it is invisible in averages. The organisation’s mean judgment may look reasonable while the individual judgments that compose it are wildly inconsistent.
Kahneman’s central claim in Noise (2021) is blunt: in most professional domains, noise is at least as large as bias, and usually larger. Studies of criminal sentencing, insurance underwriting, medical diagnosis, patent evaluation, and personnel assessment all show the same pattern. The variability between professionals making the same judgment is enormous, far exceeding what anyone involved believes. A noise audit, in which the same cases are independently assessed by multiple professionals, reliably shocks the organisation that conducts it. The professionals are shocked because they assumed their colleagues would agree. The leaders are shocked because they assumed the process guaranteed consistency. Both assumptions are wrong.
2. Why Noise Matters More Than Bias for the Deciding Phase
Bias has dominated the decision-quality conversation since Kahneman and Tversky’s original heuristics-and-biases programme in the 1970s. The organisational response has been debiasing: awareness training, structured decision processes, devil’s advocates, pre-mortems. These interventions address systematic error. They do nothing for noise.
The Deciding phase hypothesis is that decisions are design challenges. Noise reframes the design constraint. If your organisation’s decisions are noisy, then the quality of any individual decision is partly a function of who made it and when, not just what information was available or what process was followed. Two teams writing AI specifications for the same domain will produce different specifications, not because they have different information but because professional judgment varies. Two architecture review boards, assessing the same proposal on different days, will reach different conclusions. The variability is not a failure of the individuals. It is a structural feature of human judgment that the decision architecture has not been designed to manage.
Simon’s bounded rationality tells you the decision-maker cannot process everything. Beer’s requisite variety tells you the architecture must deliver information worth processing. Kahneman’s noise tells you that even when the information is right and the process is sound, the judgment applied to that information will vary in ways nobody has measured. This is a third constraint on decision quality, orthogonal to the other two, and most organisations have never even looked for it.
3. The Noise Audit: Measuring What You Have Never Measured
The most practical tool in Noise is the noise audit. The method is simple: present the same case to multiple professionals independently, and measure the variability in their judgments. The insurance company that asks its underwriters to price the same risk. The hiring committee that scores the same candidate independently before discussing. The specification review that has two teams assess the same AI-generated output against the same criteria.
Kahneman reports that when organisations first conduct a noise audit, the results are consistently disturbing. In one study, underwriters at a large insurance company estimated premiums for the same risks. The median difference between underwriters was 55%, more than five times what the company’s executives had predicted. The executives expected 10% variation. The reality was that two underwriters, trained in the same methods, working for the same company, applying the same guidelines, produced judgments that differed by more than half.
For AI transformation, the noise audit has a specific and urgent application. If your organisation relies on human judgment to evaluate AI-generated outputs, to assess specification quality, to approve AI-assisted decisions, then the consistency of that judgment is the ceiling on the quality of your AI process. An AI model that generates consistently good specifications is only as valuable as the human review process that evaluates them. If the review is noisy, the organisation cannot distinguish good AI output from bad, because the evaluation itself varies more than the thing being evaluated.
4. Decision Hygiene: The Structural Response
Kahneman’s response to noise is not cognitive. It is architectural. He calls it decision hygiene: a set of structural interventions that reduce noise without requiring anyone to become a better thinker.
The principles are straightforward. Structure the judgment: replace open-ended assessments with defined dimensions scored independently. Sequence the information: present facts before opinions, evidence before conclusions. Use independent assessment before discussion: have each decision-maker form a judgment before the group convenes. Aggregate judgments mechanically: average the independent assessments rather than letting the loudest voice prevail.
Each of these is a variety management intervention in Beer’s terms. Structured dimensions attenuate the variety of possible judgments to a manageable set. Independent assessment amplifies the variety of perspectives before the group attenuates them through discussion. Mechanical aggregation prevents the social dynamics of the meeting room from destroying the information contained in disagreement.
The connection to Evans is direct. Evans’s ubiquitous language is a noise-reduction mechanism. When two teams use the same term to mean different things, the variability in their specifications is partly linguistic noise: the judgment differs because the description differs, not because the domain differs. Establishing a shared vocabulary within a bounded context does not just improve precision. It reduces the noise that imprecision introduces into every downstream decision.
The connection to Argyris is equally direct. Argyris showed that defensive routines suppress disagreement. Kahneman shows that premature convergence, the group rushing to consensus before individual judgments have been formed, destroys the information contained in the disagreement. The remedy is the same: create conditions in which independent judgment can be expressed before social pressure compresses it. But where Argyris frames this as a psychological challenge (overcoming defensiveness), Kahneman frames it as an architectural one (sequencing the process so that convergence happens after, not before, independent assessment). Both are right. The architecture enables what the psychology permits.
5. When to Trust Intuition: The Kahneman-Klein Resolution
The Learning phase article presented Kahneman’s work as a catalogue of cognitive failures. The Deciding phase requires a more nuanced position, because the organisation that distrusts all intuition will be paralysed, and the organisation that trusts all intuition will be deluded. The question is not whether to trust judgment but when.
Kahneman and Gary Klein, whose work on expert intuition reaches the opposite conclusion from the heuristics-and-biases programme, spent years in adversarial collaboration trying to reconcile their positions. The result, published in 2009, is the most useful framework in the decision science literature. They agreed: intuition is trustworthy when two conditions are met. First, the environment must be sufficiently regular that patterns exist to be learned. Second, the decision-maker must have had prolonged practice with valid feedback, meaning feedback that is prompt, clear, and connected to the decision.
A chess master’s intuition is trustworthy because chess is regular and feedback is immediate. A firefighter’s intuition is trustworthy because fire behaviour, while dangerous, follows patterns, and feedback is visceral and fast. An executive’s intuition about AI strategy is not trustworthy, because the environment is novel (no regularities to learn from), the feedback is delayed (outcomes take months to materialise), and the feedback is ambiguous (it is never clear whether the outcome was caused by the decision or by other factors).
This maps onto the Deciding phase architecture. Domains where Evans’s knowledge crunching has produced a mature, validated model are high-validity environments: the domain expert’s intuition about what the specification should say is trustworthy, because they have years of patterned experience with clear feedback. Domains where the model is new, untested, or contested are low-validity environments: judgment should be structured, aggregated, and tested rather than trusted. Beer’s System 3* (the audit channel) is the architectural mechanism for checking whether the domain has enough regularity to justify intuitive judgment. Without the audit, the organisation cannot know whether it is in a high-validity environment or merely believes it is.
For AI, this distinction is critical. The experienced domain expert who says “that AI-generated specification is wrong” may be exercising valid pattern recognition developed over years. The executive who says “AI will transform our business within two years” is almost certainly exercising System 1 pattern-matching in an environment too novel to support it. The organisation needs both kinds of judgment. The decision architecture must distinguish between them.
6. Kahneman’s Limits
Kahneman must be read with the replication crisis visible. Several findings from the original heuristics-and-biases programme, particularly priming effects and ego depletion, have not replicated. Kahneman himself acknowledged this with unusual candour for a Nobel laureate. The core findings on noise, prospect theory, and the conditions for expert intuition remain robust, but the broader programme is less secure than Thinking, Fast and Slow suggests.
The deeper limitation is that Kahneman’s framework is individual. It explains how single decision-makers err. It says less about how organisations amplify or dampen those errors through their structures, cultures, and power dynamics. Beer provides the structural account. Bourdieu explains why the noise audit will be resisted: the variability in professional judgment is not a bug the organisation wants to fix but a feature that protects individual autonomy and status. Standardisation reduces noise, but it also reduces the professional’s sense that their judgment matters. The tension between decision quality and professional identity is real, and Kahneman’s architectural solutions must navigate it.
(An Organisational Prompt is something you can do now....)
Run a noise audit on one decision.
Pick a decision your organisation makes repeatedly using professional judgment. Perhaps it is the assessment of AI-generated code quality. Perhaps it is the prioritisation of features in a product backlog. Perhaps it is the evaluation of vendor proposals. Take 1 recent case and have it independently reassessed by a different group of equally qualified professionals, without access to the original judgments. Compare the two sets of assessments. The gap between them is the noise in your decision process. You have never measured it. It is almost certainly larger than you expect. And until you measure it, every intervention you design to improve decision quality is optimising a signal you cannot distinguish from the noise surrounding it.
Further Reading
Daniel Kahneman, Olivier Sibony, and Cass Sunstein: Noise: A Flaw in Human Judgment - Kahneman’s final major work. The distinction between bias and noise, the noise audit, and the case for decision hygiene. More immediately actionable than Thinking, Fast and Slow.
Daniel Kahneman: Thinking, Fast and Slow - The popular synthesis of the heuristics-and-biases programme. System 1 and System 2, prospect theory, and WYSIATI. Read it for the cognitive foundations; read Noise for the organisational implications.
Daniel Kahneman and Gary Klein: Conditions for Intuitive Expertise: A Failure to Disagree - The adversarial collaboration that reconciled heuristics-and-biases with naturalistic decision-making. The two conditions for trustworthy intuition: environmental regularity and prolonged practice with valid feedback. The single most useful paper for anyone who needs to know when to trust expert judgment and when to structure the decision instead.
Olivier Sibony: You’re About to Make a Terrible Mistake! - Sibony’s practitioner-oriented treatment of decision quality in organisations. More accessible than Kahneman, with worked examples from business strategy.
I write about the industry and its approach in general. None of the opinions or examples in my articles necessarily relate to present or past employers. I draw on conversations with many practitioners and all views are my own.