HEC Academic Makes Noise with Study on Flaw in Human Judgment

HEC Professor of Strategy (Education Track) Olivier Sibony joined forces with Nobel-laureate Daniel Kahneman and legal scholar Cass Sunstein to publish Noise, a monumental work on human error. The trio thus reinvents a term for this variability of judgment which pervades decision-making at all levels and in all fields. In an exclusive interview, Sibony shares insights into the groundbreaking work by three authors that one major daily dubbed “the boy band of intellectual provocateurs” (sic).

Listen to our special Knowledge@HEC podcast with Olivier Sibony:

"Noise is as important a source of error as bias. It doesn’t have an obvious cause and humans aren’t naturally equipped to deal with it. Yet, it has huge societal, legal and philosophical ramifications on our judicial, medical, business and educational systems, to name but a few.” From his Paris home base, Olivier Sibony calmly describes the essence of a major work written by three hands, which has taken the literary world by storm within days of its May publication. The 454-page book has shot into bestseller lists devised by the New York Times, Washington Post, LA Times and the UK’s Sunday Times alone.

“Every academic, policymaker, leader and consultant ought to read this book,” suggests the Washington Post. “(A) tour de force of scholarship and clear writing,” enthuses the New York Times. “'Noise’ may be the most important book I've read in more than a decade,” writes the author of bestseller, “Grit”, Angela Duckworth, before adding: “A genuinely new idea so exceedingly important you will immediately put it into practice. A masterpiece.'”  Meanwhile, the NYT journalist underlines its importance to a world currently living in a permanent crisis mode: “We are living in a moment of rampant polarization and distrust in the fundamental institutions that underpin civil society. Eradicating the noise that leads to random, unfair decisions will help us regain trust in one another.”

Defining Noise

The authors’ very concept of “noise” is far removed from the original Old French definition of a “din, disturbance, uproar, brawl”. Au contraire, the writers describe it as a quasi-invisible stalker: “In public conversations about human error and in organizations all over the world, noise is rarely recognized,” they note. “Bias is the star of the show. Noise is a bit player, usually offstage.”  

From the outset, the authors describe the difference between the two: bias defines systemic deviations, it is consistent, it is inherently causal and it is easily definable. While noise is what the writers call a “random scatter”. Sibony elaborates for us: “People make different judgments, and different errors. Part of the errors are shared: that’s bias. But there is variability in their errors, errors that are not shared. That’s what we call noise. It impacts all professional judgments where we expect consistency – judges, doctors, teachers, insurance underwriters, and so on. In an organization, people assume they are making decisions in the same way; and we presume that errors, if any, consist of bias. But our research shows that there is also a worrying amount of noise, which causes costly mistakes and injustice.”

Evaluating Noise in Risk

Sibony, Kahneman and Sunstein pack a tremendous number of examples in their book. Ironically, when the HEC academic and Princeton University Psychology professor began researching five years ago, they were unsure it would merit more than a paper: “We all tend to underestimate noise because we focus too much on bias.”, notes Sibony. However, as they built up evidence of noise in fields as diverse as college admissions, business decision-making, hiring, child custody, justice, fingerprint analysis or economic forecasts, they became aware of the need for a legal scholar on government and invited Cass Sunstein to join them for a book project three years ago.

To better understand the noise phenomenon, Sibony picks up on insurance risk evaluation: “Underwriters make judgments when they assess risks, so we measured average differences between two expert underwriters looking at the same risk. The executives of the insurance company thought that the median difference between two experts would be around 10%. They could live with that. But we found the figure to be closer to 55%! That’s five times more! In other words, noise is several times greater than people expect or than is deemed tolerable in an organization. And we find this time and again.”

(© Nicolas Reitzaum)

Practical Tools to Fight Noise

The audits done on criminal court cases threw up similar disparities: “For cases described in a very schematic and simplified way, leaving little space for variability, we found there to be 3.5 years of mean difference between sentences handed down by different judges. Again, that’s huge.”

For the three researchers, this was not a question of freedom of appreciation, but a systemic issue: “People are simply not aware of noise and how big the problem is. So we’ve devised noise audits which measure variability.” The three authors publish methods to help leaders conduct noise audits designed to improve predictions and group decisions. They also include a checklist of questions that help avoid errors. These include the following: “Were alternatives fully considered, and was evidence that would support them actively sought?” “Were uncomfortable data or opinions suppressed or neglected?” “Did the judgment rely heavily on anecdotes, stories, or analogies? Did the data confirm them?”

“We wanted to give the reader practical tools to deal with noise in organizations,” pursues Sibony, warming to the subject. “This led us to what we’ve called ‘decision hygiene’. A term which is a little off-putting, but this is intentional, as it sends a clear message: we cannot identify every source of noise, just like we cannot see every germ on our hands, but we are going to fight it and we’re going to win. This is a battle which needs to be fought each day, like a battle for hygiene in a hospital where you must wash your hands. Noise reduction, like hygiene, is a preventative measure - we can’t know what they’ve prevented, but they’re highly effective.”

Decision Hygiene and COVID-19

The three authors have spent the best part of half a decade sifting through a huge number of case studies to conjure effective antidotes to noise. They believe their application will bring down cost, instigate fairer and more humane decision-making and improve confidence in our judicial, political, educational, financial and health systems. As the co-CEO of Newsguard, Steven Brill, writes, “Noise” is a “welcome handbook for making life’s lottery a lot more coherent”. It is published at a moment where the COVID-19 pandemic has shaken the world’s convictions to its foundations, thus exacerbating this feeling of living in a lottery: “COVID-19 brings noise into a harsher light,” pursues the author of “You’re About to Make a Terrible Mistake”. “For example, I was struck by how European countries reacted differently to the same set of facts on the negative side-effects of one of the vaccines. Despite a shared objective – saving lives -, health authorities in each country took different decisions. We don’t know who is right and who is wrong, but if their conclusions are different, not all of them can be right. This is an example of noise in a domain where we would usually not be able to see it. By using more of the decision hygiene techniques we describe in the book, we would see less variability and better decisions.”

Any final thoughts to sum up what seems destined to be a book to be shelved next to other groundbreaking works like “Thinking, Fast and Slow” and “Nudge”? “Wherever there’s judgment, there’s noise – and probably a lot more of it than you think!” says Olivier Sibony with a knowing smile.

On October 7 (7 pm), Olivier Sibony will exchange with his co-author Daniel Kahneman, Emeritus Professor at Princeton University and 2002 Nobel Memorial Prize in Economic Sciences Laureate.

The title of HEC Paris Honoris Causa Doctor will be awarded to Daniel Kahneman at the end of this "HEC Talks" conference.

The conference will take place on the HEC campus, and will also be simultaneously broadcast on LinkedIn Live and Youtube.