Five Sigma

For years, physicists at the LHC had a bump in their data where the Higgs boson should be. They would not call it a discovery, and the reason was a number.

The convention is exact, and Louis Lyons — who wrote the paper on where it came from — states it plainly:

Three in ten million. About one chance in three and a half million that the background alone could have thrown up something this big.

And the discipline is enforced socially, not just statistically. Lyons notes that journals are very reluctant to allow the word 'discovery' to appear in the title, abstract or conclusions of a paper, unless the p-value is that small.

Hold that beside a fact about what was at stake. If the physicists had been wrong, the universe would have been unaffected. No particle would have suffered. The cost of a false claim was a retraction and some embarrassment.

They set the bar at one in three and a half million anyway.

documented  the convention → why 5 and not 3

The 5σ bar was not derived from theory. It was set because the field kept announcing things that turned out not to exist.

Read that as an institutional confession. Three sigma is already odds of about one in a thousand. Four sigma is about one in sixteen thousand. Physics found that even those were producing claims that evaporated, and so raised its own bar against its own enthusiasm.

That is the move worth stealing. Not the specific number — the willingness to notice your field's error rate and respond by making your own life harder.

And it went further. Lyons prints, in his own paper, the objection that the whole thing is unsound:

He does not bury that. He states it, gives the statisticians' reasoning — that probability distributions rarely describe reality accurately that far out in the tails — and then concedes: "Thus the Statisticians' criticism may well be valid."

A field that publishes the case against its own gold standard is doing science. Keep hold of that sentence; the last third of this zine is about what it looks like when a field does not.

documented  false claims → the look-elsewhere effect

Search a wide enough space and something improbable will turn up somewhere. That is not a discovery. That is arithmetic.

Physics has a name and a correction for this. The local p-value asks how unlikely a fluctuation this big is at the mass where you found it. The global p-value asks how unlikely it is anywhere you were looking — and it is always larger, sometimes enormously so.

The correction depends on how honestly you define "anywhere," and Lyons enumerates the choices: any reasonable mass, anywhere in the current analysis, or anywhere in analyses by anyone in the whole collaboration. Then this:

That is pre-registration, argued as ordinary practice, in a physics note, in 2013 — while psychology was still discovering it needed the idea.

Now count the search space on the other side. Thousands of researchers, hundreds of candidate interventions, dozens of outcome measures each, decades of trials, and a publication system that prints the positives. The look-elsewhere effect there is not a correction term. It is the dominant feature of the landscape.

And almost nobody applies it.

documented  multiplicity → extraordinary claims

How much evidence you need depends on what you are claiming. Physics builds this in deliberately, and calls it the subconscious Bayes' factor.

The reasoning is straightforward once stated. The evidence in front of you multiplies your prior belief; it does not replace it. So a result that would be convincing for an expected effect is not convincing for an impossible one.

He is explicit that this is a Bayesian argument sitting inside a frequentist practice, and defends it anyway:

So hold the question open, because the whole zine turns on it. Is it an ordinary claim or an extraordinary one to say that a standardised intervention, delivered to wildly heterogeneous children, produces a durable improvement in their lives?

Physics would want to know your prior. Ours, after a century of this collection's first chain, is not high.

documented  priors → systematics

Statistical error is the part you can compute from your sample. Systematic error is everything else — the miscalibrated instrument, the selection you did not notice, the assumption baked into the model.

It is much harder to estimate, and physics treats that difficulty as a first-class problem rather than a footnote. Lyons puts a number on the exposure:

Read that twice. Being wrong by a factor of two about your unmeasured errors turns a one-in-3.5-million result into a one-in-eighty result. The entire distance between "discovery" and "shrug" is inside a factor of two on the thing you did not measure.

And Lyons is clear that the honest response is to model the uncertainty, not to paper over it with a bigger number:

Keep that factor of 2 × 10⁴ in your hand for the next few spreads. It is the answer to a question this zine is going to ask about a literature that does not measure its harms at all — because you cannot underestimate a systematic by a factor of two if you never put a number on it in the first place.

documented  the four dials → Lyons' table

Here is the part almost nobody outside physics knows, and it is the reason this zine exists. Lyons does not merely explain 5σ. He argues against using it uniformly — and publishes a table of what different claims should require instead.

His section heading is literally "Why not 5σ?", and it opens: "There are several reasons why it is not sensible to use a uniform criterion of 5σ for all searches for new physics."

The dials are the ones we have just walked: degree of surprise, impact, look-elsewhere effect, systematics. Turn them, and the required significance moves:

Single top gets 3σ — predicted by theory, no surprise, low impact, no look-elsewhere problem. Gravitational waves get 7σ despite being unsurprising, because the search space is enormous. Superluminal neutrinos get more than 8σ, because the claim would break causality.

And Lyons is careful about the status of his own numbers:

This is a worked-out theory of how much evidence is enough, calibrated to what you are claiming and what it would cost to be wrong. It exists. It is short, readable and free.

documented  the graded table, as published

There is a second number in the world, and almost every claim ever made about a child rests on it.

In Statistical Methods for Research Workers (1925), Ronald Fisher wrote the sentence that set it:

Two things in that sentence deserve a much longer look than they get.

First: "convenient." Not derived, not calibrated, not justified by an error-rate analysis. A working statistician picking a round number because the arithmetic was tidy and the tables were printable. He said so.

Second: "1.96, or nearly 2." The most consequential threshold in the human sciences is, in the units this zine has been using all along, two sigma.

Physics looked at 3σ and 4σ results, watched them evaporate, and moved up. The human sciences settled at 2σ and stayed for a hundred years.

It is worth saying that Fisher himself did not intend a bright line — he treated significance as one input to judgement, and elsewhere used other levels. The hardening is not his fault. But it hardened, and it hardened into the gate that decides what gets published, funded, mandated, and done.

documented  Fisher's convenience → the hardened rule

Put the two numbers side by side and divide.

The ratio is about 170,000. That is the factor by which one field is more willing to be wrong than the other — and it points the wrong way.

Because now ask what a false positive costs on each side.

Physics gets it wrong: a paper is retracted. A conference is awkward. More data is collected. The particle was never there and never minded either way. The universe is not altered by our having been mistaken about it.

We get it wrong: a child spends forty hours a week for years being trained out of behaviours that were regulating them. A reader is drilled on phonics while the thing that was actually hard goes unexamined. A teenager is told their problem is insufficient grit. And there is no more data to collect, because the childhood already happened.

You cannot un-apply an intervention to a childhood. There is no revision, no fuller dataset, no second run at ages four to nine. Irreversibility is a property of the claim, and it belongs on the dial marked impact.

documented  the two thresholds, and their ratio

Lyons built an instrument for deciding how much evidence a claim needs. Let us point it somewhere he did not, and be honest that we are the ones pointing it.

The claim: this intervention should be delivered to this child, intensively, for years, during the years in which they are becoming a person. Turn the four dials.

Every one of those settings is at or above the level Lyons assigns to supersymmetry — a claim he thinks needs 7σ. Nothing here is stretched to get there. Enormous look-elsewhere and enormous impact are simply what the situation is.

contested  physics' evidentiary framework → human interventions

The previous spread is the most persuasive thing in this zine, which is exactly why it gets a spread arguing against it. Here is the case we would have to answer.

One: physics is asking a different kind of question. A 5σ threshold is for an existence claim — is the particle there at all? Most claims about interventions are about effect size in a system where something is already happening. "Does this exist" and "how much does this help" are not the same question and do not obviously take the same machinery.

Two: the null hypothesis is not neutral. In physics, "no new particle" is a genuine, costless default. With a distressed child there is no null option — doing nothing is also an intervention, with its own risks, and a threshold so high that nothing ever clears it is a decision to do nothing, made by arithmetic and disguised as rigour.

Three: physics has repeatable identical trials. Every proton collision is like every other. No two children are, and demanding particle-physics certainty about heterogeneous humans could be a demand that can never in principle be met — which would make it a rhetorical device, not a standard.

Four: the cost of a false negative is real too. Withholding genuine support from a child who needed it is also a harm, and it is one that a very high evidential bar produces systematically.

None of them argues for 1.96σ. Every one of them is a reason the threshold should be thought about — which is Lyons' actual position, and the opposite of an unexamined convenience inherited from 1925. Objection two is the strongest, and it cuts toward our argument as much as against it: if doing nothing is also an intervention, then it needs an evidence base too, and the ecosystemic answer — change the conditions, not the child — is the option that keeps getting scored as "nothing."

And none of the four explains the finding on the next spread, which is not about thresholds at all.

documented  the objections, and what they leave standing

Everything so far has been an argument about where to set a bar. This is not that. This is about a column that has nothing in it at all.

Bottema-Beutel, Crowley, Sandbank and Woynaroski reviewed 150 reports of group-design, non-pharmacological intervention studies for young Autistic children, and asked a simple question: did the study report whether anyone was harmed?

Eleven. Out of a hundred and fifty. And of the 54 studies that gave reasons for participants withdrawing, ten reported reasons that could be categorised as adverse events, and twelve more were described too vaguely to tell.

Go back to spread six and the factor of 2 × 10⁴. Lyons showed what happens to a 5σ claim when you underestimate your systematic uncertainty by a factor of two.

A drug trial that did not record adverse events would not be a weak trial; it would not be a trial. In physics, an analysis that omitted its dominant systematic would not be a marginal result; it would be unpublishable. Here it was the overwhelming majority of the literature, and that literature is what "evidence-based" points at.

The same group later put the point in a title: Evidence-b(i)ased practice: selective and inadequate reporting in early childhood autism intervention research.

documented  the reporting gap → what gets called evidence-based

It is not one literature. The same evidentiary standard is doing the same work in three places at once.

Behavioural intervention. The literature reviewed above — where the harms column is largely blank, and where the outcome measure inherited from 1987 was indistinguishability.

Growth mindset. Macnamara and Burgoyne meta-analysed 63 studies, 97,672 participants, and reported evidence of publication bias, evidence that authors with a financial incentive to find positive effects were more likely to report them, and major threats to internal validity. Their finding on quality is the one to hold: higher-quality studies were less likely to show a benefit. That is the signature of an effect that is not there.

The Science of Reading. Presented as settled science and legislated in state after state, while reading researchers point out that the case rests on a small number of concepts taken from a few simple, dated studies (Seidenberg), and that the movement's certainty outruns its evidence.

That last clause is where the evidentiary argument becomes an ethical one. A weak finding published in a journal harms nobody. A weak finding written into a state statute is applied to every child in the state. The look-elsewhere effect made the finding; the legislature made it compulsory; and the child has no standing in either process.

This is what measuring the surface, badly produces. Not merely a wrong answer — a wrong answer with the force of policy behind it, and the word science on the front.

documented  three evidence bases → the same standard

There is a word for using the authority of science without the discipline of science, and it is not science.

Scientism is the aesthetics: the p-value, the effect size, the phrase evidence-based, the citation, the graph. What it leaves behind is the practice — pre-registration, blinding, adverse-event reporting, replication, publishing your nulls, printing the case against your own standard, and calibrating your threshold to what you are claiming and what being wrong would cost.

And it has a characteristic victim, which is why this argument belongs to us and not only to methodologists. When a field claims scientific authority while ignoring inconvenient findings and lived experience, the people who have the inconvenient experience are not merely disbelieved — they are disbelieved on scientific grounds, which is a much harder thing to answer.

That is epistemic injustice (Fricker), and Robert Chapman has traced how it operates specifically against neurodivergent people: our testimony about our own lives arrives already discounted, because the apparatus has spoken.

We have watched this happen twice already in this collection. In No. 9 the border between disorder and difference is redrawn by committee while the people either side of it stay the same. In No. 39 people asking about the air are told they are catastrophising, by institutions resting on a number nobody had checked.

Same shape every time. The tools of science, pointed at human beings, used to flatten exactly the variation that the science elsewhere insists is real.

contested  the evidentiary gap → epistemic injustice

Everything in this zine is borrowed from physicists. That is the point, and it is worth being unmistakable about it.

We did not arrive with a philosophical objection to measurement. We arrived with a physicist's own paper, and asked why the standard it describes — calibrate your threshold to how surprising the claim is, how much impact it carries, how wide you searched, and how badly you understand your own errors — stops at the laboratory door.

Eugenics and behaviourism did not fail because they were too scientific. They failed because they took real mathematics — a curve for observational error, a law of effect measured honestly in cats — and applied it to human beings without the discipline that made it work in the first place. That is the whole subject of this collection.

And there is a version of this that is not a complaint but a design. Set the bar by what it costs to be wrong. Where an intervention is reversible, low-stakes and chosen, a modest evidence base is fine. Where it is compulsory, intensive, delivered during childhood and impossible to undo, ask for something closer to what you would want before announcing a particle — and until you have it, change the conditions rather than the child, which is the option that has been scored as "no intervention" this whole time.

That is not anti-scientific. A field that raised its own bar because its own results kept evaporating would recognise it immediately.