We tested 21,191 trading rules from 656 books. Zero survived.
We extracted 21,191 mechanical rules from 656 trading books and backtested every one. After correcting for the number of trials, none showed a statistically significant edge. Here is the method, the result, and what it changed.
Most trading education rests on an untested assumption: that the setups in the books work. Not that they are wise, or well-explained, but that a rule extracted from a book and applied mechanically produces a positive expectancy.
We tested it. The answer was no.
What we did
We extracted 21,191 mechanical rules from 656 trading books — entry conditions, filters, exits, position-sizing rules — and converted each into a machine-evaluable form: a set of indicator conditions on OHLCV data across seven timeframes.
Then we backtested them. Not a sample: a continuous mill that has been running 24/7 for months, evaluating each rule against historical bars per symbol, per timeframe, per direction.
The result
Of 2,791 rules with enough data to score, 31 cleared a Probabilistic Sharpe Ratio of 0.95. Zero cleared a Deflated Sharpe Ratio of 0.95.
The best DSR was 0.632. The next three were 0.310, 0.289 and 0.286.
Why that distinction is the entire finding
The Probabilistic Sharpe Ratio asks: given this track record's length and shape, how confident are we that the true Sharpe is above zero?
The Deflated Sharpe Ratio asks a harder question: how confident are we, given that we tried thousands of things and are reporting the winners?
If you test 2,800 rules against noise, roughly 140 will look significant at the 5% level by chance alone. That is not a flaw in the rules. It is arithmetic.
Our 31 "significant" rules were fewer than chance would produce from pure noise at that number of trials. Correcting for the trial count is what turns 31 apparent winners into zero real ones — and it is the step almost nobody publishes.
The trap we nearly fell into
An earlier pass of the same data looked far more encouraging: 48% of rules showed a profit factor above 1.
That number is meaningless. Coin flips produce roughly 50% profit factor above 1. A metric that a random process satisfies half the time cannot distinguish skill from luck, and reporting it as evidence would have been the same error in a friendlier costume.
We also found the harness itself was distorting the answer, in the opposite direction. Scored through our live filter stack, rules were being suppressed before they could trade — measured, that made them look 4.5x worse than they were. Removing that distortion was necessary to get an honest number. It did not reveal an edge. It removed an artifact of systematic harm and left the real answer visible underneath.
What does measure
The same research found something that replicates cleanly: these setups predict magnitude, not direction.
Across ten years, 44 symbols and 154,743 events, directional expectancy showed no significance at any threshold — maximum t-statistic 1.32. But the size of the subsequent move is monotone in the strength of the signal, replicated across four independent setups, and large: a big-gap momentum event lands at roughly the 75th percentile of that symbol's own hourly range.
That is a genuinely useful thing to know. It is just not the thing the rules were written to claim.
Four controls, each of which changed the answer
Every one of these was necessary before any number meant anything:
- Exact-minute time-of-day matching. Bucketing by half-hour made every 09:30
- Day-clustering. Treating each signal as independent called 43 of 49 slices
- Bonferroni correction for the number of slices tested.
- Avoiding barrier-censoring, so a wider stop does not manufacture a higher
gap look predictive: one setup scored 57.2 with a t of +4.0, and dropped to 51.3 once matched properly.
significant. One observation per day left three.
win rate.
The rule we took from this: on this data, an apparent edge is a confound until a tighter control has failed to remove it.
What we did with the result
We did not deploy the rules. We did not quietly bury the test and ship the feature anyway.
The library still runs, because a mechanically-evaluated rule is a useful lens even when it is not a signal. What changed is the claim attached to it. Our platform describes what the tape did and what our own record shows, and where a number has no validated record behind it, the interface says so on the page.
If you are evaluating any research product, including ours, the question worth asking is not what its win rate is. It is what it tested, how many things it tried, and whether it corrected for that.