Why most backtests flatter
A backtest tuned on the same data it is judged on mostly proves that the tuning worked. Every choice you make by looking at results (indicator, period, stop, market, session) fits the rule a little more closely to the past, including its noise. The result looks better than the rule really is, and the gap only shows on data the rule has never seen.
This tutorial is the short version of the method we used for 882 pre-registered rule settings on CFDs, crypto perpetual futures and synthetic indices. Fourteen candidates came out of the search periods. None survived. The method is what told us so, before any money did.
In-sample, out-of-sample and the lockbox
Split time into blocks and decide the split before you look at any result:
| Block | Our CFD dates | What you do there |
|---|---|---|
| SEARCH (in-sample) | June 2017 to Dec 2022 | Explore freely; log every setting you try |
| VALIDATION (out-of-sample) | 2023 and 2024 | Run once, only on a short list frozen after the search |
| LOCKBOX | 2025 onwards | Open once, at the very end, only for rules that passed validation |
The lockbox protects you from a slow leak: if you test many ideas on your validation block and keep the ones that pass, the validation block quietly becomes part of the search. In our research the lockbox was never opened, because nothing passed validation, and the 2025+ data remain untouched for future tests.
Worked example: what the second block did
SEARCH (2017-2022): +0.13R per trade after costs, t 2.84 on 985 trades
Positive on 3 of 3 markets and in 5 of 6 years; neighbouring settings also positive
VALIDATION (2023-2024): -0.022R per trade, t -0.35
49.5% of 200 coin-flip copies did as well as the rule
It passed every check that can be made on the data it was built on. Only fresh data could show it was a run of favourable days. In our 39-strategy assessment the pattern was the same: 6 of 223 tests reached t 2.0 on 2017-2023, all on Bitcoin's one-hour chart, and only 1 confirmed on 2024-2026, and that one stopped passing once overnight financing was charged.
Costs and fills: the simulation must be fair
- Fill at the next bar's open after the signal, not at the signal bar's close.
- Charge the spread recorded at each fill, plus commission and overnight financing.
- When a bar touches both stop and target, count the stop first unless you have finer data.
- Never fill a limit order at a better price than its own level. In our Smart Money Concepts tests, 45% of simulated limit fills were better than the level; re-pricing them took the best setting from +0.133R per trade to about zero.
- Skip trades whose stop is inside the spread; their results in R are meaningless.
- Require the result to stay positive at 1.5 x the spread.
Two extra rules protect against lucky settings. A plateau: a setting is only interesting if its neighbours (one step either way on each parameter) also work, because a real effect rarely lives at one exact number. Breadth: it should work on most of the markets you test, not one. In our search a near miss that bought markets near their 52-week high failed the plateau rule (a neighbour lost 0.034R per trade), and a 20-day breakout took 88% of its profit from Bitcoin and Ethereum.
Step by step
- Write the plan in a text file: rule, settings to try, markets, the three date blocks, costs, and exactly what counts as a pass.
- Fingerprint it: run sha256sum plan.txt (Linux) or certutil -hashfile plan.txt SHA256 (Windows) and save the hash with the date somewhere you cannot edit later.
- Search on the first block only. Log every setting you try, including the ones you throw away.
- Freeze a short list by rules fixed in advance: neighbouring settings also positive (a plateau), profitable on most markets tested, enough trades (we required 200 and t >= 2.5).
- Run the frozen list once on the validation block. Pass bar: positive with t >= 2.0 on at least 100 trades, better than 95% of 200 coin-flip copies, still positive at 1.5 x the spread.
- If you find a mistake, fix it openly: add a dated amendment, hash it again, and raise the bar for anything already seen (we used t >= 2.5).
- Open the lockbox once, only for survivors, with the bar raised for how many survivors there are.
Limits
A hash proves the plan was not changed after the results, not that nobody peeked before writing it. Short histories, especially in crypto, give few independent regimes to test on. And a pass is a historical simulation, not a promise or a recommendation.
Key takeaways
- Split your data into search, validation and a sealed final block before you look.
- Charge real costs, fill at the next open, count the stop first and stress at 1.5 x the spread.
- Write the plan down and hash it; correct only with dated amendments.
- Of our 14 frozen candidates, none survived the validation block.
Sources
- Notices of the American Mathematical Society (2014). Pseudo-mathematics and financial charlatanism: the effects of backtest overfitting on out-of-sample performance. Link
- Econometrica (2000). A reality check for data snooping. Link
- Journal of Finance (1999). Data-snooping, technical trading rule performance, and the bootstrap. Link
Education only, not investment advice. Results quoted from TradeLab research are historical simulations; trading can lose more than you expect.
