Luck looks exactly like skill
Over a few hundred trades, a rule with no edge can produce a smooth, rising equity curve by chance. Nothing on the chart tells you which part came from the rules. A study of Taiwanese day traders from 1992 to 2006 found that fewer than 1% were predictably profitable after fees: many had good years, very few could repeat them.
There are three kinds of luck: direction luck (your entries happened to catch a few big moves), market luck (you only bought while the market rose) and selection luck (you kept the best of many tries). Each needs its own check.
Test 1: random side (coin flips)
A signal claims to know which way price will go. So keep every entry time, stop distance, target distance and exit time exactly as they were, and let a coin decide buy or sell for each trade. Replay those trades on the real prices with the same costs. Do it 200 times.
Replay properly: the reverse of a winning trade is not simply a loser of the same size, because the stop and target swap sides and both directions pay the spread.
Test 2: random entries
A long-only rule in a rising market passes the coin-flip test easily, because the flipped copies sell into a rally. So for rules that mostly buy, place the same number of long trades at random times in the same period, with stops scaled to the market's volatility at that moment, the same holding times and the same costs. Again, beat 95% of 200 runs. Now the rule must beat other buyers, not sellers.
Buying at every one of the strategy's entries (same stops, targets, exits): +226.4R in total
Selling at every one of them: +11.2R
The strategy itself: +340.9R
Its longs averaged +0.535R per trade (t 3.1); its shorts +0.089R (t 0.8)
Bitcoin rose about 8 times over the period: most of the result was being long
Step by step on your own trade list
- Export each trade: entry time, direction, stop distance, target distance, exit time and result in R after all costs.
- Compute your average R and t: t = average / (standard deviation / square root of the number of trades). Write both down, with the date, before going on.
- Build one coin-flip copy: same entries, stops, targets and exit times; random direction per trade; replay on the price data with the same costs.
- Repeat 200 times and record each copy's total R. Sort the 200 totals.
- Count how many copies did at least as well as your real total. More than 10 means you fail the test.
- If your rule mostly buys (or mostly sells), add 200 random-entry runs in the same direction and apply the same 95% bar.
- Count how many versions you tried before this one. If it is many, raise your t bar (our 25-idea test needed t >= 3.11).
Average +0.15R per trade over 50 trades: t = 0.15 / (1 / square root of 50) = about 1.06
The same average over 200 trades: t = 0.15 / (1 / square root of 200) = about 2.12
Same edge, four times the trades: twice the t
A real example: the trend-day rule
A trend-day continuation rule on three US indices made +0.13R per trade on 2017-2022 (t 2.84 on 985 trades). On 2023-2024 it made -0.022R per trade, and 49.5% of its 200 coin-flip copies did as well. A coin was as good a guide to direction as the rule. Of the 14 candidates our edge search froze for validation, 9 had more than 5% of their coin flips doing as well on the new data.
It also helps to know what pure noise looks like. On synthetic indices, whose prices come from a broker's random number generator, we ran 491 tests for links between products. With no real links, about 5% of such tests should look 'significant' at the 5% level and about 1% at the 1% level, by chance. We found 6.1% and 1.2%, and none survived a correction for the number of tests. Test enough things on noise and some will always look good.
Limits
The test checks direction and market drift. It does not prove the edge will last, that it is large enough to trade, or that it survives your broker's costs. It is a filter: a strategy that fails it has shown nothing yet.
Key takeaways
- Keep entries, stops, targets and exits; randomise only the direction; beat 95% of 200 copies.
- For buy-mostly rules, also beat 200 random long entries.
- Count your tries and raise the bar to match.
- Our trend-day rule matched coin flips on new data: 49.5% did as well.
Sources
Education only, not investment advice. Results quoted from TradeLab research are historical simulations; trading can lose more than you expect.
