Why backtests overstate live performance

The gap is mechanical

When a live account underperforms its backtest, the explanation offered is often about the trader: hesitation, deviation from the rules, emotional interference. Those things happen. They are also the least likely explanation when the strategy is automated and the rules were followed exactly.

In automated execution the gap is mechanical. Something the simulator assumed did not hold. The value of treating it that way is that mechanical causes can be found and measured, whereas behavioural explanations invite guesswork.

Six mechanisms account for most of the difference.

1. Bar sequencing that the data does not contain

A price bar records the open, high, low and close over an interval. It does not record the order in which the high and the low occurred. For any bar where price reached both the stop level and the target level, the outcome of the trade is undetermined by the data.

The simulator must pick. Picking the target produces a win, picking the stop produces a loss, and the two choices differ by the full width of the trade. In a strategy with a stop of a few points on a five minute chart, the proportion of trades containing this ambiguity can be substantial, and the reported return depends almost entirely on how it is resolved.

How to detect it: count the trades whose entry bar, or any bar held, contains both levels. Then rerun the backtest resolving every one of those bars as a loss. The difference between the two runs is the size of the assumption. If it is a large fraction of the total return, the simulator has been giving the strategy the benefit of the doubt several hundred times.

The remedy is to simulate on data finer than the decision timeframe. A strategy that decides on five minute bars and simulates fills on one minute or one second data observes the sequence instead of assuming it. Where the finer data does not exist, the pessimistic resolution is the only defensible one.

2. Fills at prices that were touched but not traded

A price appearing as the high of a bar means the market printed there. It does not mean size was available there, or that a resting order at that price was filled. Assuming a fill at the exact extreme of a bar is optimistic in a way that compounds across every trade.

The same category includes limit orders assumed to fill because price reached the level, stop orders assumed to fill at the trigger price rather than beyond it, and market orders assumed to fill at the last traded price rather than at the offer.

How to detect it: check what happens to the result when every entry is moved one tick against the strategy and every exit is moved one tick against it. A strategy whose profitability survives that adjustment is not depending on precise fills. A strategy that does not survive it was reporting a result that requires the market to cooperate.

3. Costs modelled as an average rather than as a drag

Spread, commission and slippage are commonly applied as a single average figure per trade, expressed in price. Expressing them in units of risk changes the picture.

Costs are incurred in price terms and are roughly independent of the stop distance. Their weight in units of risk therefore scales inversely with the stop. The same cost that consumes a twentieth of an R on a wide stop consumes a quarter of an R on a stop a fifth of the size. Strategies are often developed with wide stops and later tightened for efficiency, and the cost assumption travels with them unchanged.

The spread is also crossed twice, once on entry and once on exit, and it widens exactly when short timeframe strategies tend to trade: around the open, around scheduled data releases, and in the final minutes of a session.

How to detect it: express costs in units of risk rather than in price, recompute for the actual stop distribution of the strategy, and use the wider end of the observed spread during the hours the strategy is active.

4. Look-ahead through the data pipeline

Look-ahead bias enters through paths that are easy to miss because the code contains no obvious error.

• An indicator computed over a window that includes the bar being traded, so the decision uses the bar's own close.
• Economic or corporate data timestamped by its reference period rather than by its release moment, which places information in the past before it existed.
• A daily reference level, such as an opening range or a prior close, computed from a session that had not finished at the time of the decision.
• Data corrected after the fact, so the simulation trades on a price series that was different when it was live.

How to detect it: shift every input back by one full bar and rerun. A strategy whose result collapses was using information from the bar it was trading. Also check that any daily or session level used in a decision was fully determined before the decision, particularly around holidays and early closes, where the boundary rules change.

5. Parameter search on the data being reported

Trying many variants and reporting the best one produces an optimistic figure by construction, and the optimism grows with the number of variants tried. A search across fifty combinations will find something that performs well on any dataset, including a random one.

This is the mechanism behind strategies that look excellent in development and ordinary afterwards. Nothing was falsified. The number reported was the maximum of a search, and the maximum of a search is a biased estimate of what the strategy is worth.

How to detect it: record how many variants were evaluated before the reported one was chosen, and report the distribution of all of them rather than only the best. A strategy where most variants are profitable is describing a broad effect. A strategy where one variant works and its neighbours do not is describing an accident.

6. Survivorship in the instrument list

A strategy tested on the instruments that exist today has been tested on a list from which failures were removed. Delisted equities, discontinued contracts and closed pairs are missing, and they are missing precisely because of what happened to them.

The effect is largest in equity strategies and small for a strategy trading a handful of continuously listed futures. It is worth checking rather than assuming, because the correction is not available after the fact if the historical constituent list was never kept.

What an honest backtest looks like

The pattern across all six mechanisms is the same. Each involves the simulator resolving something it cannot know, and each resolution tends to favour the strategy unless deliberately forced the other way.

An honest simulation therefore has a specific character. It resolves ambiguity against the strategy. It prices costs at the worse end of what was observed during the hours it trades. It uses data finer than its decision timeframe to determine what happened inside a bar. It reports the number of variants examined. And it reports its result on data that took no part in its own construction.

Such a backtest produces a smaller number than the alternative. It is also the only version of the number that survives contact with a live account.