How to avoid overfitting in a backtest
Overfitting is tuning a strategy so closely to past data that it fits that history's noise and fails on anything new. To avoid it, use few parameters and many trades, nudge each setting to check the result degrades gradually, test on data the tuning never saw, resample the outcomes, and treat an unusually beautiful backtest as a reason for suspicion.
Key takeaways
- Overfitting makes a backtest look better precisely because the strategy was shaped to fit it.
- Profit on the data you tuned on is the one thing overfitting guarantees, so it proves nothing.
- The defence is structural: fewer parameters, more trades, data held back, and checks that try to break the result.
- A good result should degrade smoothly when you nudge a setting, not fall off a cliff.
- Every filter you add makes the curve prettier and the strategy less likely to survive anything new.
Why is a better-looking backtest sometimes a worse sign?
History is one sequence of events. It contains real, repeatable behaviour and a great deal of noise: moves that happened once for reasons that will not recur. A strategy with enough adjustable settings can be bent to catch the noise as well as the behaviour, and from the inside the two are indistinguishable. Both just make the curve go up.
That is overfitting, and it has an uncomfortable property. The better you tune, the better the backtest looks and the worse the strategy is likely to be on new data. The number you are optimising and the thing you care about have come apart. Read the glossary entry for the short definition. The rest of this guide is about catching it.
The warning signs
- The result collapses when a parameter moves a little. A stop of 1.47% works and 1.4% and 1.5% both lose. A real edge degrades gradually near its best setting.
- Many rules, few trades. Each rule is a chance to fit noise, and few trades cannot tell an edge from luck.
- It works in one period and not the next. A strategy that earns in the tuned window and loses in every other has told you where it was fitted.
- A suspiciously smooth equity curve. Real strategies have rough patches. A curve with none was probably shaped.
- Rules you cannot explain. A filter that improves the backtest for no reason you can state is a candidate for noise.
- The result got better each time you tweaked it. That is the process of overfitting, observed.
Defence 1: keep the model small
The number of free parameters relative to the number of trades is the single most useful ratio. Every setting you can adjust is another degree of freedom for the data to be fitted by. Prefer a rule you can state in a sentence with two or three settings over one with ten, and prefer to test on hundreds of trades rather than dozens.
AKQX enforces a floor on this: a crypto result with fewer than 100 trades is rejected, and Indian-market results need at least 30, since higher timeframes there produce fewer signals. Below that, a good-looking number cannot be distinguished from luck.
Defence 2: nudge every parameter
After you find a setting that works, move it up and down by a small amount and rerun. You are looking at the shape, not the peak. A broad plateau, where results stay reasonable across a range of neighbouring values, means the setting is not critical and the behaviour is robust. A narrow spike means you found a value that happens to fit this particular history.
Do it for every parameter, not just the one you are proudest of. The result you want is one where you could be somewhat wrong about every setting and still be fine.
Defence 3: hold data back, and walk forward
The honest test is data the tuning never touched. The simplest version is to keep a slice of history aside, tune on the rest, and look at the held-back slice exactly once at the end. Looking twice, and adjusting, turns it back into training data.
Walk-forward testing repeats that idea through time: tune on a window, test on the next, slide forward, repeat. If a strategy is positive on most of the unseen windows its edge is less likely to be an accident of one period. If it earns only on the windows it was tuned on, it was fitting. AKQX scores walk-forward behaviour and rejects a result whose score is below 5.
Defence 4: resample the outcomes
A backtest is one ordering of your trades. A different ordering of the same trades gives a different equity curve, and a different worst drawdown. A Monte Carlo check shuffles the outcomes many times to see how bad the unlucky orderings could have been. AKQX runs 1,000 simulations and treats a fall of 30% from peak as ruin. What you look at is the share of runs that end in ruin, not the average run, because the average run is not the one that stops you trading.
This does not detect overfitting directly, but it catches a related failure: a result that looks fine only because the trades happened to arrive in a kind order.
A worked example, with made-up numbers
Illustrative only. A trader tests a moving-average crossover on 18 months of data, trying every fast length from 5 to 50 and every slow length from 20 to 200. That is thousands of combinations. The best one shows a profit factor of 2.4 over 60 trades.
Two things should worry them straight away. First, they chose the best of thousands, so some combination was always going to look good by chance alone. Second, 60 trades is a small sample for that much searching. Nudging the best pair by one step in either direction drops the profit factor to 1.1 and then below 1.0. The best combination is a spike, not a plateau.
The sound next step is not to trust the 2.4. It is to test that pair on a period it was not chosen from. If the profit factor there is close to 1.0, the 2.4 was the search finding noise. That is a useful result, because it was discovered with no money at stake.
The habits that keep you honest
- Write your hypothesis before you test. A rule chosen because of a reason is less likely to be a rule chosen because of noise.
- Count how many variations you tried. The more you searched, the less any single result means.
- Add a filter only if you can say why it should help. Not because the curve improved.
- Test other markets and timeframes. A real behaviour usually shows up, weakly, in related places.
- Then paper trade it. Live prices are the final out-of-sample test. See backtest vs paper vs live.
A reminder about what this is. AKQX is software only. It is not a SEBI-registered Investment Adviser or Research Analyst and gives no investment advice. These checks reduce the chance of being fooled by a backtest. They do not prove a strategy will make money, and simulated results never guarantee future ones.
Questions about how to avoid backtest overfitting
What is overfitting in a backtest?
Overfitting is tuning a strategy so closely to past data that it captures that history's noise instead of a behaviour that will repeat. The backtest looks excellent because the strategy was shaped to fit it, and it then disappoints on new data.
How can I tell if my backtest is overfitted?
Nudge a parameter slightly and rerun it. If the result collapses, the setting was fitted to noise. Other signs are many rules for few trades, a result that works in one period and not the next, and an equity curve that looks too smooth to be true.
How many parameters is too many?
There is no fixed number, but the ratio matters. Every adjustable setting is another degree of freedom for the data to be fitted by. A rule with eight settings tested on forty trades has far more room to be fitted than to be right. Fewer parameters and more trades both help.
Does walk-forward testing remove overfitting?
It reduces it and, more usefully, exposes it. Tuning on one slice and testing on the next unseen slice gives you an honest out-of-sample result. It stops working the moment you re-tune until the test windows look good, because then they have become training data.
Can a strategy be profitable and still be overfitted?
Yes, and that is the dangerous case. It made money on the history it was tuned to, and that profit tells you nothing about whether it will make money on anything else. Profit on the tuning data is the one thing overfitting is guaranteed to produce.
Does AKQX check for overfitting?
Yes. The scorecard includes an overfitting score, and a result is rejected automatically if it scores below 5. Walk-forward and Monte Carlo scores below 5 also cause rejection, and so do too few trades. These checks reduce the problem. They do not prove a strategy will work.
Keep reading
Test whether your result survives
A free account includes 10 backtests a month. AKQX rejects results that fail its overfitting, walk-forward and Monte Carlo checks.