Free tool

Backtest overfitting checklist

This checklist has eleven yes-or-no questions that show how likely it is that a good-looking backtest is a product of the search rather than a real behaviour. Tick what is true of your test. The more boxes you can honestly tick, the less likely the result is overfitted, though no list can prove a strategy works.

Try it

Runs entirely in your browser. Nothing you type is sent, stored or logged, and no account is needed. The starting numbers are made-up and illustrative. Educational only, not investment advice.

How it works

Overfitting happens when a strategy is tuned so closely to past data that it captures that history's noise. It is hard to see from inside, because the symptom, a better backtest, is exactly what you were trying to get. So the checklist asks about the process rather than the result: how the rule was written, how many trades back it, how it was tested and what was held back.

Each item is something you can answer from memory about your own test. They cover writing the rule down first, having many more trades than settings, including costs, a minimum trade count, nudging parameters, testing on data you did not tune on, covering different market conditions, being able to explain each filter, counting your variations, checking a related market, and not changing the rules after looking at held-back data.

The tool simply counts the ticks and shows a short reading. Nothing you tick is saved or sent anywhere.

How to read the result

Start with the unticked items rather than the score. Each one is a specific weakness, and some matter more than others. Never having tested on data you did not tune on, and not knowing how many variations you tried, are the two most common reasons a strong result later disappoints.

The bands, ten or more, seven to nine, and fewer than seven, are an illustrative rule of thumb, not a standard or an AKQX scoring rule. They exist to nudge you toward the weak spots, not to certify anything.

A high count does not make a strategy good. It makes one particular failure less likely. Follow it with paper trading on live prices, which is the final out-of-sample test.

The example loaded above

Suppose you tested a breakout rule and can honestly tick seven items. You wrote the rule first, included costs, have well over 100 trades, nudged the settings, covered a rise and a fall, can explain your filters and counted your variations. You did not hold any data back, did not check another market, and have more settings than your trade count comfortably supports. Seven of eleven reads as some gaps. The useful next step is to hold data back, since that is the check most likely to change your mind about the result.

What this tool does not do

It cannot examine your test. It relies on you answering honestly, and the person most easily fooled by a backtest is its author.
A full score is not proof. It lowers the chance of one failure and says nothing about regime change, costs you underestimated or plain bad luck.
It does not replace out-of-sample testing. It only asks whether you did it.
The bands are illustrative. Different people reasonably weigh the items differently.

Educational only. AKQX is software only. It is not a SEBI-registered Investment Adviser or Research Analyst and gives no investment advice. This calculator does arithmetic on the numbers you enter. It does not tell you what to trade, how much to risk or whether any setup is sound. See the risk disclosure.

Questions about backtest overfitting checklist

What is overfitting in a backtest?

Overfitting is tuning a strategy so closely to past data that it fits that history's noise instead of a behaviour that will repeat. The backtest looks excellent because the strategy was shaped to fit it.

Why does the checklist ask about the process, not the result?

Because the result is what overfitting distorts. A better equity curve is what both a real edge and a fitted one produce, so the only way to tell them apart is how the test was run.

Does AKQX check for overfitting?

Yes. The scorecard includes an overfitting score, and a result is rejected automatically if it scores below 5. Walk-forward and Monte Carlo scores below 5 also cause rejection, as do too few trades. This checklist is a separate, human-run set of questions.

Why is the 100-trade item there?

Because AKQX rejects a crypto result with fewer than 100 trades. Below that, a good-looking number cannot be distinguished from luck.