What is Overfitting?

What does Overfitting mean in practice?

Every added filter or tweaked parameter makes the historical curve look better and the strategy slightly less likely to survive anything new. With enough parameters you can fit almost any past, which is why a beautiful backtest is a warning as often as a recommendation.

The signs are performance that collapses when a parameter moves slightly, a large number of rules for few trades, and a result that works on one period and not the next. The defence is fewer parameters, out-of-sample testing and robustness checks.

A worked example

Illustrative only: a stop of 1.47% looks best, while 1.4% and 1.5% both lose money. A real edge should degrade smoothly near the best setting, not fall off a cliff.

How to read or test it

Count your free parameters and your trades. A rule with eight adjustable settings tested on forty trades has more room to be fitted than to be right. Nudge each parameter up and down by a small amount and watch the result: a plateau means the setting is not critical, while a spike means it was tuned to noise. Then keep a slice of data you never touch while tuning, and look at it exactly once at the end.

How AKQX handles Overfitting

AKQX scores overfitting directly and auto-rejects any result whose overfitting score is below 5. It also runs walk-forward and Monte Carlo checks, which push against the same problem from other directions.

See Backtesting methodology for the feature page.

A common mistake

Adding one more filter to rescue a weak result. Each filter is another degree of freedom for the data to be fitted by.

Related terms