Overfitting is what happens when a strategy learns the accidents of one stretch of price history instead of any behaviour that will repeat. It is not a rare failure mode reserved for careless people — it is the default outcome of building a system on data you can see, and it gets worse every time you change a setting and re-run. Walk-forward validation is the discipline that fights it: you fix rules on one block of history, score them only on a block you have never looked at, and repeat that honestly enough that the number you end up with was never optimised.
Why is an in-sample backtest not evidence?
Take the sequence of NSE daily closes for the last ten years. Now hand a researcher a strategy template with six adjustable settings and let them try combinations until the summary page looks good. They will succeed. Not sometimes — reliably, and usually within an afternoon.
They will succeed because they are not testing a hypothesis against the data. They are searching the data for a shape that fits it. Those are opposite activities that produce identical-looking reports.
This is the whole problem in one sentence: **a result measured on the same data you used to choose your rules is not a measurement, it is a description**. It tells you what already happened. It carries no information about what happens next, because nothing about the future was ever consulted.
Watch out — The most dangerous backtest is not the one that fails. It is the one that looks excellent after eleven rounds of adjustment, because by then the researcher genuinely cannot tell which parts of the result came from the market and which came from their own hands.
What exactly is overfitting, in trading terms?
- Signal
- Behaviour that would show up again in data you have not seen. A tendency for large-cap NSE stocks to keep moving after a genuine range breakout, for example, if it exists at all, would be signal.
- Noise
- Everything else in the price series — the particular sequence of gaps, result days, policy announcements and one-off flows that happened to fall in that order and will never fall in that order again.
- Overfitting
- Building rules that fit noise. Every extra condition you bolt on carves the historical record more precisely, and past a point the extra precision is entirely spent on describing accidents.
- Curve fitting
- Used interchangeably in trading, though it usually names the crudest version: tuning numeric parameters until the historical curve looks the way you want.
- Selection bias in research
- The subtler cousin. You did not overfit any single strategy — you tested forty strategies and published the one that survived. The survivor's score is inflated by the thirty-nine you never mention.
How do parameter sweeps manufacture an edge out of nothing?
Suppose your rule has a lookback length, a filter threshold and an exit multiple. You test 12 lookbacks, 8 thresholds and 8 exits. That is 768 combinations, each scored against the same ten years of history.
One of those 768 will come out on top. It will always come out on top — not because it is best, but because the top of any list of 768 noisy measurements is the place where good behaviour and good luck happen to coincide.
This is the multiple-comparisons problem, and it is arithmetic, not opinion. The more combinations you compare against one fixed history, the more of the winner's score is luck and the less of it is behaviour. Reporting only the winner and not the sweep is how a 768-ticket lottery win gets presented as skill.
How much tuning can your data actually support?
There is a rough budget here, and most retail research blows through it without noticing.
Every adjustable choice is a degree of freedom: each numeric parameter, but also each filter you add, each instrument-universe rule, each 'except during results season' carve-out, and — critically — each re-run after you looked at the answer. Looking at the result and changing something is a degree of freedom even if no parameter changed, because your brain did the fitting.
What those degrees of freedom have to be paid for with is not years of history. It is **independent observations of the thing you are modelling** — which for a swing system means completed trades and distinct market episodes, not candles.
Effective evidence ≈ independent episodes observed ÷ degrees of freedom used- Independent episodes — distinct market conditions your rules actually traded through, not the number of rows in your CSV. Ten years of NSE data contains far fewer independent regimes than it contains days.
- Degrees of freedom — parameters + filters + universe rules + carve-outs + the number of times you re-ran after seeing a result.
- The ratio has no threshold to memorise. Its only job is to show you that adding a seventh filter to a system with 40 trades is not refinement, it is invention.
Note — This is a way of thinking, not a statistical test. There is no number that certifies a strategy as un-overfitted, and any tool claiming to compute one is selling comfort.
In-sample, out-of-sample, walk-forward, anchored — what is the difference?
These four terms are used loosely everywhere, including by people who should know better. They describe genuinely different amounts of evidence.
| Method | What you do | What it can honestly tell you |
|---|---|---|
| In-sample only | Fit and score on the entire history. | Nothing about the future. It is a description of the past, useful only to check that the code does what you meant. |
| Holdout (out-of-sample) | Fit on the earlier block, lock the rules, score once on a later block you never opened. | One honest reading — and only if you truly score once. It is a single sample, so it is noisy. |
| Rolling walk-forward | Fit on a fixed-length window, test on the window immediately after, roll both forward, repeat. Stitch the test windows together. | A sequence of honest readings under changing conditions, which is far more informative than one. Assumes only recent history matters. |
| Anchored walk-forward | Same, but the fit window starts at a fixed date and keeps growing rather than sliding. | The same sequence, under the assumption that older history stays relevant. Slower to adapt, less prone to refitting on a short recent stretch. |
How do you actually run a walk-forward test?
- 1
Decide the window lengths before you look at any result
Fit window and test window are themselves parameters. If you choose them by trying combinations and keeping the best, you have overfitted your validation scheme — the one thing that must stay clean. Pick them from a reason: the fit window should be long enough to contain several distinct market conditions, the test window long enough to produce a meaningful number of trades.
- 2
Fit on window one, and only window one
Run whatever optimisation you intend to run — including the full sweep. Sweeping is allowed here. What matters is that everything you learn stays inside this window.
- 3
Lock the rules, then score on the next window
No adjustment, no exceptions, no 'that one trade was clearly a data error'. The whole value of the test window is that it was not consulted.
- 4
Roll both windows forward and repeat
Do this across the entire history. You now have a chain of out-of-sample segments, each produced by rules chosen without seeing it.
- 5
Read the sequence, not the average
The average of the test windows hides the thing you most need to see: whether performance was scattered across the chain or concentrated in one or two segments. A system that only worked in one stretch of the chain has told you when it works, which is different from whether it works.
- 6
Re-run the whole thing with the anchored variant
If rolling and anchored disagree sharply, the difference is itself the finding — the system is sensitive to how much history it is fitted on, which is a fragility, not a detail.
- 7
Charge full Indian costs in every window
Brokerage, STT, exchange transaction charges, stamp duty, GST and a realistic slippage assumption. A validation run with costs switched off is not a validation run. Costs are covered in full in Costs, Slippage & Taxation.
What is the parameter plateau, and why does it matter more than the peak?
Here is the single most useful robustness check in systematic trading, and it takes minutes.
Do not look at your chosen parameter value. Look at its neighbours. Shift the lookback one step up and one step down; shift the threshold either side. Then ask: does the picture change gently, or does it fall apart?
A **plateau** is a broad band of neighbouring settings that all behave similarly. It suggests you have found something with some width to it — a behaviour that does not care whether the lookback is 18 or 22. A **spike** is one excellent setting surrounded by poor ones. A spike is almost always an artefact: the market has no reason to treat 20 as special and 19 and 21 as worthless.
Given the choice, take the middle of a plateau over the top of a spike, even though the plateau's centre will always score worse in the backtest. That gap is the price of not fooling yourself.
Why can you only spend the test set once?
This is the rule everyone agrees with and almost nobody follows.
You run the holdout. It disappoints. You go back, change two things, and run the holdout again. That second run is no longer out-of-sample — you have now used the test data to make a decision, which is the definition of fitting to it. Do it five times and your holdout has quietly become a second in-sample block, except now you have no honest test left at all and you believe you do. That is strictly worse than where you started.
The professional discipline is to treat the final block of history as a scarce, non-renewable resource. Everything — every idea, every sweep, every argument with yourself — happens on the development data. The final block gets opened once, at the end, to answer one question: does this thing survive contact with data it has never met?
Pro tip — Practical version: before you start, physically move the last stretch of your data into a separate directory and write the date you are allowed to open it. If you need a second honest test later, the only legitimate way to get one is to wait for the market to produce more data — or to run the system forward on paper, which does the same thing in real time.
What are the ways overfitting sneaks past a walk-forward test?
- **You designed the rules after reading the whole chart.** If you spent a year staring at NSE history before writing the rule, the entire history is in-sample no matter how you split it afterwards. The split cannot un-see what you saw.
- **Universe selection done with hindsight.** Testing on today's Nifty 50 constituents over ten years builds the answer into the question — those companies were selected for having done well. Survivorship and corporate actions are covered in Market Data, Corporate Actions & Bias.
- **Tuning the walk-forward scheme itself.** Trying six fit/test window combinations and keeping the best one reintroduces exactly the disease the scheme exists to cure.
- **Re-running after a disappointing out-of-sample.** Covered above, and worth repeating because it is the most common failure by a wide margin.
- **Leakage in the data pipeline.** Using a close to make a decision executed at the same close, or using an adjusted price series that embeds later corporate actions. The engine-side version of this is covered in Building Your First Backtesting Engine.
- **Publication bias against yourself.** You have run thirty ideas this year. The one you are excited about today is the survivor of thirty comparisons, and its score should be discounted accordingly — even though each one was individually validated.
What do the other validation terms mean, and when do they apply?
You will meet several more terms in the literature and in code libraries. Most of them are refinements of the same idea, and a few of them are traps when moved from general data science into markets.
- K-fold cross-validation
- Split the data into k parts, train on k-1 and test on the remaining one, rotate. Standard practice everywhere else and dangerous here in its plain form, because the folds are shuffled — which means the model gets to train on next month and test on last month. Time-ordered data does not permit that.
- Purging and embargo
- The fix that makes time-series cross-validation defensible. Purging drops training observations that overlap the test period; an embargo drops a further buffer immediately after it, so information does not leak across the boundary through overlapping holding periods.
- Paper-forward testing
- Running the finished, locked rules on live incoming data without money. Slower than everything above and better than all of it, because the data genuinely did not exist when the rules were fixed. It is the only test that cannot be accidentally reused.
- Sensitivity analysis
- Deliberately perturbing inputs — parameters, start dates, the instrument universe, the assumed slippage — and observing how much the picture moves. A result that only exists under one exact configuration is not a result.
- Reality check on the sweep
- Formal statistical corrections exist for the multiple-comparisons problem. They are useful and they do not rescue a badly designed study; a correction applied to a sweep whose size you did not honestly record corrects nothing.
Why does adding more data not simply solve this?
The instinctive fix is more history. Test on twenty years instead of ten and the problem should halve. It does not, for three reasons that are worth being explicit about.
**Older data describes a different market.** Indian market microstructure has changed repeatedly — settlement cycles, lot sizes, circuit rules, the instruments available, the composition of participation. A rule validated on a decade that no longer exists has been validated against a market you cannot trade in.
**Rows are not evidence.** Moving from daily to five-minute candles multiplies your rows by seventy-five and adds almost no independent episodes. The same market conditions are simply being sampled more finely. This is the most common self-deception in intraday research: an enormous dataset that contains very little independent information.
**More data invites more tuning.** In practice, researchers who obtain a longer history do not run the same sweep on more data. They run a bigger sweep, because the extra data feels like permission. The ratio that matters gets worse, not better.
Pro tip — A genuinely useful use of extra history is not to re-tune on it. It is to hold it back entirely and use it as a second, later honest test — or to test the finished rules on a completely different instrument universe you never developed on. Both give you information a longer sweep never will.
What does overfitting look like once the system is live?
The failure mode has a recognisable shape, and knowing it is the difference between diagnosing a problem and blaming the market.
An overfitted system does not usually break loudly. It performs roughly as expected for a short while — long enough for you to add capital — and then drifts into a period where the trades keep triggering, keep looking like the backtest, and simply stop working the way they did on paper. There is no single event to point at, because there was never a mechanism to break.
The honest tell is the comparison you should have set up before going live: the live sequence against the out-of-sample sequence, on the same terms. If live results sit outside the range of the walk-forward segments almost immediately, the walk-forward was not measuring what you thought.
This is different from genuine strategy decay, where a real edge is competed away over time and the deterioration follows an identifiable change in the market. Telling the two apart is covered in Strategy Decay & When to Switch Off — and the distinction only exists at all if you kept the out-of-sample record to compare against.
How is this different from overfitting a machine-learning model?
It is the same disease with different symptoms, and the Academy covers both because the defences differ.
A rule-based system overfits through **human hands**: you see the result, you change a setting, you re-run. The degrees of freedom are countable and visible, and the defence is procedural — split the data, budget your parameters, spend the test set once.
A learned model overfits through **capacity and leakage**: it has vastly more effective parameters than you wrote down, and the most common failures involve a feature quietly containing information from the future rather than a human tuning a knob. The defences there are about how features and labels are constructed and how folds are separated in time.
If you are building models rather than rules, read the AI module's treatment of features, labels and leakage alongside this page. If you are building rules, this page is the one that applies — and it applies fully, because a spreadsheet with four settings can be overfitted just as thoroughly as a neural network.
When is a walk-forward result itself untrustworthy?
Walk-forward is a defence, not a certificate. Here is what invalidates one.
- The out-of-sample chain contains too few trades to say anything. A handful of trades per test window is a story, not a sample.
- Performance in the chain is concentrated in one or two segments. The system worked in a regime, and you have learned which regime rather than that it works.
- The rolling and anchored variants disagree materially. The result depends on how much history you fitted on, which no live system can guarantee.
- Neighbouring parameter values collapse. You are sitting on a spike, and the walk-forward merely validated a spike repeatedly.
- Costs were modelled optimistically, or not at all. In Indian markets, cost realism decides more strategies than signal quality does — especially anything intraday.
- The universe was chosen with hindsight, or the data carries survivorship or corporate-action distortion.
- You cannot state, in one sentence, why the behaviour should exist. A validated result with no economic or behavioural story is a pattern in a specific past, and it has nothing holding it in place.
Note — This page is education, not advice, and nothing on it is a recommendation to build, buy or run any strategy. No performance figure of any kind is stated here, and none should be inferred from the diagrams — every figure on this page is schematic and deliberately carries no scale.
A checklist to run before you trust any validated result
Before you believe your own backtest
- Count your degrees of freedom honestly — including every re-run after seeing a result.
- State the number of combinations you compared. If you cannot remember, that is the answer.
- Confirm the test data was opened exactly once, on a date you wrote down in advance.
- Check the neighbours of every parameter. Plateau, or spike?
- Read the out-of-sample chain segment by segment, not as an average.
- Re-run with full Indian costs and a pessimistic slippage assumption.
- Re-run on a different instrument universe you did not develop on.
- Write the one-sentence reason the behaviour should exist. If you cannot, stop.
- Then paper-trade it forward. Live data you have not seen is the only test that cannot be reused.
Key points
Pro tip — Before you optimise anything, write down the number of parameter combinations you intend to test and the date you are allowed to open the final block of data. Tape both to the monitor. Almost every overfitting disaster in retail systematic trading is a researcher quietly exceeding one of those two numbers and never telling anyone, including themselves.
Frequently asked questions
What is overfitting in algorithmic trading?
Overfitting is when a strategy's rules end up describing the accidents of one stretch of price history rather than any behaviour that repeats. It happens whenever the rules are chosen using the same data they are then scored on — which is why adding parameters, filters and exceptions always improves the backtest and never improves the strategy past a point.
What is the difference between in-sample and out-of-sample testing?
In-sample data is what you used to choose and tune the rules. Out-of-sample data is a block you locked away and never opened while developing, then scored the finished rules on exactly once. Only the out-of-sample number carries information, and it stops carrying it the moment you go back and adjust something because you did not like it.
How does walk-forward validation work?
You fit the rules on one window of history, lock them, and score them on the window immediately after — which the fitting never saw. Then you roll both windows forward and repeat across the whole record. Stitching the test windows together gives a chain of honest readings under changing conditions instead of a single noisy one.
What is anchored walk-forward, and how is it different from rolling?
In rolling walk-forward the fit window is a fixed length that slides forward, so the system is always fitted on recent history. In anchored walk-forward the fit window starts at a fixed date and keeps growing, so older history never drops out. Rolling adapts faster; anchored is less prone to being refitted onto a short recent stretch. If the two disagree sharply on the same strategy, that disagreement is itself a fragility worth investigating.
How many parameters is too many for a trading strategy?
There is no fixed number, because what matters is the ratio between adjustable choices and independent market episodes your rules actually traded through — not candles, and not years of data. A system with four settings and several hundred trades across multiple regimes is on far firmer ground than one with two settings and thirty trades. Remember to count filters, universe rules and every re-run after seeing a result as degrees of freedom too.
What is a parameter plateau, and why does it matter?
A plateau is a broad band of neighbouring parameter values that all behave similarly, which suggests the underlying behaviour is real and has some width to it. A spike is one setting that works while its immediate neighbours do not, which is almost always an artefact of the specific data — markets have no reason to treat a 20-candle lookback as meaningful and 19 as worthless. Choosing the middle of a plateau over the peak of a spike costs you backtest score and buys you a result that might survive.
Can I re-test on my out-of-sample data after changing the strategy?
Not without destroying it. The moment a test result influences a change, that data has been used for fitting and is no longer out-of-sample. Doing this repeatedly leaves you with no honest test while making you more confident, which is worse than having no test at all. The only legitimate ways to get a fresh honest test are to wait for the market to produce new data or to run the system forward on paper in real time.