Features, Labels and Overfitting — Why Most ML Backtests Are Fiction

A feature is an input the model sees. Concretely, it is one column of numbers, with one value for every row of your dataset. If your rows are "one stock, one week", then a feature might be the distance of the weekly close from the prior swing high, or this week’s traded volume divided by its own recent average. The defining property of a feature is not what it measures. It is when it becomes knowable — and that timing is where most of the damage in this article originates.

A label is the thing the model is asked to predict. It must be computable for past data, because that is what the model learns from. A label can be a number — the size of the move over the next twenty candles — or a category, which is usually easier: did price close above the high of the signal candle before it closed below its low? Categories tend to be more honest labels in market work, because they ask a question the data can answer cleanly rather than one that noise dominates.

Training is the fitting process. The algorithm looks at the training rows, compares its output to the labels you supplied, and adjusts its internal numbers to reduce the gap. That is all it does. It has no concept of markets, no idea what a stock is, and no interest in whether the pattern it found makes economic sense. It reduces the error on the rows you gave it, and it will reduce that error using anything in the columns that helps — including things that should not have been there.

A test set is data the model has never seen, held aside from the beginning and used exactly once, at the very end, to estimate what you actually have. Between training and test sits a third slice, the validation set, which is where you compare variants and make choices. The distinction matters enormously: the moment you use the test set to choose something, it has quietly become part of your training data and it no longer tells you anything.

Look-ahead bias is the use of information that was not available at the moment the decision would actually have been made. It is the most damaging of the three defects because it is invisible in the results and spectacular in its effect. A model that can see even a fragment of the future produces a backtest that looks extraordinary, and nothing about the equity curve announces the cheat.

The obvious form is a timing mismatch between the signal and the entry. Your rule says the signal is confirmed when the candle closes above the prior swing high, and your code records the entry at that same candle’s open. Read as prose, that sounds harmless. Read as timing, it means you entered several hours before you knew the candle would close where it did. The same error appears whenever a filter uses the day’s high or low and the entry is placed anywhere before that day ended.

A second form hides inside averages. A rolling calculation that includes the current bar is fine if the decision is made after that bar completes, and is look-ahead if the decision is made during it. Because the code is identical in both cases, the only way to catch this is to state, in writing, the exact moment at which each decision is taken — and then to check every feature against that moment. Not against the day. Against the moment.

The discipline that prevents all of this has a name: point-in-time thinking. For every column in your dataset, you should be able to answer one question — at the instant the decision is taken, would I have had this value, in this form, with this timestamp? If you cannot answer confidently, the feature is a suspect until proven otherwise, no matter how sensible it looks.

The obvious timing errors get caught. The quiet ones do not, and they are the reason experienced people still ship broken studies. The first is the restated figure. A financial number in a database today may be a later, revised version of what the company originally reported. Your model trains on the corrected figure; the market at the time was reacting to the original. You have handed the model a small, systematic piece of hindsight, distributed across exactly the rows where it matters most.

Closely related is the difference between the period a number describes and the date it was published. A quarter ends, and the results reach the exchange some weeks later. If your dataset attaches the figures to the period-end date, then every row in that gap is being fed information that did not exist yet. This single mistake has produced more impressive-looking fundamental backtests than any other, because the effect is largest precisely around the events that move price.

The third quiet form is the universe list. If you assemble your stock list today — the current NIFTY 50, or a screener output as of this morning — and then run a test back over the last ten years, every row in that test benefits from your knowing which companies would still be in the index a decade later. That is future information dressed up as a data-loading step, and it overlaps directly with survivorship bias, which the next section takes apart properly.

The fourth is whole-sample computation. If you normalise a feature using the mean and standard deviation of the entire dataset, or pick a threshold by looking at the distribution across all your data, then every early row has been scaled using information from the future. The fix is to compute such statistics on an expanding or rolling basis, using only data up to each row. It is a small code change and it frequently removes a large part of an impressive result.

Survivorship bias is simpler to state and harder to fix. A list of stocks assembled today contains only companies that still exist and still trade today. Everything that delisted, merged away, was suspended or simply faded off the exchange is missing. Run a ten-year backtest on that list and you have tested a strategy in a world where failure was quietly deleted before the test began.

The distortion is not evenly spread, which is what makes it dangerous. It hits hardest exactly where a strategy is most likely to claim an edge — in smaller companies, in turnaround situations, in anything that had a genuine chance of not making it. A rule that says "buy weakness and wait" performs very differently in a universe where the companies that never recovered have been removed from the sample.

The same problem appears in index work. Backtesting "the NIFTY 50" using today’s fifty constituents is not a test of the index. It is a test of a hand-picked basket that we now know performed well enough to still be in the index. What you want is a point-in-time universe — the list as it actually stood on each historical date, including the names that later left.

Point-in-time constituent data is genuinely hard for a retail trader to obtain, and pretending otherwise helps nobody. The honest response when you cannot get it is not to ignore the problem. It is to shrink the claim: state in your own notes that the universe is survivorship-affected, treat the result as an upper bound rather than an estimate, and be especially sceptical if the strategy leans on the weakest names in the list.

Overfitting is what happens when a model has enough flexibility to memorise the particular noise in your training data rather than the pattern underneath it. Think of a student who memorises the answer key for last year’s paper. Their score on that paper is perfect, and it tells you nothing about whether they understand the subject. Ask a new question and the performance collapses. A model with too many parameters relative to the amount of genuine information in your data behaves exactly like that student.

The pattern is easy to recognise once you look for it. Error on the training data keeps falling as you add complexity — more features, more parameters, more finely tuned thresholds. Error on data the model has not seen falls for a while, then turns and starts rising. That turning point is the boundary between learning the signal and memorising the noise, and it is the single most important curve in applied machine learning. Everything past it is self-deception with extra steps.

Market data makes this worse than in most fields for the reason the previous article set out: the signal is weak, so there is a great deal of noise available to memorise. And your effective sample is smaller than the row count. Consecutive rows are not independent — this week and last week on the same stock share most of their story — so a dataset that looks like tens of thousands of rows may carry the information of a few hundred genuinely distinct situations.

Then there is the defect that hides outside the model entirely: how many variants you tried. Every version you built, looked at and discarded was a hidden test. If you flip enough coins, one of them comes up heads many times in a row, and it is not a special coin. The variant you kept is the maximum over everything you tried, and the maximum of many noisy results is systematically flattering. The number of attempts is part of the result, and almost nobody writes it down.

This is also why the test set must be touched once. If you look at the test result, adjust something and look again, you have made yourself the optimiser and the test set has become training data by proxy. The protocol is not bureaucracy. It is the only thing standing between you and a number you generated by searching until you liked it.

The first defence is a strict three-way split of the data, decided before you look at anything. The training slice is where the model is fitted. The validation slice is where you compare variants and make every choice — which features, which parameters, which model. The test slice is sealed until the end and opened exactly once, to estimate what you have. If the test result disappoints and you go back and change something, you have not improved the strategy. You have spent the test set, and the honest move is to acknowledge that the next number you produce is a validation number, not a test number.

The second is walk-forward validation, which the next article covers in detail. In outline: fit on a window of history, test on the window immediately after it, then roll both windows forward and repeat. It matters because it mirrors how the strategy would actually have been run — refitted periodically, always applied to data that came later — instead of assuming a single fit that somehow held across two decades.

The third is counting. Keep a research log with one line per variant you tested, including the ones you abandoned after ten minutes. Most people record only the versions that worked, which is precisely the wrong half. When you eventually report a result to yourself, report it alongside the number of variants it beat. A strong result from the fourth thing you tried is a different animal from the same result found on the two-hundredth.

The fourth is a preference for robustness over peaks. If your strategy only works at one specific parameter value and degrades sharply on either side, you have found a fitted peak in the noise. If it works across a broad neighbourhood of settings, with performance changing gradually, you have something more likely to be real. Sensitivity analysis — deliberately varying each parameter and watching how gently the result changes — costs an afternoon and saves months.

The same audit works on your own study and on a result someone is showing you. Run through it before the number is allowed to change any behaviour. Most claimed edges do not survive the first four questions, which is a useful thing to discover in ten minutes rather than after you have committed capital to it.

Notice how much of the audit is about construction rather than performance. That is deliberate. Performance numbers are the output; construction decides whether the output means anything. A study with a modest result and a clean construction is far more valuable than a spectacular result whose universe, timing and attempt count are unstated.

What does a result that survives look like? It is worse than the in-sample version, usually noticeably worse, and it stays positive anyway. It holds across several walk-forward windows rather than resting on one exceptional stretch. It does not depend on a handful of enormous winners. It degrades gently when parameters move. And it still exists after realistic costs, which is the subject of the next article.

Finally, treat surviving the audit as permission to test further, not as proof. Non-stationarity means even a genuinely sound result has a shelf life. The audit tells you the study was built honestly. It does not tell you the relationship will still be there next year, and nothing can.