Backtesting an AI-Generated Strategy — The Honest Test

Generated strategy code carries the same defects as hand-written code, plus three of its own. None of them is exotic, and every one of them is easy to miss precisely because the code reads so well. Fluent, well-commented, correctly structured code invites you to review it for correctness rather than for timing — and correctness is not the property in question.

The first is that the model writes the most conventional version of whatever you asked for. Ask for a breakout backtest and you will get the textbook implementation, which in an enormous number of published examples enters at the same bar the signal is computed on. That convention is inherited from thousands of tutorials where the shortcut was never the point. It arrives in your code as a default you did not choose and did not notice, because it looks exactly like every other backtest you have read.

The second is that the model does not know your data. It writes plausible code against an assumed data frame — assumed columns, assumed timestamps, assumed adjustment for splits and bonuses, assumed handling of corporate actions. If your actual data has already-adjusted prices, or timestamps in a different convention, or figures attached to period-end rather than announcement date, the code will still run. It will simply be answering a slightly different question from the one you asked, silently.

The third is the one that compounds fastest: generation is cheap, so you try far more variants than you ever would have by hand. Twenty strategies in an afternoon is now routine. Every one of those is a hidden test, and the best of twenty noisy results is flattering by construction — a point the previous article makes in detail. The speed that makes generation attractive is exactly what inflates the multiple-comparisons problem underneath it.

The practical response is not to distrust generated code. It is to review it for a different property. Read it for timing, data assumptions and execution realism — not for whether it compiles or whether the logic matches your description. It almost certainly does match your description. That was never the risk.

The single most consequential decision in the whole procedure is made before a line of code runs: how the data is divided, and the commitment to divide it that way regardless of what the results look like. Made afterwards, a split is not a split. It is a selection.

Three slices, in chronological order. The earliest and largest becomes training data, where the rule is fitted or the parameters are chosen. The middle slice is validation, where you compare variants and make every choice — which features, which thresholds, which of your twenty generated candidates survives. The final and most recent slice is the test set, which stays sealed until the end and is opened exactly once, to estimate what you actually have.

Chronological order is not optional in market work. Randomly shuffling rows into three buckets, which is the standard approach in most machine learning tutorials and therefore a common default in generated code, destroys the entire exercise. Shuffled rows put next month next to last month in the training data, and the model gets to learn from a period that a live trader would not yet have lived through. It is look-ahead bias introduced by a single line of convenience.

There is a subtler version worth naming: a gap between slices. If your label looks forward twenty candles, the last twenty rows of the training slice overlap in time with the first rows of the validation slice. Leaving a buffer of at least the label horizon between slices removes that overlap. It costs a little data and removes a leak that would otherwise flatter every validation number you produce.

And the rule that gives the split its meaning: if the test result disappoints and you go back to change something, you have not improved the strategy — you have spent the test set. The honest move at that point is to say so, and to treat every subsequent number as a validation figure rather than an estimate of unseen performance. Nothing about this is bureaucratic. It is the only thing standing between you and a result you obtained by searching until you liked it.

A single train-validate-test split answers one question: would this rule, fitted once on early data, have held up later? That is worth knowing, but it is not how anybody actually operates a strategy. Real strategies are refitted periodically as new data arrives. Walk-forward testing reproduces that, and it is the closest thing to an honest simulation available to a retail researcher.

The mechanics are simple. Fit on a window of history — say the first stretch of data. Test on the window immediately following it, using only what the fit on the earlier window produced. Then roll both windows forward by the length of the test window and repeat, all the way to the end of your data. What you finish with is not one out-of-sample result but a sequence of them, each produced by a model that only ever saw data preceding it.

That sequence is far more informative than any single number, and it is the reason to bother. A strategy that holds across most windows is a different proposition from one that is carried entirely by a single exceptional stretch — and the two can produce an identical aggregate figure. Averaging the windows throws away exactly the information you wanted. Look at the sequence.

There are two variants and the choice between them is a real one. An anchored walk-forward keeps the training start fixed and lets the window grow, so later fits see all history. A rolling walk-forward keeps the training window a fixed length, so old data drops out as new data comes in. Anchored assumes relationships persist indefinitely; rolling assumes recent conditions matter more. Neither is correct in general, and running both is informative: if the two disagree sharply, that itself tells you the relationship is unstable over time.

Two practical cautions. First, walk-forward is not immune to overfitting — if you tune the window lengths until the walk-forward result improves, you have overfitted the walk-forward procedure itself, which is a genuinely popular way to fool yourself. Set the window lengths from a reason you can state, such as how often you would realistically refit, and leave them alone. Second, every window must independently respect point-in-time discipline. A leak inside one fit propagates into every window that follows it.

Generated backtests almost never model costs, and when they do it is usually a single percentage applied to the trade value, chosen because it sounded reasonable. For an Indian equity trader the real cost stack has several distinct components, they behave differently, and the total is what turns a great many nominally positive strategies negative.

Enumerate them explicitly rather than lumping them. Brokerage, which varies by broker and plan and differs between delivery and intraday. Securities Transaction Tax, which differs by segment and by whether the trade is a delivery, an intraday sale, or a derivatives transaction. Exchange transaction charges levied by NSE or BSE. SEBI turnover fees. Stamp duty, charged on the buy side. Goods and Services Tax, which applies to brokerage and to certain of the charges rather than to the trade value. And, for anyone testing derivatives, the same list with a different structure again.

Do not take any of these rates from a model’s memory. Rate schedules change, they differ by segment, and a confidently recalled figure from training data may be several revisions out of date. Read the current rates from the exchange’s published schedule, from SEBI, and from your own broker’s tariff — then verify against an actual contract note, which is the one document that reflects what you were really charged. That reconciliation takes an hour and is the difference between a cost model and a guess.

Then there are the two costs that never appear on any schedule and often exceed the ones that do. Slippage is the gap between the price your backtest assumed and the price you would actually have received. If your test fills at the closing price, ask honestly whether you could have transacted at that close, in your size, and how often. Impact cost is the movement your own order causes, and in less liquid names it is not small. A strategy that trades thin mid-caps in meaningful size and assumes frictionless fills at the close is testing a market that does not exist.

A piece of illustrative arithmetic, with invented numbers, purely to show the shape of the problem. Suppose a rule takes two hundred round trips a year, and suppose the all-in cost of a round trip — every charge above, plus slippage — comes to 0.15 percent of turnover. Two hundred multiplied by 0.15 is 30 percent of turnover consumed in charges over the year. Those two inputs are assumptions invented for this example and represent nothing measured; substitute your own verified figures. The point is only that turnover multiplies cost, and a high-frequency rule must clear a bar that a low-turnover rule never faces.

Which leads to the test that costs nothing to run. Take your cost assumption and double it. If the result still stands, you have something with a margin of safety in it. If doubling costs eliminates the entire edge, then your strategy was never a market observation — it was a bet on your cost estimate being exactly right, and cost estimates are the one input you can be certain will vary.

Suppose the study survives everything so far. It still has one question left, and it is the one that decides whether any of this matters in practice: could you actually have run it? Not could a spreadsheet have run it — could you, with your capital, your temperament and your obligations, have kept executing it through its worst stretch without stopping?

That is what drawdown measures. The peak-to-trough decline is only the headline; the two figures that decide behaviour are how deep and how long. A decline that recovers within a few weeks is an inconvenience. The same depth stretched across a year or more is a period during which every input you have — your own results, other people’s opinions, your doubts about the method — argues for abandoning the rule. Most strategies are not abandoned because they stopped working. They are abandoned in the middle of a drawdown that the backtest had shown, and that the researcher had looked at without imagining living through.

The number of consecutive losing trades matters for the same reason and gets far less attention. A rule that produces long strings of small losses before an eventual gain is psychologically very different from one that loses less often. The backtest is indifferent to that sequence. You will not be, and neither will anyone whose money is involved.

Then ask what the result depends on. Remove the largest few winners from the trade list and see what remains. If the entire outcome rests on three exceptional trades, you are not looking at a repeatable process — you are looking at a sample that happened to contain three exceptional events, and the next sample may not. This test is trivial to run and it retires a surprising number of promising studies.

And ask whether the sample is a sample at all. A strategy that produced eleven trades over a decade has produced a story, not a statistic. There is no threshold above which a count becomes sufficient, because it depends on how much genuinely independent information the trades carry — consecutive trades on the same stock in the same regime are far from independent. But a count in the low tens should stop the process, no matter what the aggregate figure says.

Position sizing sits on top of all of this and is not a detail. A backtest that assumes full capital deployed on every signal is describing a different strategy from the one you would run with a fraction of capital per position. If the sizing rule changed, the drawdown changes with it, and the result you validated no longer applies. Test the sizing you would actually use.

There is a genuinely good use of a language model in this whole process, and it is not writing the strategy. It is attacking it. Models are agreeable by default — ask whether your idea is good and you will usually be told it is promising — but that default is a matter of instruction, and it reverses cleanly when you tell it to argue the other side.

The adversarial pass is useful precisely because it is cheap and you are the wrong person to do it. You have spent hours on the study. You know which parts you checked carefully and you have quietly stopped examining them. A model with no attachment to the result, instructed to find every route by which the number could be an artefact, produces a list you would not have generated — and its value does not depend on it being right about any particular item.

Give it the construction, not the performance. What the universe was and how it was assembled. What period. The exact decision moment and the execution assumption. The features. The split. How many variants preceded this one. What costs were modelled. Withhold how well it did, or the answer will be shaped by the number rather than by the method.

Ask it for three things: the routes by which the result could be an artefact, ranked by how likely each is given your specific setup; the evidence that would rule each one out; and the questions you have not answered that it most needs answered. That last item is usually the most valuable output. It is a list of the things you skipped, produced by something that cannot be embarrassed about having skipped them.

Then treat the output the way you should treat all model output: as a set of claims to check, not findings. It will occasionally flag something that is not a problem in your setup, and it will miss things too. Its job is to widen your search, not to certify anything. The certification, if it exists at all, comes from the procedure — and even a perfectly executed procedure certifies only that the study was built honestly, never that the relationship will still be there next year.

It is worth describing the shape of an honest result, because it does not resemble what most people expect and the mismatch causes real damage. Researchers discard sound work for looking unimpressive and pursue flattering work for looking convincing, which is the wrong way round in both directions.

A result that survived is worse than the in-sample version — usually noticeably worse — and remains positive anyway. That gap between in-sample and out-of-sample is not a defect in the study; it is the study working. A test that comes back as good as the fit should increase your suspicion rather than your confidence, because the most common cause of that outcome is a leak rather than a discovery.

It holds across most walk-forward windows rather than resting on one. It survives the removal of its largest few winners. It degrades gently as parameters move rather than collapsing off a peak. It still exists after realistic costs, and preferably after doubled costs. It has enough trades to be a sample. And its worst drawdown is one you can describe honestly to yourself as survivable.

It also comes with its limitations written down beside it, in the same document as the result. The universe is survivorship-affected because point-in-time constituent data was unobtainable. The costs are modelled at an assumed level. The period covers these conditions and not those. A caveat you have to read again next to the number is worth many times one you remembered once and can no longer locate.

And it comes with an expiry date, which is the last thing to internalise. Markets are non-stationary: participants change, rules change, liquidity changes, and relationships that were real can stop being real without announcing it. Surviving this entire procedure is permission to proceed carefully and keep watching — never proof. What to watch for, and how a working process degrades silently rather than failing loudly, is the subject of the model drift article.