Evaluating an AI Trading Tool Before You Pay — Claims and Red Flags

Take the number off the landing page and try to reconstruct what it means. You immediately find that you cannot, because at least four things have been left out, and each of them can move the figure enormously. How many trades is it computed over. What period those trades span. What counted as a win. And whether costs were deducted before the win was counted.

Sample size comes first because small samples produce impressive numbers by accident. A handful of outcomes can look extraordinary for reasons that have nothing to do with skill, in the same way a short run of coin flips can come up heads repeatedly. The larger the sample, the harder it becomes for luck to carry the result — which is exactly why a vendor with a genuinely large sample tends to state it prominently, and a vendor without one tends to state a percentage instead.

Period matters because market conditions are not uniform. A method tested only across a stretch when the broad market trended upward has not been tested against the conditions that hurt it. Ask which years, and ask specifically whether the test period includes a sharp drawdown and a long directionless range. A method that has only met one kind of market has not been tested; it has been introduced.

Then costs, which is where more claims die than anywhere else. Indian equity and derivative trading carries brokerage, exchange charges, statutory levies and — the one most often ignored — slippage, the gap between the price you expected and the price you got. A short-holding-period method touches those costs constantly. A result computed before costs and a result computed after costs are not the same result with a small adjustment; for high-turnover approaches they can be different conclusions entirely.

The useful property of the questions below is that they are cheap to answer if you have done the work and impossible to answer if you have not. Someone who has genuinely tested a method knows the sample size the way you know your own phone number. Someone who has not will produce a paragraph about their proprietary approach, and the length of the paragraph is inversely related to the strength of the evidence.

Send them in writing, together, and read what comes back as a whole. You are not looking for perfect answers — a small sample honestly disclosed is far more encouraging than a large one vaguely implied. You are looking for whether the vendor treats measurement as something they did or as something they are being interrogated about.

One question deserves special attention because it is the one almost nobody thinks to ask: how many variants were tried before this one was chosen. If a hundred parameter combinations were tested and the best one is being sold to you, the reported performance is partly a measure of how many attempts were made, not how good the method is. Testing many things and reporting the winner is the most common way an honest person produces a misleading result.

The follow-up that separates the serious from the rest: is this backtested, forward-tested, or live? A backtest is a simulation on history the developer could see while building. A forward test runs on data that arrived after the rules were fixed. Live means real money, real fills, real costs. These sit in ascending order of credibility, and the gap between the first and the last is where almost all disappointment lives.

The difference is not tone and it is not confidence. Plenty of meaningless claims are stated modestly and plenty of meaningful ones are stated with pride. The difference is falsifiability: can you, in principle, check it and find it wrong? A claim you could never disprove is a claim that was never at risk of being false, which is another way of saying it told you nothing.

“Our AI analyses millions of data points” cannot be wrong, because analysing is undefined and millions is unverifiable. “We processed every quarterly filing for the NIFTY 50 constituents from this date to that date, and here is the extraction schema” can be wrong — you could check a filing and find the extraction missing. The second sentence is less exciting and vastly more informative.

This test also works on capability claims, not just performance ones. “Understands market context” means nothing. “Reads a PDF you upload and returns a fixed set of fields with a page reference beside each one” means something, because you can upload a PDF and see whether the page references are correct. Ask for the checkable version of any claim, and notice how often the vendor cannot produce one.

Apply the table below to whatever is in front of you. Most marketing copy sits entirely in the right-hand column, and the exercise of trying to move a claim from right to left is usually enough to end the evaluation without any further work.

Some signals are not about weak evidence. They are about the shape of the offer itself, and they should end the conversation regardless of how impressive the technology looks. The first is any language that promises or implies a return. Guaranteed, assured, risk-free, sure-shot, fixed monthly income from trading — none of these can honestly be said about a market-linked activity by anyone, and a party willing to say them has already told you how they handle inconvenient truths.

The second is testimonials as evidence. A screenshot of a profitable trade proves that one profitable trade existed. It says nothing about the trades not screenshotted, and the selection is being done by the person trying to sell you something. Reviews and profit screenshots are the weakest form of evidence in this domain and are treated as the strongest by most buyers, which is precisely why they are used.

The third is manufactured urgency. Closing in six hours, three seats left, price doubles tomorrow. Urgency exists to prevent the thing you are doing right now — thinking about it. A product with real evidence does not need you to decide before you have read the evidence, and a genuine capacity limit can be stated once without a countdown timer attached.

The fourth is subtler and specific to India. A “signal” service that tells you which security to buy or sell, at what price, with a target and a stop, is not a neutral piece of software however it is packaged. Telling people what to buy or sell is a regulated activity here, and the label on the product does not change what the product does. If a person or entity is issuing recommendations to you, the registration question below is not optional.

Before the accuracy questions, before the trial, before anything: if the offering involves someone telling you what to buy or sell, find out whether that person or entity is registered with SEBI. Research analysts and investment advisers operate under registration, and registration is what makes a real person accountable for what they publish — with disclosures about their own positions and conflicts, and a defined channel when something goes wrong. An unregistered party offering recommendations has none of that structure behind them, and neither do you.

The check itself is quick and you should do it yourself rather than accept a claim of registration. SEBI publishes lists of registered intermediaries on sebi.gov.in. Look up the registration number, confirm the name attached to it matches the party you are dealing with, and confirm the category is the one relevant to what they are offering. A registration number printed on a website is a claim like any other until you have seen it on the regulator’s own site.

Be clear about what registration does and does not tell you. It tells you there is an accountable, identifiable party operating under a regulatory framework with disclosure obligations. It does not tell you the person is skilled, that their calls will work, or that you will make money — SEBI itself is explicit that registration guarantees neither performance nor returns. It is a floor, not a recommendation, and treating it as an endorsement is its own error.

This applies to this site too. Rohit Singh is a SEBI Registered Research Analyst, registration number INH000015297, trade name INVESTOLOGY, and you should verify that on sebi.gov.in rather than take this sentence for it. Every question in this article — what is the sample, what is the period, were costs included, what is the evidence — applies to content published here exactly as it applies anywhere else. A writer who asks you to be sceptical of everyone except themselves is asking for the wrong thing.

Suppose the vendor answered well and the registration checks out. You still know nothing about whether the tool helps you, because that depends on your process, not on their evidence. So the trial is a real experiment and needs to be run like one, which mostly means deciding what would count as success before you start rather than deciding afterwards.

Pick one task you already do — reading quarterly filings for a set of names, say, or the first pass over your watchlist — and run the tool against work you have already done or can verify. This is important. If you test it on unfamiliar material, you cannot tell a good output from a confident wrong one. Testing on your own ground is the only way to see the failure modes that matter to you.

Keep a plain log for the trial: date, what you asked, what it returned, what you checked, what was wrong. Two weeks of that log is worth more than any amount of reading about the product, because it measures the only thing that matters — whether the output was defensible on your material. And the log has a second use: it stops you from remembering the trial through the lens of whichever outcome happened last.

At the end, ask three questions. Did it save time that was genuinely being wasted, or did it move the work from producing to checking? Did the errors it made cluster somewhere predictable, so you know where not to trust it? And would you still want it if the market had been flat throughout? If the honest answer to the third is no, you were evaluating conditions rather than the tool.