Advanced8-12 min readTopic 18 of 20

    Infrastructure, Monitoring & Kill Switch

    Rohit Singh

    Mr. Chartist · SEBI RA

    Module Progress
    0/20
    Module

    A strategy is the part everyone wants to talk about. The part that decides whether you survive is the layer underneath it: the machine it runs on, the signals it reports about itself, the log that lets you reconstruct any decision months later, the alert that reaches you when you are nowhere near a screen, and the switch that stops everything. None of that is glamorous and none of it improves a single trade. It exists for the session where the feed stalls at 09:34, or the position tracker and the broker disagree, or a rule you wrote at midnight starts sending orders you did not intend — and on that day the operational layer is the entire difference between an incident and a disaster.

    The five things a live system has to report about itself, the state that should raise an alarm, and — the part that matters — the action each alarm triggers.Five monitored signals in rows: heartbeat, data freshness, order-reject rate, position against expected position, and profit and loss against the modelled figure. Each row shows the healthy state, the state that should raise an alarm, and the action that state triggers — halting new entries, stopping order flow, or halting and flattening. The position mismatch row is the one that flattens rather than pauses.SIGNALHEALTHYRAISE THE ALARM WHENWHAT IT TRIGGERSHeartbeata beat every 30stwo beats missedhalt new entriesData freshnesslast tick < 5s oldfeed stalledhalt new entriesOrder-reject raterejects near zeroa run of rejectsstop sendingPosition vs expectedthe two agree exactlyany mismatchhalt, then flattenPnL vs modelledmoves togetherdiverges all sessionhalt, then reviewA signal nobody has agreed an action for is a dashboard decoration, not monitoring.
    The five things a live system has to report about itself, the state that should raise an alarm, and — the part that matters — the action each alarm triggers.

    What is the operational layer, and why does it come before the strategy?

    Once a system places orders without you, you have stopped being a trader and started being an operator. The questions change. Not 'is this a good setup' but 'is the thing running', 'is it doing what I think it is doing', and 'how fast can I stop it'.

    The reason this comes first is asymmetry. A weak strategy loses money slowly and visibly. A broken operational layer loses it quickly and invisibly — a stalled feed that keeps firing rules on a frozen price, a reconnect that opens a second copy of every position, a retry loop that sends the same order forty times. None of those is a strategy failure, and no amount of backtesting finds them.

    The correct order is therefore: build the controls, then let the strategy trade inside them.

    Note — This page is education, not advice, and it recommends no broker, no hosting provider and no vendor. The thresholds below are illustrative specifications of a mechanism — set your own against your own system and your own instruments.

    What must a live trading system monitor about itself?

    Monitoring is not a dashboard. A dashboard is something you look at, and the whole problem is that at 09:34 you are not looking. Monitoring is a set of signals, each with a threshold you agreed in advance and an action that fires without you.

    Five signals carry almost all of the value.

    1. Heartbeat — is it alive?
    The system writes a timestamp on a fixed interval to somewhere outside itself. A watcher checks that timestamp. If two beats are missed, the system is presumed dead and the halt chain runs. The heartbeat must be written by the trading loop itself, not by a wrapper — a wrapper that is alive while the loop is wedged tells you nothing.
    2. Data freshness — is the picture current?
    The age of the last tick received, checked against the clock, not against the last tick's own timestamp. A feed that stalls is far more dangerous than a feed that disconnects, because a stalled feed looks fine: prices are present, they simply stopped changing. Every rule keeps firing on a frozen market. During NSE hours, a quiet feed is a fault until proven otherwise.
    3. Order-reject rate — is the broker refusing you?
    Count rejects in a rolling window. A single reject is information; a run of them is a system fighting a condition it does not understand — margin exhausted, a frozen quantity limit, a throttle, an instrument restriction. The action is to stop sending, not to retry. A retry loop against a reject is how one mistake becomes a thousand orders.
    4. Position vs expected position — do you own what you think you own?
    The single most important check on the list. Compare the position your code believes it holds against the position the broker reports, continuously. They must be equal. Any mismatch — one share — means your book is wrong, every subsequent risk calculation is wrong, and the system must halt and flatten rather than pause and hope.
    5. PnL vs modelled — is it behaving like itself?
    Not a profit target. A consistency check: the live figure and the figure the strategy's own logic implies should move together. A persistent divergence through a session means the system is doing something other than what you modelled — a wrong multiplier, an unaccounted charge, a stale reference price, or a trade you did not know about.

    Watch out — A monitored signal with no agreed action is decoration. Before you add a metric, write the sentence: 'when this crosses X, the system does Y, without me.' If you cannot finish that sentence, you have built a chart, not a control.

    Where should the system actually run?

    Two honest options, and neither is the right answer for everyone.

    A machine you own is free, fully under your control and immediate to debug — the screen is right there. It is also subject to your building's power, your broadband, a Windows update at 09:25, someone unplugging it, and the laptop lid closing. A rented server in a data centre removes your household from the failure list and keeps running while you are on a train, at the cost of a monthly bill, remote-only debugging, and a dependency on somebody else's uptime you cannot influence.

    The decision is not about latency. At the horizons a retail systematic trader operates on, the milliseconds are almost never the binding constraint, and 'low latency' is mostly a sales pitch. Decide on which failure you can live with at 09:20 on a day the market is moving.

    The same system, two homes, five honest differences. No provider is named here on purpose — this is a category decision, not a product recommendation.A five-row comparison between running a trading system on a local machine and on a rented remote server: power and internet dependency, whether the session runs while you are away, how debugging works, cost, and whose failure you end up owning. Neither column is presented as the right answer, and no provider is named.NEITHER COLUMN IS THE ANSWER — THEY FAIL DIFFERENTLYYour own machineA rented remote serverPower and internetyour flat's, and its outagesa data centre's, with its own failuresYou are away from itthe session runs without youthe session runs without youDebuggingimmediate — it is on your screenthrough a remote session and logsCostalready paid fora monthly rentalFailure you ownone machine, one dependencyone machine, plus someone else'sChoose on which failure you can live with at 09:20, not on latency you cannot measure.
    The same system, two homes, five honest differences. No provider is named here on purpose — this is a category decision, not a product recommendation.

    What has to be true of the machine, wherever it runs?

    • It starts the system automatically after a reboot, and the restart recovers real state — the actual position and pending orders, never a fresh empty book.
    • Its clock is synchronised. Every timestamp you will later reconcile against depends on it, and drift silently corrupts your entire log.
    • Credentials and API keys live outside the code, in environment configuration, and never in a repository.
    • It does exactly one job. A machine also running downloads, a browser and a game is a machine with a failure mode you did not plan.
    • You have a written manual procedure for flattening positions without the system — a broker terminal, an app, a phone number. If the machine is the only way to reach your positions, you have no fallback.
    • You know what happens if it dies mid-session, because you have deliberately killed it once and watched.

    What should the log contain?

    The purpose of a log is not to record that things happened. It is to let you answer, weeks later and under pressure, exactly why the system did what it did at 11:07:33 — without guessing, and without re-running anything.

    That means logging decisions and their inputs, not just outcomes. 'Placed buy order' is useless. 'At 11:07:33.104, on the bar closing 11:07:00, with the last tick aged 0.4s, rule breakout_v3 evaluated true because close 100.40 exceeded range high 100.25; position was 0 of a cap of 400; sent buy 200 limit 100.45, client id X' can be argued with. The first log makes you a witness. The second makes you an investigator.

    Every entry a reconstructable log needs

    • A timestamp with milliseconds, from the synchronised clock, on every line.
    • The inputs the decision saw — the bar, its close time, and how old the data was at that instant.
    • The rule that fired and, in plain terms, the comparison that made it true.
    • The state at decision time: current position, pending orders, margin available, limits used.
    • The exact order payload sent, with your own client order id.
    • Every response received, including rejects, with the broker's own message verbatim.
    • Every fill, with quantity and price, as its own line — a partial fill is an event, not a footnote.
    • Every limit check, including the ones that passed. A limit that only logs when it blocks cannot be proven to have been running.
    • Every halt, every alert raised, and every manual intervention, with who did it and why.
    • Files rotated daily and kept. Storage is cheap; an unreconstructable session is not.

    Pro tip — Test the log by trying to use it. Pick a random trade from last week and reconstruct the full decision from the log alone, with the code closed. Whatever you had to guess is the field you are missing.

    What should alerting look like when you are away from the screen?

    Alerting is the bridge between the monitor and you, and it fails in one of two directions. Too quiet and you learn about the incident at 15:35. Too noisy and you learn to ignore it, which is the same thing with extra steps.

    The fix is tiering. Decide, in advance, which conditions are worth interrupting your day for, and route everything else to a place you read later.

    • A critical alert must not depend on the trading process still being healthy — the process that is broken cannot be trusted to report that it is broken.
    • Every alert names the action, not just the condition. 'Position mismatch — system halted, flatten manually and do not restart' is an alert. 'Warning: mismatch' is an anxiety.
    • Send a daily heartbeat message even when nothing is wrong. Silence is ambiguous: it means either a quiet day or a dead alerter, and you cannot tell which.
    • Route critical alerts through a different path from the trading connection. Losing your internet must not also lose your ability to be told you lost your internet.
    TierExample conditionHow it reaches youWhat the system does
    CriticalPosition mismatch, heartbeat lost, run of rejectsA push that gets through — phone, not emailHalts by itself first, then tells you
    WarningFeed slow, one reject, a limit approachedA message you will see within the sessionKeeps running; you decide
    InformationalStarted, stopped, orders placed and filledA log or a channel you review after hoursNothing
    DailyEnd-of-day reconciliation summaryOne message, same time, every sessionNothing — but its absence is itself an alert

    How should the kill switch work, and who can trigger it?

    A kill switch is not a button that stops your program. Killing the process does nothing about resting orders sitting in the exchange book or the position already open — both survive your terminal closing perfectly well.

    A kill switch is a sequence, and it has to run in order.

    Stop new, cancel resting, flatten open, stay down. Three things may pull it, and the broker's own controls are the layer that still works when your code is the thing that is broken.A kill switch runs four steps in order: stop opening new positions, cancel every resting order, square off what is already open, and stay down until a human restarts it. Three triggers can start the chain — the monitor automatically, the operator from a phone, and the broker's own controls as the last resort that does not depend on your code. A footer records that the chain must carry a date it was last tested.IN THIS ORDER — STEP 3 IS THE ONE PEOPLE FORGET TO BUILD1 · STOP NEWno fresh entries2 · CANCELpull every resting order3 · FLATTENsquare off what is open4 · STAY DOWNrestart needs a humanTHREE THINGS MAY PULL ITThe monitorautomatically, on a breached limitYoufrom a phone, away from the deskThe broker's own controlsthe layer that survives your code dyingEvery chain carries a stamp: LAST TESTED — and a date you can point at.last_tested = <date> · fired_live = true · time_to_flat = measured, not estimatedA kill switch you have never fired is a paragraph in a document.Test it in live hours on size small enough that the test costs only time.
    Stop new, cancel resting, flatten open, stay down. Three things may pull it, and the broker's own controls are the layer that still works when your code is the thing that is broken.

    What are the rules a kill switch has to satisfy?

    1. 1

      It runs outside the strategy

      A check inside the trading loop cannot be trusted to fire when the trading loop is the thing misbehaving. The halt must live in a separate process, or better, in a control your broker enforces on their side.

    2. 2

      It flattens, it does not merely pause

      Stopping new orders while leaving an open intraday position running into the broker's square-off is not a halt. Decide in advance whether the switch also closes positions — and for most retail systematic setups, it should.

    3. 3

      It is reachable from a phone

      Incidents do not wait until you are at your desk. If firing it requires a laptop, a VPN and a remote session, it does not exist on the day you are on a train.

    4. 4

      Anyone accountable can pull it

      The monitor pulls it automatically on a breached limit. You pull it manually on judgement. And your broker's own controls are the final layer, because they keep working when your machine, your code and your connection are all gone.

    5. 5

      Restarting requires a human and a reason

      After a halt the system stays down until a person checks what happened and records why it is safe to resume. Auto-restart after a kill is the single worst feature you can build — it converts one incident into a loop.

    6. 6

      It is tested, on a schedule, with a date attached

      Fire it deliberately during live hours, with a position small enough that the test costs only time. Measure how long it actually takes from decision to flat. Repeat after every change to the system, and write the date down where you can see it.

    Watch out — The kill switch you have never tested is not a kill switch. It is a paragraph in a document, and the day you need it is the day you discover the API call was renamed, the credentials expired, or step three was never written at all.

    What does the end-of-day reconciliation involve?

    Every session ends the same way, whether it was busy or the system never traded: three records are tied out against each other, and the tie-out happens before the next session opens.

    Your system's book, the broker's reported orders and trades, and the contract note. Quantity, price and charges must agree across all three, and a break of one share is still a break.Three records must agree at the end of every session: the system's own book of what it believes it did, the broker's reported order and trade book, and the contract note that settles with every charge included. Quantity, price and charges are tied across all three, and any break is investigated the same evening rather than carried into the next session.ALL THREE, EVERY SESSION, BEFORE THE NEXT ONE OPENSYour system's bookwhat the code believes it didBroker order bookwhat the API reports backContract notewhat settles, with all chargesTIE OUT THESE THREEQUANTITY, PER INSTRUMENTAVERAGE PRICE, PER FILLCHARGES, AGAINST MODELLEDA break of one share or one paise is still a break.An unexplained break is not carried forward. It halts the system the next morning.Intraday products square off on the broker's schedule — reconcile against that, not your assumption.
    Your system's book, the broker's reported orders and trades, and the contract note. Quantity, price and charges must agree across all three, and a break of one share is still a break.

    What do you actually check at the end of each session?

    The end-of-day tie-out

    • Position, per instrument, in your book versus the broker's — including flat, which must be flat in both.
    • Every fill in your log matched to a trade in the broker's trade book, and every trade matched back. Both directions, because an unmatched broker trade is worse than an unmatched log line.
    • Average price per instrument, reconciled to the contract note rather than to your own arithmetic.
    • Charges — brokerage, exchange charges, GST, STT or CTT, stamp duty, SEBI fees — against what you modelled. This is where a strategy quietly stops being viable.
    • Any intraday product that was squared off by the broker rather than by your exit rule, listed explicitly. Your system did not make those trades and must not treat them as if it did.
    • Every reject and every error line from the session, each with a cause you can name.
    • Whether any limit was hit, and whether the halt chain behaved as designed if one was.
    • The date the kill switch was last tested, restated, so it cannot silently go stale.

    Pro tip — Automate the comparison and read the exceptions, not the matches. A reconciliation you have to do by hand is one you will skip on the exact evening it mattered.

    What are the failure modes that catch people out?

    The failureWhat it looks likeWhat catches it
    Stalled feedPrices present, unchanged. Rules keep firing on a frozen pictureData-freshness check against the wall clock, not the tick's timestamp
    Duplicate systemTwo copies running after a restart, doubling every positionA single-instance lock, plus the position-versus-expected check
    Retry stormOne rejected order resent hundreds of times in secondsA rolling reject counter that stops sending, and an order-rate cap
    Silent partial fillOrder treated as complete; the book is wrong from that momentPosition built from the running sum of fills, reconciled continuously
    Unexpected square-offThe broker closed an intraday position your system still believes is openThe position-versus-expected check, and the end-of-day tie-out
    Clock driftBars and orders timestamped wrongly; reconciliation stops making senseTime synchronisation on the machine, checked as part of startup
    Expired credentialsEverything looks healthy until the first order of the day is refusedA startup self-test that authenticates before the market opens
    The alerter itself dyingTotal silence, read as 'nothing is wrong'The daily heartbeat message whose absence is the alert

    Where does the Indian session calendar bite?

    The operational layer is where market structure stops being an abstraction.

    The pre-open session runs before continuous trading begins, and prices there behave differently — a system that starts polling into it can act on a picture that is not yet a market. The close arrives on a schedule your code must know about, and intraday products are squared off by your broker on their timetable, not yours, at whatever price is available. Expiry days change liquidity in derivatives sharply. Trading holidays, special sessions and truncated days all need to be in a calendar your system reads, because a system that does not know the market is closed will spend the day alerting you about a stalled feed.

    None of this is exotic. All of it has ended sessions badly for people who assumed the session was a rectangle from 09:15 to 15:30.

    What about the regulatory side?

    Running automated orders into NSE or BSE through a broker sits inside a framework set by SEBI and administered by the exchanges and your broker — covering registration and identification of retail algorithms, what your broker must do before your orders reach the exchange, and the controls they are required to have on their side.

    That framework has been revised more than once and continues to be, so the specifics are not something to take from any article. Confirm what applies to your setup, in writing, with your broker, and read the current circulars on sebi.gov.in and your exchange's site. It is also worth asking your broker directly what kill-switch and risk controls they enforce at their end — that is a layer you get for free and should design around rather than duplicate.

    Note — Nothing here states a current regulatory requirement as fact. Verify the position that applies to you with your broker, SEBI and your exchange before placing an automated order.

    What invalidates your operational setup?

    If any of these is true, the system should not be trading — regardless of how the strategy has been doing.

    • The kill switch has never been fired for real. Untested means unproven, and there is no partial credit.
    • The kill switch stops new orders but cannot flatten what is already open.
    • Any control lives inside the trading loop it is supposed to be able to stop.
    • A monitored signal has no agreed action attached to it.
    • The alert path shares its dependency with the trading connection, so one outage removes both.
    • The log cannot reconstruct a decision from last week without opening the code.
    • Position and expected position have disagreed at any point and the cause was never found.
    • The end-of-day reconciliation was skipped, or a break was carried into the next session.
    • The system auto-restarts after a halt without a human recording why it is safe.
    • You have no way to flatten positions if the machine is gone.

    Key points

    Once orders go out without you, you are an operator — build the controls before the strategy trades.
    Monitor five things: heartbeat, data freshness, reject rate, position vs expected, PnL vs modelled.
    A stalled feed is more dangerous than a disconnection — check tick age against the wall clock.
    Position must equal the running sum of fills, continuously. Any mismatch halts and flattens.
    A metric with no agreed action is a chart, not a control.
    Local machine or rented server is a choice about which failure you can live with, not about latency.
    Log the inputs to each decision, not just the outcome — including the limit checks that passed.
    Critical alerts must not depend on the broken process, or share a path with the trading connection.
    A kill switch stops new orders, cancels resting ones, flattens, and stays down until a human restarts it.
    It must live outside the strategy and be reachable from a phone.
    Reconcile three ways every session — your book, the broker's, the contract note — before the next open.
    The kill switch you have never tested is not a kill switch.

    Pro tip — Put the date your kill switch was last fired for real at the top of your daily reconciliation message, so you read it every single session. Anything that is not in front of you goes stale silently, and this is the one control whose staleness you only discover on the day it was supposed to save you.

    Frequently asked questions

    What exactly is a kill switch in algo trading?

    It is a control that stops new orders, cancels every resting order, closes what is already open, and then keeps the system down until a person restarts it. Killing the process is not a kill switch — resting orders in the exchange book and open positions are entirely unaffected by your terminal closing. It also has to live outside the strategy, because a check inside the trading loop cannot be relied on when that loop is the thing misbehaving.

    Do I need a VPS to run an algo trading system in India?

    Not necessarily, and the choice is rarely about latency at retail horizons. A rented remote server removes your home's power and broadband from the failure list and keeps running when you are away, at the cost of a monthly bill, remote-only debugging and a dependency on someone else's uptime. Your own machine is free and immediate to debug but fails with your building. Decide on which failure you can live with mid-session, and note that neither option removes the need for monitoring and a tested halt.

    What should a trading system log?

    Decisions and their inputs, not just outcomes. Each line needs a millisecond timestamp from a synchronised clock, the data the decision saw and how stale it was, the rule that fired and the comparison that made it true, the state at that moment — position, pending orders, limits used — the exact order payload with your own client id, every broker response including rejects verbatim, and every fill as its own entry. Log the limit checks that passed too, or you cannot prove they were running.

    How do I know if my algo has stopped working during the session?

    A heartbeat written by the trading loop itself to somewhere outside the process, checked by a separate watcher, with a missed-beat threshold that triggers the halt chain automatically. Pair it with a data-freshness check against the wall clock, because a stalled feed is more dangerous than a disconnection — prices are still there, they have simply stopped changing, and every rule keeps firing on a frozen market.

    How often should I test my kill switch?

    On a schedule you keep, and after every change to the system, its credentials or its dependencies. Fire it deliberately during live hours with a position small enough that the test costs nothing but time, measure the actual seconds from decision to flat, and record the date somewhere you see regularly. A switch last tested before three deployments ago is a switch whose current state you are guessing at.

    What is end-of-day reconciliation and why does it matter?

    It is a three-way tie-out between your system's own book, the broker's reported orders and trades, and the contract note — matched on quantity, price and charges, in both directions, before the next session opens. It matters because every risk calculation the system makes tomorrow assumes today's book is correct. A break of one share means the book is wrong, and an unexplained break should stop the system rather than roll into the next day.

    Are there SEBI rules for retail algo trading in India?

    There is a framework covering how automated orders reach NSE and BSE through a broker, including registration and identification of retail algorithms and the controls brokers must enforce. It has been revised more than once, so treat no article — including this one — as current. Ask your broker in writing what applies to your setup and read the live circulars on sebi.gov.in and your exchange's site.