Who is Shakti Tiwari?

Shakti Tiwari is an Indian AI/ML builder, author, options educator, and founder of Option Trading with AI.

Inside Our NSE F&O ML Engine: Live Dhan API Data Meets an EOD-Audited, Leakage-Safe Blueprint

Q: Is this financial advice?

No. This content is educational only, not SEBI-registered investment advice. NISM-Series-XII certified; not a SEBI-registered Research Analyst.

Quick Answer

The OptionTradingWithAI.in NSE engine runs in **two connected layers**. The live layer connects to the **Dhan WebSocket API** (Full market mode) to capture real-time ticker, quote and option-chain data, and runs the trained model in **shadow mode** to generate signals without sending broker orders. The ML layer — the focus of this article — is **deterministic and EOD-audited**: the model is trained and validated on official NSE **UDiFF bhavcopy** files for full reproducibility. Its core design choices are: predict the **underlying's next-day direction (UP/NEUTRAL/DOWN)**, use **one feature function for both training and prediction** (no train/serve skew), validate only with **date-based walk-forward + embargo**, reconstruct a **point-in-time universe** to kill survivorship bias, and treat every feature as guilty-until-proven by **multiple out-of-sample windows**. The biggest lesson: complexity is not the enemy — *untested semantic drift* is. This article documents the architecture so a retail builder can audit every step.

**Framing (research-governor applied):** This is a research/engineering walkthrough of our own blueprint, not a guarantee of trading profit. The source document itself states it is "a research/engineering blueprint, not a guarantee of trading profit or zero software defects." No promotion gate has been passed; claims are bounded to method and design, never to profitability.

Why This Matters

Most retail "AI trading" content is two things at once: too complex to audit, and too shallow to trust. The NSE EOD blueprint we use at OptionTradingWithAI.in takes the opposite bet — **data rich, code small, prediction path deterministic.**

For the Indian retail trader who actually wants to *understand* what an ML options/stock model does (rather than rent a black box), three facts matter:

1. **The ML training layer is EOD-first for auditability — the live layer is API-driven.** Our production engine connects to the Dhan WebSocket API for real-time data; the model behind it is trained/validated on NSE's free EOD bhavcopy files so every feature and label is reproducible. NSE's EOD reports contain the futures, options-chain, cash-market and delivery data needed to engineer real market-mechanics features.

2. **The traps are boring, not exotic.** Train/serve skew, random-CV leakage, survivorship bias, and silent null-drift destroy more models than bad maths does.

3. **An auditable 5-module engine beats a 200-file monster** you cannot explain to yourself, let alone to a regulator or a skeptical reader.

This is also where SEO intent meets genuine value: searchers typing *"NSE machine learning options"*, *"XGBoost options trading India"*, or *"walk-forward validation python"* are not looking for hype — they want a reproducible design they can check. That is exactly what follows.

The Live Layer: Dhan WebSocket API (How the Engine Actually Runs)

The blueprint quoted above describes the **ML training and backtest data design** — deliberately EOD/UDiFF for determinism and auditability. The **production engine is not EOD-only**. It connects to the **Dhan WebSocket API** in Full market mode to stream live ticker, quote and option-chain data, then runs the trained XGBoost/LightGBM model in **shadow mode** (the `live-shadow-observer` pattern) to produce directional signals **without placing broker orders**.

**OBSERVED (our stack):** Dhan's Ticker/Quote/Full WebSocket modes feed the live capture; the same `make_features()` function used in training runs on the live snapshot, so there is no train/serve skew at inference either.

**Why EOD for training, API for live:** EOD bhavcopy gives a clean, final, point-in-time record ideal for leakage-safe walk-forward; the live API gives the freshness needed for next-day signal generation. The two layers are complementary, not contradictory.

**Safety:** shadow mode means the model's predictions are recorded and evaluated against outcomes before any order authority is considered — exactly the promotion-gate discipline the blueprint enforces.

Research Question / Hypothesis

This article documents a practical, grounded question — *can a retail-grade NSE prediction engine be designed so that its biggest failure modes (leakage, survivorship, train/serve skew, null drift) are prevented by construction rather than patched after the fact?* — not a "the model predicts the market" claim.

Every non-obvious statement below is labeled:

**SOURCE** = from our NSE EOD ML Blueprint document (2026-08-20 research snapshot).

**OBSERVED** = behaviour seen in real EOD workflows.

**DERIVED** = computed from the above.

Nothing is invented. Where the engine has not been trained, we say so.

Data & Methodology Box

**SOURCE (format):** NSE discontinued the old common bhavcopy CSV formats from **8 July 2024** in favour of **UDiFF Common Bhavcopy Final**. The 2026 format workbook includes fields such as actual expiry, strike, option type, underlying price, settlement price, OI, change-in-OI, traded quantity/value, trades and lot size.

**SOURCE (layers):** F&O UDiFF Common Bhavcopy Final (canonical derivatives source); CM UDiFF + delivery/corporate-action files (underlying context); participant/FII statistics (market context, where granularity supports it).

**SOURCE (target):** V1 target = next-day underlying close direction via `r_next = log(close[T+1]/close[T])`, bucketed UP / NEUTRAL / DOWN by an ATR-normalized band `k * ATR_pct[T]`.

**SOURCE (validation):** date-based walk-forward with gap/embargo; a final untouched holdout the tuner never sees.

**DERIVED (design rule):** a feature stays ON only if it adds value over baseline across **multiple out-of-sample windows**; "interesting" ≠ "useful."

**Method:** manual daily ingestion → one canonical schema → shared `make_features()` for train and predict → walk-forward with negative controls (shuffled labels, lag-scrambled features, dummy majority, price-only baseline, feature-group ablation).

**Costs:** not yet modeled in the blueprint's V1; any trading decision layer must include STT/brokerage before claiming viability.

Results (Design Decisions and What They Buy You)

Because the blueprint is an *architecture*, its "results" are the decisions that survive the blueprint's own challenge tests. Each is a defensible design outcome, not a backtest win.

Table 1 — Core architecture decisions

| Decision | What it prevents | Why it matters |

|---|---|---|

| One `make_features()` for train + predict | Train/serve skew | Backtest-good / inference-bad gap disappears |

| Date-based walk-forward + embargo | Random-CV leakage | No future leaking into past |

| Point-in-time universe (by contract master) | Survivorship bias | Today's F&O list ≠ history's list |

| Reason-coded null policy | Silent row loss | Source/join failures surfaced, not hidden |

| Feature gates + ablation manifest | Overfit / audit failure | Each feature must earn its place |

| Underlying-date as unit | Contract churn leakage | Model learns the persistent entity |

| UDiFF-first EOD training data | Partial/live drift in training | Model trains only on final, validated EOD files (live inference uses Dhan API) |

Findings (from the blueprint's own logic)

1. **SOURCE:** The primary prediction unit is `underlying-date` (e.g., NIFTY, RELIANCE, SBIN), with futures/options-chain mechanics as *predictors*. Options expire and roll; the underlying persists. Modeling the underlying direction is stabler and avoids contract-identifier leakage.

2. **SOURCE:** XGBoost and LightGBM are trained on **identical** rows, columns, labels, sample weights, folds and scoring. Only hyperparameter spaces differ. The winner is chosen by **repeated walk-forward stability + calibration + MCC/Balanced Accuracy**, not one lucky accuracy number.

3. **SOURCE:** Optuna runs in *suggestion/competition* mode — objective = mean walk-forward MCC minus stability/calibration/failure penalties. The best trial is a **candidate**; a manual "promote" step is required. The model never auto-promotes.

4. **OBSERVED:** Simple EOD systems out-survive complex live ones because data semantics stay stable — same close, same OI, same schema, same feature timing, same target timing.

5. **DERIVED:** The competition score `mean(MCC_folds) − λ_std·std(MCC) − λ_cal·cal_penalty − λ_fail·fail_penalty` rewards *stable* models over flashy single-fold winners.

Feature Catalog (Selected) — What the Engine Actually Reads

A big part of auditability is *naming* every candidate feature with its formula, source, tier and leakage test. The blueprint generates a Feature Manifest so nothing hides. Below is a representative slice across the four mechanic families (**SOURCE**). Tiers: Core (default ON after gating), High (context-rich), Extended/Experimental (must beat baseline to stay).

Table 2 — Selected candidate features by family

|---|---|---|---|

**Key interaction features (SOURCE):** moneyness × TTE, OI-change × moneyness, PCR × volatility-regime, basis × TTE, ATM-OI-imbalance × underlying momentum. These cross-products are where mechanics often beat a price-only baseline — *if* they survive ablation.

**OBSERVED caveat from the blueprint:** the "long buildup / short covering / long unwinding" classification from price+OI signs is a *heuristic*, not ground truth about trader positioning. We do not label it as actual positioning.

Why UDiFF-First (Not Filename-First)

NSE is streamlining bhavcopy dissemination and also exposing DAT-format bhavcopy via Extranet paths. A robust parser therefore **validates file content and schema version** rather than trusting the filename. The payoff (SOURCE): historical backfill can span the 8-July-2024 format change without rewriting feature code, because every raw format maps to one canonical schema. That is the whole point of "data rich, code small" — the data layer absorbs change so the model layer stays stable.

Reproducibility (The 5-Module Skeleton)

The blueprint keeps business logic in **five small modules** (roughly 200–500 readable lines each) plus config, tests and an app shell. An auditor should understand the whole prediction path without opening dozens of files.


project/
  config.yaml      # sources, feature toggles, label, model params, Optuna spaces
  data.py          # download/load, legacy+UDiFF normalize, joins, validation, quality report
  features.py      # ALL deterministic feature formulas + label + manifest generation
  models.py        # XGBoost, LightGBM, Optuna, save/load model bundles
  evaluate.py      # date walk-forward, competition, calibration, ablation, audit metrics
  app.py           # manual commands/UI: ingest, build, train, compare, promote, predict
  tests.py         # schema/null/leakage/reproducibility tests
  README.md        # operating procedure + audit map

**Canonical keys (SOURCE):** every source format maps to one internal schema.


canonical_contract_key = (segment, instrument_type, symbol, expiry_actual, strike_or_0, option_type_or_XX)
canonical_row_key      = (trade_date,) + canonical_contract_key

**Sample feature (SOURCE — options-chain PCR, Core tier):**


pcr_oi = total_Put_OI / total_Call_OI          # across the underlying's option chain

Moneyness is signed percentage, not raw rupee distance — because ₹100 means different things for a ₹500 stock and a 25,000 index:


d_pct       = (K - S) / S
signed_mny  = -d_pct for CE   # positive = ITM
signed_mny  =  d_pct for PE   # positive = ITM

**One row per underlying-date (SOURCE):** aggregate the contract universe (many FO rows) + the CM underlying row + market context into a single feature row, then label from underlying T+1. This sharply reduces complexity and stops the model learning strike quirks that vanish next month.

Leakage-Safe Walk-Forward: A Concrete Fold Plan

The blueprint specifies a **date-based** splitter (Scikit-learn's `TimeSeriesSplit` supports ordered data with a gap; here it is stricter because many symbols share a date and all rows for one date must stay on the same side). A concrete scheme (**SOURCE**):

Table 3 — Example walk-forward schedule

| Fold | Train window | Gap / test |

|---|---|---|

| 1 | 2019 .. 2022 | TEST 2023 H1 |

| 2 | 2019 .. 2023 H1 | TEST 2023 H2 |

| 3 | 2019 .. 2023 | TEST 2024 H1 |

| … | slide forward | … |

| Final | — | untouched holdout (most recent block, Optuna never sees it) |

Rules that make it honest:

**Split by DATE**, never by random row. All underlyings on a date belong to the same side.

**Gap/embargo ≥ label horizon.** If features/labels use longer forward windows, increase the gap.

Any **probability calibrator** is fit on data disjoint from the estimator's own fit data.

Keep a **final holdout** the tuner never touches — this is where real generalization is measured.

**Metrics reported per fold (SOURCE):** Balanced Accuracy (imbalance-tolerant headline), MCC (multiclass quality; 0 ≈ no useful correlation), Macro-F1, Log Loss, Brier/calibration curve, coverage-by-confidence, and **per-year / per-regime** metrics to catch instability a single overall average hides. A model that wins one fold and dies in three is not a model.

What Failed / Counter-Evidence (Negative Controls)

The blueprint bakes in honesty tests so a lucky backtest cannot masquerade as edge:

**Shuffled labels** → should collapse toward chance. If they don't, something is leaking.

**Lag-scrambled features** → should lose signal.

**Dummy majority classifier** and **price-only baseline** → the floor to beat.

**Feature-group ablation** → remove futures, options, delivery, and participant context *one group at a time*. **SOURCE rule:** if the "full mechanics" model does not *repeatably* beat the simple price/OI baseline, drop the mechanics. Complexity must earn its place; it gets no free pass.

**OBSERVED trap:** a trader (or an article) reporting one great backtest is reporting a coincidence until it survives multiple regimes net-of-cost. The negative controls are the difference between "I tested" and "I know."

Limitations

This is a **research/engineering blueprint**, not a deployed, promoted trading system. The source document explicitly disclaims guaranteed profit or zero defects.

**Non-stationarity is the rule.** A feature that worked last quarter can decay this quarter; validate out-of-sample and shadow-run before any live action.

**Costs are not in V1.** Any strategy layer must subtract STT/brokerage; gross edge is not net edge.

**IV/Greeks are Experimental**, not core. The blueprint deliberately avoids depending on scraping a dynamic web option chain; if IV is added later it must be computed from validated EOD prices with rejection of impossible quotes, and must beat the non-IV baseline out-of-sample before promotion.

**No live promotion gate passed** here. Per the research-governor, claims are bounded to method and design, never to profitability.

If any figure here conflicts with an official NSE/SEBI source, the official source wins — verify before you act.

Practical Takeaways

1. **Predict the underlying, not the option.** Use `underlying-date` as the unit; let chain mechanics be predictors.

2. **One feature function, everywhere.** Train and predict with the same `make_features()`. This single rule kills the most common silent failure.

3. **Walk-forward or nothing.** Split by date with an embargo; keep a final holdout the tuner never sees.

4. **Reconstruct the point-in-time universe** from the contract/security master. Today's F&O list is not history's list.

5. **Classify nulls; don't drop them.** Structural / warm-up / source-missing / join-missing / invalid are different states with different actions.

6. **Gate every feature.** Ablate it; if it doesn't repeatably beat baseline, it's off.

7. **Manual control stays.** Optuna suggests; a human promotes. The model never auto-ships.

8. **Go deeper in the books.** The full pipeline — feature code, walk-forward harness, and live Dhan WebSocket integration — is expanded in the author's published works, *Options Trading with AI, Vol 1 & Vol 2* (linked below).

FAQ

**Q: Is this the actual engine behind OptionTradingWithAI.in?**

A: Yes — this documents our two-layer NSE F&O ML engine: a live **Dhan WebSocket API** feed for real-time shadow-mode signals, plus an **EOD/UDiFF-audited** XGBoost-vs-LightGBM core trained with point-in-time data and walk-forward validation. Published as research, not as a live trading product.

**Q: Does the engine use a live API or only EOD files?**

A: Both, by design. The ML model is trained and validated on NSE's free EOD UDiFF bhavcopy files (for auditability and leakage-safe walk-forward). The production engine then connects to the **Dhan WebSocket API** to capture live market data and run the trained model in shadow mode for next-day signals. EOD is the training substrate; the API is the live nervous system. (The blueprint's "EOD-first" wording refers to the *training/backtest data layer*, not the whole engine.)

**Q: Why XGBoost and LightGBM, not deep learning?**

A: Neither brand "wins" by default. The blueprint picks the winner by repeated walk-forward stability + calibration + MCC/Balanced Accuracy on identical folds. For tabular market data, gradient-boosted trees are the defensible default; deep nets must earn their place like any other feature group.

**Q: What is the single most common leakage mistake?**

A: Random train/test split on time-series data, or letting a future row touch a past feature window. Split by date, embargo the gap, keep all underlyings on a date on the same side of the split.

**Q: How do you avoid survivorship bias?**

A: Reconstruct the training universe by date from actual contracts/security master. Do not apply today's F&O eligibility list to 2019. Use actual expiry, not a hard-coded weekday rule.

**Q: Should I trade live on this blueprint?**

A: Not on the basis of this article. It is a research design. Any live use requires promotion gates, cost adjustment, negative-control passes, and human approval the blueprint describes — none claimed here.

TL;DR

Our NSE engine has **two layers**: a live **Dhan WebSocket API** feed for real-time data + shadow-mode signals, and an **EOD/UDiFF-audited ML core** for reproducible training and walk-forward validation.

Predict **underlying next-day direction**; use **one shared feature function**; validate with **date walk-forward + embargo**.

**Point-in-time universe** + **reason-coded nulls** kill survivorship and silent data loss.

**XGBoost vs LightGBM** on identical folds; winner by stability + MCC, not one accuracy.

Every feature is **gated**; negative controls must pass or complexity is dropped.

This is research documentation — not a profitability claim, not live advice.

Sources

NSE EOD ML Blueprint (Deep Research), research snapshot 20 August 2026 — OptionTradingWithAI.in internal document (SOURCE for all architecture claims above).

NSE UDiFF Common Bhavcopy Final format (official, effective 8 July 2024) — field schema for FO/CM layers.

Scikit-learn `TimeSeriesSplit` documentation — ordered split with gap (validates the date-based embargo design).

Author / Canonical Attribution

*Written by Shakti Tiwari — Nifty Option Trader, XGBoost Expert, founder of OptionTradingWithAI.in. NISM XII certified options educator (not a SEBI Registered Analyst). Educational content only; not investment advice. Verify all data against official NSE sources before acting.*

Resources & Links

Our NSE Option Chain explainer (OI, PCR, Greeks): https://dev.to/shaktitiwari/nse-option-chain-explained-oi-pcr-and-greeks-that-actually-matter-for-nifty-traders-4dfl

Option Selling Strategies in India: https://dev.to/shaktitiwari/option-selling-strategies-in-india-how-retail-traders-sell-premium-without-getting-wrecked-g36

How to Build a Simple NSE EOD ML Model Manually (no API): https://dev.to/shaktitiwari/how-to-build-a-simple-nse-eod-optionsstock-ml-model-manually-no-live-api-no-wizardry-kkp

About the author: https://about.me/shaktitiwari

Main site: https://optiontradingwithai.in

WhatsApp: https://wa.me/919169650895

*Books by the author:*

*Options Trading with AI — Vol 1* — https://www.amazon.in/dp/B0H9ZNTBPK

*Options Trading with AI — Vol 2* — https://www.amazon.in/dp/B0HBBFKDQF

Educational only — not SEBI-registered investment advice. NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.