Who is Shakti Tiwari?

Shakti Tiwari is an Indian AI/ML builder, author, options educator, and founder of Option Trading with AI.

How to Build a Simple NSE EOD Options/Stock ML Model Manually (No Live API, No Wizardry)

Q: Is this financial advice?

No. This content is educational only, not SEBI-registered investment advice. NISM-Series-XII certified; not a SEBI-registered Research Analyst.

**Framing (research-governor applied):** This is a research/educational walkthrough, *not* trading advice and *not* a claim that any model predicts the market. The blueprint this article is based on states plainly it is "a research/engineering blueprint, not a guarantee of trading profit or zero software defects." Nothing here has passed a production promotion gate. Build it to learn the *discipline*, not to mint money.

Executive Summary

You do **not** need a live WebSocket, a co-located server, or a PhD to start learning how institutional-style prediction on NSE data actually works. The single most robust starting point is the one the pros keep coming back to: **end-of-day (EOD) data, downloaded once per day, processed with simple, auditable logic.**

This article shows a manual, no-API pipeline:

1. Pull NSE's free **EOD bhavcopy** files (the official "Common Bhavcopy Final" in UDiFF format).

2. Normalise them into one clean table.

3. Engineer a handful of honest market-mechanics features (PCR, OI change, futures basis, delivery %).

4. Define a simple target: *will the underlying go UP / NEUTRAL / DOWN tomorrow?*

5. Train a tiny model you fully control (Excel, Google Sheets, or 40 lines of Python).

6. Validate with **walk-forward** — the only honest test — not a random train/test split.

The whole point of "manual and simple" is that *you can see and defend every step*. Complexity that you cannot explain is a liability, not an edge.

Research Question / Hypothesis

This article tests a practical, grounded question — *can a retail builder replicate the core of an EOD prediction workflow using only free official NSE files and manual/light automation, while avoiding the classic traps (data leakage, survivorship bias, train/serve skew)?* — not a "predict the market" claim.

Claims are labeled:

**SOURCE** = verified from the blueprint document / NSE official material.

**OBSERVED** = behaviour seen in real EOD workflows.

**DERIVED** = computed from the above.

Nothing is invented.

Data & Methodology Box

**SOURCE (NSE format change):** NSE discontinued the old common bhavcopy CSV formats from **8 July 2024** in favour of **UDiFF Common Bhavcopy Final**. The 2026 format workbook includes fields such as actual expiry, strike, option type, underlying price, settlement price, OI, change-in-OI, traded quantity/value, trades and lot size (per the blueprint document provided).

**SOURCE (data layers):** F&O UDiFF Common Bhavcopy Final is the canonical derivatives source; CM UDiFF + delivery/corporate-action files support the underlying context (blueprint §3).

**OBSERVED:** Simple EOD systems often survive longer than complex live ones because the *data semantics stay stable* — same close, same OI, same schema, same feature timing, same target timing.

**DERIVED (design rule):** a feature earns its place only if it adds value across *multiple out-of-sample windows*; "interesting" ≠ "useful."

**Method:** manual daily download → canonical table → feature function shared by training and prediction (no train/serve skew) → walk-forward evaluation with an embargo gap.

Why "Simple EOD" Beats "Complex Live" (for Learners)

The blueprint's section 2 is the best argument for starting manual. Complexity itself is not the enemy — **untested semantic drift is.** When you bolt on a live/API layer, silent mismatches creep in:

Timestamps that mean different things in live vs EOD.

Contract rollover handled wrong (near vs next expiry joined incorrectly).

Missing values, partial bars, revised expiries, duplicate rows.

A manual EOD process kills most of these by construction: you work only with **final** files, once per day, after the exchange has published them. You trade a little freshness for a *lot* of cleanliness. For a learner, that trade is almost always worth it.

**OBSERVED trap — train/serve skew:** a backtest looks great, but daily inference is terrible, because the feature code differed between training and live. The fix is boring: use the *same* `make_features()` function for both. In a manual setup, that means the Excel formula or Python function you use on history is the exact one you run tomorrow.

Step 1 — Get the Data Free (Manual Download)

You do not need an API key to start. NSE publishes EOD reports.

**FO core:** F&O UDiFF Common Bhavcopy Final — contract OHLC/LTP, expiry, strike, option type, underlying, OI, change-in-OI, volume/value, trades, lot. **(SOURCE: blueprint §3, MUST layer)**

**CM core:** CM UDiFF Common Bhavcopy Final — underlying OHLC/volume/value/trades. **(SOURCE, MUST)**

**CM delivery:** Security-wise Delivery Positions — delivery % as a genuine interest proxy. **(SOURCE, HIGH)**

**Corporate actions / security master:** splits, bonuses, dividends, symbol lifecycle. **(SOURCE, MUST for stocks)**

**Manual workflow:** each evening (or next morning), download the FO and CM final files for the previous session, drop them in a dated folder (`data/2026-08-20/`), and record the file hash. That hash is your integrity receipt — if the file ever changes or truncates, you will know.

**SOURCE note:** NSE is streamlining bhavcopy dissemination and also making DAT-format bhavcopy available through Extranet paths. A robust manual parser should validate *file content/schema version* rather than trusting the filename. For a learner, just confirm the columns match your expected schema before ingestion.

Step 2 — Build One Canonical Table

Raw files are immutable. Every source format maps to **one internal schema** so your historical backfill survives format changes without rewriting feature code.

Minimal canonical fields you actually need (DERIVED from blueprint §4):

| Field | Meaning | Applies to |

|---|---|---|

| `trade_date` | Trading date | all |

| `symbol` | Underlying/security | all |

| `instrument_type` | FUTIDX/FUTSTK/OPTIDX/OPTSTK | all |

| `expiry_actual` | Revised/actual expiry | derivatives |

| `strike` / `option_type` | Strike / CE-PE | options |

| `close` / `settlement_price` | EOD price | all |

| `open_interest` / `change_oi` | End-of-day OI / ΔOI | FO |

| `traded_qty` / `traded_value` | Volume | all |

| `market_lot` | Lot size | all |

A **canonical contract key** = `(segment, instrument_type, symbol, expiry_actual, strike_or_0, option_type_or_XX)`. A **canonical row key** = `(trade_date) + contract_key`. These two keys prevent the "duplicate row" and "wrong expiry join" failures.

In Excel this is just a consolidated sheet with those columns. In Python it is one DataFrame. Keep it dumb and explicit.

Step 3 — Engineer a Few Honest Features

Do not engineer 400 features on day one. Engineer five you can explain in a sentence each. Market-mechanics features the blueprint highlights (SOURCE):

1. **PCR (Put-Call Ratio)** — total put OI ÷ total call OI for the underlying's option chain. *DERIVED.* Crowded fear (high) is a contrarian hint **only with trend context**.

2. **Change-in-OI regime** — is OI building or unwinding at key strikes? *OBSERVED:* rising OI at a strike is a positioning proxy, not a hard wall.

3. **Futures basis / term structure** — spot vs future, near vs next expiry. *SOURCE:* futures basis/term-structure/roll are core mechanics predictors.

4. **Delivery %** — from CM delivery file. *SOURCE:* a genuine underlying-interest proxy, free of option-chain noise.

5. **Moneyness / TTE buckets** — is OI concentrated ATM or far OTM? *DERIVED:* OI concentration and strike-wall distances matter.

**The anti-complexity rule (SOURCE):** a feature stays ON only if it shows incremental value over the baseline across *multiple out-of-sample windows*. An "interesting" feature is not a "useful" feature. A manual builder enforces this by ablation: remove the feature, re-run walk-forward, and check whether the score actually dropped.

Step 4 — Define the Target (Honestly)

Primary target = **T+1 underlying direction**, preferably **3-class: UP / NEUTRAL / DOWN** (SOURCE: blueprint §1). A regression target (expected return) is optional later.

Compute next-day return from the underlying close.

Bucket: UP if return > +threshold, DOWN if < −threshold, NEUTRAL between.

The threshold is a *modelling choice*, not a discovered truth — state it and keep it fixed across all tests.

**Why underlying-date, not option contract?** (SOURCE) The primary prediction unit is `underlying-date` (e.g., NIFTY, RELIANCE, SBIN), with futures/options-chain mechanics as *predictors*. Options expire; the underlying persists. Modelling the underlying direction is stabler and avoids contract-churn leakage.

Step 5 — Train a Tiny Model You Control

"Model" can be laughably small to start:

**Excel / Google Sheets:** a logistic-style or linear score on your 5 features. You can literally compute `score = w1·PCR + w2·ΔOI + ...` with hand-set weights and see the confusion matrix by hand.

**Python (light):** `sklearn` or `xgboost` with **manual** parameter control. The blueprint recommends XGBoost **and** LightGBM on identical folds/features and picking the winner by *repeated walk-forward stability + calibration + MCC/Balanced Accuracy* — not one lucky accuracy number.

**Optuna = suggestion, not authority (SOURCE):** use it in competition/suggestion mode if you like, but keep manual parameter control available. A learner should be able to name every weight.

Step 6 — Walk-Forward Validation (The Only Honest Test)

This is where manual discipline pays off. **Never** use a random train/test split on time-series data — it leaks the future into the past and gives you fake-high scores (SOURCE: "Random CV leakage → unrealistically high score").

Manual walk-forward:

1. Sort by date.

2. Train on window `[start, t]`, test on `[t+1, t+gap]`.

3. Slide `t` forward; repeat.

4. **Embargo/gap** between train and test so yesterday's features don't leak into today's label.

5. Report **Balanced Accuracy** and **MCC** across *all* folds — and whether the model stays stable across regimes (trending, choppy, event weeks).

A model that wins one fold and dies in three is not a model; it is a coincidence.

Step 7 — Point-in-Time & Null Discipline

Two traps kill manual projects silently:

**Survivorship bias (SOURCE):** do not apply *today's* F&O list to *history*. NSE's F&O eligibility changes month to month. Your training universe on date T must be "symbols that actually had valid eligible FO contracts on T." Reconstruct the universe by date from the contract/security master.

**Null policy (SOURCE):** "drop NaN" is not a strategy. Classify the null first:

`STRUCTURAL_NULL` — field doesn't apply (e.g., strike on a future). Keep as structural; never impute zero.

`WARMUP_NULL` — rolling indicator lacks history. Exclude until minimum history met.

`SOURCE_MISSING` / `JOIN_MISSING` — data-quality failures, not model features.

`INVALID` — impossible value; quarantine and report.

Zero is **not** a generic missing sentinel — zero OI change can be a real value (OBSERVED).

Worked Example (Manual, on NIFTY)

Imagine you track NIFTY only, one row per day, from the FO + CM final files:

Day 1–60: you collect close, settlement, total put OI, total call OI, NIFTY futures basis, and NIFTY delivery %.

You compute PCR = put OI ÷ call OI each day (DERIVED).

You label T+1: UP if next-day NIFTY return > +0.3%, DOWN if < −0.3%, else NEUTRAL.

You train a 5-feature score on days 1–40, test on 41–45 (with a 1-day embargo), then slide.

Across 12 such folds you find Balanced Accuracy ≈ 0.52 in trending windows and ≈ 0.47 in choppy ones. **Interpretation (DERIVED, honest):** the naive feature set is roughly a coin-flip once costs are considered — which is *exactly* the lesson. The workflow is correct; the edge is not yet there. You now have a reproducible harness to test *real* features, instead of a backtest that lied to you.

That negative result is worth more than a fake 85% accuracy from a random split.

Common Myths

**"I need a live API to do real ML."** No. EOD final files are cleaner and free; live data mainly adds freshness *and* silent drift risk. Start EOD.

**"More features = better model."** Only if they survive multiple out-of-sample windows. Otherwise they are overfit (SOURCE anti-complexity rule).

**"High backtest accuracy means it works."** Not if the split leaked the future. Walk-forward or it didn't happen (SOURCE).

**"Dropping NaN rows is fine."** It hides data-quality problems and can delete valid signal. Classify nulls instead (SOURCE).

**"I can use today's stock list for history."** That's survivorship bias; it inflates past performance (SOURCE).

FAQ

**Q: Is this legal / allowed for a retail trader?**

A: Downloading NSE's public EOD files for personal research/education is standard. Respect NSE's terms and don't redistribute the raw data. This article is educational, not a licensed research service.

**Q: Do I need to know Python?**

A: No — Excel/Google Sheets can do steps 2–6 with formulas and a hand-built confusion matrix. Python just scales it and makes walk-forward easier.

**Q: Which is better, XGBoost or LightGBM?**

A: Neither is "better" by brand. The blueprint picks the winner by *repeated walk-forward stability + calibration + MCC/Balanced Accuracy* on identical folds. As a learner, either is fine; consistency of method matters more than the library.

**Q: How do I avoid data leakage in a manual setup?**

A: Use walk-forward with a date embargo, define the target from *future* close only, and use the same feature function for history and tomorrow. Never let a future row touch a past feature window.

**Q: What is the single biggest beginner mistake?**

A: Random train/test split on time-series data, producing a fake-high score and a false sense of edge. Walk-forward or nothing.

**Q: Should I trade live on this?**

A: Not on the basis of this article. This is research discipline, not a production system. Any live use needs promotion gates, cost adjustment, and human approval the blueprint describes — none of which are claimed here.

Practical Takeaways

1. Start with **EOD final files**, not live APIs.

2. Normalise everything into **one canonical table** with explicit keys.

3. Engineer **five explainable features**, not 400.

4. Target **T+1 direction (3-class)** on the **underlying**, not the option.

5. Validate only with **walk-forward + embargo**; report Balanced Accuracy and MCC.

6. Reconstruct the **point-in-time universe**; classify **nulls** instead of dropping them.

7. Treat any "great" backtest with suspicion until it survives multiple regimes net-of-cost.

Limitations

This is a **research/education** blueprint, not a deployed, promoted trading system. The provided document explicitly disclaims guaranteed profit or zero defects.

**Non-stationarity is the rule.** A feature that worked last quarter can decay this quarter; validate out-of-sample and shadow-run before any live action.

**Costs are not modelled here.** A 0.52 Balanced Accuracy is below the bar once STT/brokerage is subtracted — the worked example is intentionally honest about that.

**No live promotion gate passed.** Per the research-governor, claims are bounded to method and discipline, never to profitability.

If any figure here conflicts with an official NSE/SEBI source, the official source wins — verify before you act.

---

*Author: Shakti Tiwari — NISM XII certified options educator (not a SEBI Registered Analyst). Educational content only; not investment advice. Verify all data against official NSE sources before acting.*

**More from the author:** [about.me/shaktitiwari](https://about.me/shaktitiwari) | [optiontradingwithai.in](https://optiontradingwithai.in) | WhatsApp: [+91 9169650895](https://wa.me/919169650895)

*Books by the author:*

*Options Trading with AI — Vol 1* — [B0H9ZNTBPK](https://www.amazon.in/dp/B0H9ZNTBPK)

*Options Trading with AI — Vol 2* — [B0HBBFKDQF](https://www.amazon.in/dp/B0HBBFKDQF)

Educational only — not SEBI-registered investment advice. NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.