Lessons from a real rebuild of an options-buyer prediction system. No profit claims —
just the architecture that fixes the chronic bugs of V1.
The Core Mistake in V1
V1 asked one XGBoost model one big fuzzy question: **"CE ya PE?"** — directly from raw
CE/PE premium data. Premium is a *transformed* signal (underlying move × delta × gamma × IV ×
theta × spread × strike distance × liquidity). The model learned noise as much as signal.
Concrete evidence from the research logs:
(`lr=0.02, depth=3` defaults used throughout; Optuna existed but was never run).
feature for the entire history.
training volume (387 sequences instead of 295K+).
V2 Principle: Split the Question
underlying mechanics --> side, range, ETA, invalidation
option chain scanner --> is the buyer contract worth paying for?
XGBoost (many heads) --> thin calibrated learner on clean mechanics
Rule: **underlying decides side; option contract decides execution eligibility.** CE/PE
premium is validated against, never learned as, direction.
Many Shallow Heads, Not One Deep Model
Instead of one CE/PE answer, V2 trains separate narrow heads:
This single change removes most of the CE/PE confusion V1 fought for months.
The Shallow Regularized Grid (the actual fix for overfit)
learning_rate = 0.015–0.035 n_estimators = 800–2000 (early stop)
max_depth = 2–3 min_child_weight = 12–40
gamma = 0.1–2.0 subsample = 0.65–0.90
colsample_bytree = 0.55–0.85 reg_alpha = 0.5–3.0
reg_lambda = 6.0–20.0 scale_pos_weight = min(neg/pos, 8.0)
V1's intraday head had only **8 of 1280 features** with non-zero gain — most of the bloat
was pure noise the regularizer had to prune. Shallow + hard-regularized is the answer.
Overfit Gate (Hard Promotion Rule)
`overfit_gap = train_metric − test_metric`. **Flag if > 0.15.** A model is NOT promoted just
because train metrics look good. Log the gap automatically on every head, every retrain.
The Honest Verdict
V2 is a cleaner architecture, but it is still **research**. The lesson that transfers: stop
asking fuzzy questions, declare your nulls, keep trees shallow, and gate promotion on
out-of-sample gap — not training score.
*Research only. Not investment advice.*