The Kaggle Trading Revolution
Every year, top data scientists compete in Kaggle trading competitions sponsored by firms like Optiver, Jane Street, and Two Sigma. These competitions use real market data and the solutions reveal how institutional traders use machine learning. The winners consistently use XGBoost, LightGBM, and CatBoost — not because they are trendy, but because they work.
Optiver Trading at the Close: 1st Place Solution
The 2024 Optiver competition asked participants to predict how 200 NASDAQ stocks would move in the final 60 seconds of trading. The winner, hyd, combined three models:
- CatBoost: 50% weight — best single model for tabular data
- GRU (Neural Network): 30% weight — captures temporal patterns
- Transformer: 20% weight — captures cross-stock relationships
The key insight: no single model dominates. Ensembles of different model types perform better than any individual model.
Feature Engineering: What Top Performers Do
The winning solutions shared common feature engineering patterns:
- Price features: Raw price, mid price, weighted average price, returns
- Imbalance features: Bid-ask imbalance, order book depth, size imbalance
- Rolling features: 5-minute, 15-minute, 60-minute rolling means, std devs
- Historical targets: Past 5, 10, 20 days of target values
- Time features: Seconds in bucket, time of day, day of week
The 1st place winner used 300 features selected by CatBoost feature importance. This is critical — feature selection matters more than model complexity.
The Online Learning Secret
Top performers retrained their models periodically during the competition. The 1st place winner retrained every 12 days, 5 times total. This adapts to changing market conditions. Most beginners train once and never retrain — this is a mistake.
Post-Processing Tricks
The winner used a simple but powerful trick: subtract weighted mean from predictions. This zero-sum adjustment improved scores by 0.001 — significant in a competition where margins are tiny.
Jane Street: Autoencoder + MLP Approach
The 2021 Jane Street winner used a supervised autoencoder to create new features, then fed them into an MLP. The XGBoost model was blended with this deep learning approach. Key techniques:
- Gaussian noise: Added before encoder for data augmentation
- Swish activation: Better than ReLU for financial data
- Multiple seeds: Trained 3 models with different random seeds, averaged predictions
- Purged time-series CV: Prevented data leakage
Jane Street Real-Time Forecasting: XGBoost vs LSTM
In the 2024-2025 Jane Street competition, XGBoost outperformed LSTM for most participants. The 9th place winner found:
- XGBoost with 75 trees: Score 0.006171
- Two-layer LSTM: Score 0.004329
- LSTM with lags: Score -0.000567 (negative!)
Lesson: Trees outperform RNNs when feature interactions dominate over sequential patterns.
Optimal XGBoost Hyperparameters for Trading
From analyzing winning solutions, here are the optimal parameters:
# XGBoost for Trading - Optimized Parameters
import xgboost as xgb
model = xgb.XGBRegressor(
n_estimators=75, # Fewer trees, prevent overfitting
max_depth=4, # Shallow trees for tabular data
learning_rate=0.1, # Standard learning rate
subsample=0.71, # 71% of data per tree
colsample_bytree=0.73, # 73% of features per tree
gamma=0.26, # Minimum loss reduction
tree_method='hist', # Fast histogram-based training
early_stopping_rounds=10 # Stop if no improvement
)The Feature Selection Methodology
Top performers do not use all features. They use feature importance to select top features:
- Train XGBoost on all features
- Get feature importance scores
- Select top 200-300 features
- Retrain on selected features only
- Repeat until performance stabilizes
This reduces overfitting and improves generalization.
How to Apply This to Your Trading
- Start simple: Use XGBoost with 50-100 features
- Add features gradually: Test each new feature's impact
- Use time-series CV: Never use random splits for financial data
- Retrain regularly: Markets change; your model must adapt
- Ensemble models: Combine XGBoost with LightGBM or neural networks
SEBI Disclaimer
This article is for educational purposes only. Algorithmic trading involves substantial risk of loss. Past performance does not guarantee future results.
What the Public Notebooks Don't Teach: The Time Leak
Kaggle notebooks run on a fixed test set, and the winning tricks that dazzle publicly - clever feature crosses, aggressive ensembles - often carry a silent time leak: the training folds overlapped the test window, or the feature lag was inadequately represented. In trading, that leak is instant death, because the market's test set is the future and it has no patience for a notebook's assumptions. The durable lesson from the winners is not the features they posted but the discipline they themselves broke in competition: replicate their feature families, then re-validate everything with a calendar-true, cost-audited walk-forward. The public scoreboard never pays the spread; the market always does.
Replicating the Optiver-Style Pipeline on Nifty Data
The quantitative-trading "predict the close" competitions teach a transferable trio: clean the features by their time-of-availability, order the samples so prediction never sees the target's future, and normalise by the volume or activity of the row, not by a global mean. Applied to Nifty data the pipeline becomes: build per-row features computable at the close, split by a calendar cut, and report the hit rate on the untouched tail. The transfer is the structure, not the coefficients - the winners' individual numbers re-fit to a different market's noise, while the pipeline's shape survives to the Indian series.
Online Learning and the 2026 Shift
The more recent winning solutions lean on online and incremental learning, feeding the model the market's drift session by session rather than a one-shop batch fit. The trading translation is direct: a model that relearns the regime's slow rotation stays honest through the year; a batch model inherited from January botches October. The 2026 takeaway for the retail XGBoost trader: build the model's retrain cadence into the pipeline as a first-class feature of the system, and let the slow learner be tested on the regime before the system trusts its edge again.
Post-Processing Tricks That Transfers
The winners' post-processing - rank-normalising the raw outputs, applying an outlier cap, blending the top models by a weight fitted to the validation fold - transfers cleanly to trading when the validation is a chronological window. The ranked blend deserves the discipline's attention because ranking removes the scale-heroics of a single model; the capped outlier prevents a single bombarded row from wrecking the equity curve's average. Import the tricks that survive the cost model, apply them to the walk-forward, and ignore the tricks that only beat a leaderboard.
The One Habit Worth Stealing: Documentation
The competition-winning habits are not the architecture but the memory: the winners document every experiment, every kernel's parameter, and every validation quirk. The trader who runs one labelled experiment matrix and one README per project inherits the habit that turns research into a pipeline a future version can rebuild. When the market's regime reshuffles the leaderboard, the documented path is the one that re-proves its edge in time; the others watch it fade from memory as the session quietly untrades itself.
- Re-validate every public trick with a calendar-true, cost-audited split.
- Copy the pipeline's time-of-availability shape, never the coefficients.
- Build the retrain cadence in as a first-class feature.
- Blend by rank and cap the outliers after the walk-forward.
- Document every experiment the way the winners data-mine their own.
The Winner's Public Secret: Cash Flow Features and Blended Targets
Reading the market as a Kaggle table works when the top teams do one public thing: engineer the feature budget from the cash-flow statement and the sector peer group, normalise each column within its own sector, and let the tree hunt the interactions the single-column screens miss. The blended target is the second public secret - the winning teams density weeks by training on a smoothed or compounded target - because smoothing teaches the model the trend and the trees learn the noise faster, and the composite loss the winners cite is a curve the model is asked to follow, not memorise. Practise the format on Kaggle datasets for two months to internalise the discipline - the leaderboard's median honest, the public score un-gamed, the feature importance audited - and only then point the same habits at NSE data. The winners' real edge is procedural, not mystical: they prod the data the same audit each day until the edge names itself.
