Optiver Kaggle Competitions

Optiver has hosted multiple Kaggle competitions for stock prediction. Here is how the winning solutions worked.

Competition 1: Realized Volatility Prediction (2021)

Predict volatility using order book data. 3,862 teams competed.

1st Place Solution

  • Model: LightGBM + Neural Network ensemble
  • Features: Realized volatility, order book imbalance, price ranges
  • Key insight: Feature engineering contributed 80% of performance
# LightGBM model
import lightgbm as lgb

params = {
    'objective': 'regression',
    'metric': 'mae',
    'num_leaves': 31,
    'learning_rate': 0.05
}

model = lgb.train(params, train_data, 1000)

Competition 2: Trading Algorithm (2022)

Build a trading algorithm for stock market. 2,000+ teams.

Winning Approach

  • Strategy: Mean reversion with momentum overlay
  • Features: RSI, MACD, order flow, VWAP
  • Execution: Limit orders with urgency scoring

Competition 3: Optiver REAL (2023)

Realized volatility prediction with options data.

Key Findings

  • Options data improved predictions by 15%
  • Time-series cross-validation essential
  • Feature normalization matters more than model choice

Key Takeaways

  1. Feature engineering is 80% of success
  2. Simple models often win
  3. Proper validation is critical
  4. Ensemble methods improve performance

SEBI Disclaimer

Algorithmic trading involves risk of loss. This article is for educational purposes only.

Why the Competition Was Designed That Way

Optiver's realized-volatility competitions forced entrants to forecast a quantity that market makers truly use: near-term realized volatility, defined from a modest number of minutes and a futuristic-looking set of features including order-book imbalance at multiple time gaps. The deliberate design taught a specific lesson: feature construction, not exotic models, decided the ranking. Winning top solutions overwhelmingly standardised features, handled outliers hard, and ran ensembles of gradient boosting over any novel architecture.

Feature Engineering From the Winners

The first-place teams converged on a small set of high-leverage transforms rather than hundreds of raw columns:

  • Log-scaling of target-adjacent quantities: realized volatility is positive and skewed, so log and the square-root transforms stabilised variance.
  • Book imbalance at matched time gaps: the difference between bid and ask depth normalised by total depth, computed at the same gaps as the target.
  • Price-return ladders: returns over short and long lags combined with the imbalance in a difference form.
  • Winsorisation: clipping heavy tails at high percentiles to let boosting learn the mass of the distribution instead of chasing outliers.

Consistently, clean log-of-target and imbalance features beat raw levels by wide margins, and nobody needed a neural net to get those gains.

Model Budget: XGBoost Plus LightGBM Ensemble

The 1st place recipe was boring in the best way:

  1. Train an XGBoost with a tweedie or huber objective on the full standardised set.
  2. Train a LightGBM on the same folds with different hyperparameters: shallower trees, higher bagging.
  3. Blend predictions with rank-averaging plus a weight that was fit out-of-fold on a holdout slice.
  4. Add a light model on grouped averages to snap the ensemble toward the strong time-grouping of the data.

The result: the two gradient boosters captured overlapping but different curvature, and the blend consistently beat each alone by more than the spread between them.

Rebuilding a Mini-Optiver Dataset Yourself

You can train the same skills on NSE tick data without entering a competition:

  • Download a year of Nifty futures depth snapshots (bid/ask plus cumulative volumes).
  • Build realized-volatility targets over 1, 5 and 10 minute windows.
  • Compute the identical imbalance and return-lag features.

A weekend of this on free broker data builds exactly the intuition the Optiver competition was designed to teach: improving data construction beats improving models at low signal-to-noise.

From Kaggle to Real Books

Competition skills transfer to trading with three corrections:

  • Competition targets are known ex-post; live realised volatility is a bet made in real time, with fees and slippage attached.
  • Leaderboard overfitting gave blenders a penalty structure real trading does not; use walk-forward evaluation instead.
  • Tiny edges (a 0.001 rank improvement) become nothing once STT and spreads are applied, so filter signals for significance, not for rank.

The Optiver competitions are the best public lesson that disciplined feature work plus a modest gradient-boosting ensemble remains the professional baseline in market microstructure modelling.

Lessons Retail Traders Can Borrow

The winning edge was disciplined feature engineering plus ensembling the two boosters on cross-validated folds, not a magical model. They validated with time-aware folds and kept public/private congruity in mind. Replicate the pipeline habits, not the parameters: walk-forward folds, feature stability checks, and honesty about validation leakage.