The Missing Data Problem in Finance

Financial data is full of missing values. Stock data might have gaps from trading halts, option chain data has missing strikes, and technical indicators need warmup periods. Most ML algorithms require imputation before training. XGBoost handles this automatically.

How XGBoost Handles Missing Values

XGBoost uses a novel approach called sparsity-aware split finding. During training, it learns the optimal direction for missing values at each split. The algorithm:

  1. When evaluating splits, treats missing values as a separate category
  2. Calculates the best direction (left or right child) for missing values
  3. Remembers this direction for future predictions
  4. At prediction time, sends missing values to the learned direction

The Algorithm

# Pseudo-code for sparsity-aware split finding
for each feature in features:
    # Calculate gain for non-missing values going left
    gain_left = calculate_gain(data[~missing], direction='left')
    
    # Calculate gain for non-missing values going right  
    gain_right = calculate_gain(data[~missing], direction='right')
    
    # Calculate gain for missing values going left
    gain_missing_left = calculate_gain(data[missing], direction='left')
    
    # Calculate gain for missing values going right
    gain_missing_right = calculate_gain(data[missing], direction='right')
    
    # Choose direction with higher gain for missing values
    if gain_missing_left > gain_missing_right:
        missing_direction = 'left'
    else:
        missing_direction = 'right'
    
    # Store this direction in the split node

Why This Matters for Financial Data

  • No imputation bias: Mean/median imputation can distort distributions
  • Preserves missingness information: Sometimes missing data is informative (e.g., halted stock)
  • Automatic feature selection: Missing features are effectively excluded
  • Handles different types: Works for MCAR, MAR, and some MNAR patterns

Practical Example

import xgboost as xgb
import numpy as np
import pandas as pd

# Create data with missing values
X = pd.DataFrame({
    'rsi': [45, np.nan, 65, 30, np.nan, 55],
    'volume': [1000, 2000, np.nan, 1500, 3000, np.nan],
    'momentum': [0.02, -0.01, 0.03, np.nan, 0.01, -0.02]
})
y = np.array([1, 0, 1, 0, 1, 0])

# XGBoost handles NaN automatically
model = xgb.XGBClassifier(n_estimators=100, max_depth=3)
model.fit(X, y)  # No imputation needed!

# Check learned missing direction
booster = model.get_booster()
trees = booster.get_dump()
print('Missing direction learned in each tree')

Comparison with Imputation Methods

  • Mean imputation: Distorts variance, reduces signal
  • Median imputation: Better for outliers, still distorts distributions
  • KNN imputation: Better but computationally expensive
  • XGBoost native: Learns optimal direction, no preprocessing needed

When to Preprocess Anyway

While XGBoost handles missing values well, consider preprocessing when:

  • Missing percentage > 50% in a feature
  • Missing values are systematic (not random)
  • You need probability estimates (missing values affect calibration)
  • Using the model in production where missing patterns may differ

Best Practices

  • Let XGBoost handle missing values by default
  • Monitor missing value patterns over time
  • Document which features have missing values
  • Use NaN indicators as additional features if missingness is informative

A Controlled Test: Delete, Impute, or Let XGBoost Decide

Run the experiment yourself before trusting any rule of thumb. Take the Nifty feature set, inject missing values at random into a quarter of the columns, and compare three pipelines on identical walk-forward folds: rows with missing values dropped, missing values filled with the column median, and XGBoost left alone on the NaN. On this data XGBoost's native handling ties or beats median imputation in most tests, because the sparsity-aware algorithm separately learns the direction of each missing branch. But the conclusion is not universal; the experiment is cheap and the answer is yours.

How Missing Branches Actually Learn

XGBoost treats NaN as its own route at every split. During training, each split considers sending missing rows left or right and keeps whichever reduces loss, and at scoring time a row with a missing feature simply follows the learned path. The model is not filling the value; it is learning that missingness is a state, which is more flexible than any imputation because the missing branch can mean different things in different leaves. This is the property the "handles missing data automatically" slogan is selling, and it is genuinely real on financial features.

Why Missingness Can Be Informative in India

In Indian market data, holes are rarely random. A feature missing because the underlying was suspended, because an F&O series was illiquid, or because a company had a corporate action carries information about the event itself. A model that routes missing rows to their own branch can exploit that signal, while median imputation scrubs it out. Log why each gap exists in the feature pipeline, because a gap filled silently with yesterday's value performs fine in backtest and invented in production.

The Danger Threshold: When the Holes Win

Native handling degrades gracefully but not forever. When more than roughly three-quarters of a feature's rows are missing, the missing branch has almost no counterpart for comparison and the learned split becomes a coin flip. Collapse such columns before training: a binary flag for "was present" plus an imputed constant beats a 90 percent-missing feature with fake routing. Similarly, missing data in the label region of a regression must be handled before training, because XGBoost's native machinery protects features, not targets.

Combining Manual Handling Where It Belongs

Keep the native behaviour for internal pipeline gaps, and apply explicit care to the two places it does not belong: target construction and live-feature parity. If a live feed delivers NaN for a feature your training set always observed, the model has no missing branch to follow and the scoring path is undefined against backtest behaviour. Standardise missing values to a sentinel, document the convention, and test the live scorer against the exact representation the model saw in training. That parity check is worth more than the entire imputation library.

  1. Benchmark delete vs median vs native on your own folds.
  2. Log the reason for every gap; let informative missingness stay.
  3. Drop or flag columns that are more than roughly 75 percent empty.
  4. Never train with a feature your live scorer can omit.
  5. Handle target-side missing values explicitly before training.

Missing Flags and Live Parity

Wrap each gap-prone feature with a missingness flag - a binary column recording that the value was absent - and let the model decide whether the absence itself carries information, which it often does on suspended stocks and silent sessions. The flag keeps the missingness signal explicit beside the imputed value, and it survives the live pipeline cleanly because a zero-or-one column is unaffected by the imputation's conventions. The live-parity audit closes the loop: replay last month's data through the production scorer, confirm the missing representation matches the training convention exactly, and log any NaN that arrives where training saw a number. A system whose live values never produce a NaN its training set did not contain is a system whose missing handling cannot surprise the model mid-session.