Why LightGBM for HFT?

High-frequency trading generates millions of data points daily. XGBoost struggles with this volume due to its exact greedy algorithm. LightGBM's histogram-based approach makes it the natural choice for HFT applications.

Data Characteristics

  • Volume: Millions of ticks per day
  • Features: Order book, trade flow, microstructure
  • Latency: Millisecond-level prediction requirements
  • Noise: Extremely high signal-to-noise ratio

Feature Engineering for HFT

def create_hft_features(ticks):
    """Create features for high-frequency trading."""
    features = pd.DataFrame(index=ticks.index)
    
    # Order book features
    features['bid_ask_spread'] = ticks['ask'] - ticks['bid']
    features['mid_price'] = (ticks['ask'] + ticks['bid']) / 2
    features['bid_ask_imbalance'] = ticks['bid_volume'] / (ticks['bid_volume'] + ticks['ask_volume'])
    
    # Trade flow features
    features['trade_imbalance'] = (ticks['buy_volume'] - ticks['sell_volume']) / (ticks['buy_volume'] + ticks['sell_volume'])
    features['vwap'] = (ticks['price'] * ticks['volume']).cumsum() / ticks['volume'].cumsum()
    features['price_vs_vwap'] = ticks['price'] / features['vwap']
    
    # Microstructure features
    features['spread_ratio'] = features['bid_ask_spread'] / ticks['price']
    features['volume_per_trade'] = ticks['volume'] / ticks['num_trades']
    
    # Momentum (very short-term)
    features['momentum_100ms'] = ticks['price'].pct_change(100)
    features['momentum_1s'] = ticks['price'].pct_change(1000)
    features['momentum_10s'] = ticks['price'].pct_change(10000)
    
    # Volatility (realized)
    features['realized_vol_1s'] = ticks['price'].pct_change().rolling(1000).std()
    features['realized_vol_10s'] = ticks['price'].pct_change().rolling(10000).std()
    
    return features

LightGBM for HFT

import lightgbm as lgb

# Ultra-fast LightGBM for HFT
params = {
    'objective': 'binary',
    'metric': 'auc',
    'boosting_type': 'gbdt',
    'num_leaves': 7,  # Very simple for HFT
    'max_depth': 3,
    'learning_rate': 0.1,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 5,
    'min_child_samples': 50,  # Large minimum for noise
    'verbose': -1
}

# Train on millions of samples efficiently
train_data = lgb.Dataset(X_train, label=y_train)
model = lgb.train(params, train_data, num_boost_round=200)

Speed Benchmarks

Training on 1 million samples, 50 features:

  • XGBoost: 12 minutes
  • LightGBM: 2.5 minutes
  • LightGBM is 4.8x faster

Prediction Latency

# LightGBM prediction is very fast
import time

start = time.time()
predictions = model.predict(X_test)
latency = (time.time() - start) / len(X_test) * 1000  # ms per prediction
print(f'Prediction latency: {latency:.3f} ms per sample')

Typical latency: 0.01-0.05 ms per prediction — suitable for HFT.

Online Learning

For HFT, models need frequent retraining:

# Incremental training
model = lgb.Booster(model_file='model.txt')  # Load existing model
new_data = lgb.Dataset(X_new, label=y_new)
model = lgb.train(params, new_data, num_boost_round=50, init_model=model)

SEBI Disclaimer

High-frequency trading involves substantial risk. This article is for educational purposes only. HFT requires significant infrastructure and capital. Past performance does not guarantee future results.

Tick Normalisation Before Anything Else

HFT models eat millions of raw ticks, and raw ticks are poison: gaps, duplicate timestamps, exchange bursts and vendor join artifacts all corrupt volume and order-flow features.

  • Normalise timestamps to exchange time and reject ticks with vendor arrival earlier than the exchange timestamp.
  • Deduplicate at the exchange-message level, not the timestamp level, because two genuine trades can share a millisecond.
  • Window the day: end-of-day balancing ticks and the 09:15 opening auction deserve separate handling, never a shared feature space.

Memory-Mapped Bars: The Ingestion Trick

At millions of ticks a day, normal DataFrame loading blows RAM before the model ever trains. The practical route:

  • Convert ticks to fixed-width binary (numpy or parquet) and memory-map the files; the OS caches the hot tail, and reads cost microseconds.
  • Build bars of your chosen size (1-second, 1-minute) in a streaming pass, keeping per-bar aggregates: open, high, low, close, volume, twap, imbalance.
  • Keep the raw tick archive on disk forever; every research question starts from a replay, and re-downloading is the most expensive habit in quant work.

The Micro-Feature Latency Budget

HFT features add their own latency. Cost them the way you cost prime money:

  • An order-flow imbalance feature with a rolling book depth (say 50 levels) adds roughly 5-15 microseconds per event in Python, worse in naive loops.
  • Batch the feature update per bar rather than per tick where signals allow; the model's decision cadence, not the tick rate, sets the budget.
  • A LightGBM predict on 50 features takes ~5-50 microseconds; the feature pipeline, not the tree, is where your edge's latency lives.

Order-Flow Imbalance Features

The single most informative family for microstructures is imbalance between buy and sell pressure:

  • Order-flow imbalance = (active buy volume - active sell volume) / (total aggressive volume) at sliding windows.
  • Stake-based versions weighting trades by trade size beat sign-only versions on most Indian index futures data.
  • Combine imbalance at 50ms, 500ms and 5s horizons; the model learns whether pressure is short-lived or persistent.

Tick-Level Backtest Fidelity

Backtests on 1-minute bars lie about HFT fills. At tick fidelity enforce:

  • Fill only when the model's price level was actually touchable at the decision instant, never the bar's close.
  • Model queue position: a marketable order that enters a long queue faces partial fills and later price realisations than the "touch printed" implies.
  • Charge explicit spread crossing on every market order; producers of pure maker logic survive, taker strategies die on the model's own simulation.

High-frequency LightGBM is a data-engineering business wearing an ML hat. Normalise the tick stream, memory-map the archive, cost every microsecond of feature latency, and validate fills at the tick, and the model's millions-of-ticks advantage stays real; skip any one of those and the "HFT edge" becomes expensive fiction.