Production Deployment

Moving from research to production requires careful planning. LightGBM models need to be serializable, fast, and monitorable.

Model Serialization

import lightgbm as lgb
import joblib
import json

# Save model
model.save_model('model.txt')  # LightGBM format
joblib.dump(model, 'model.pkl')  # Pickle format

# Save metadata
metadata = {
    'feature_names': model.feature_name(),
    'training_date': '2026-03-01',
    'parameters': params,
    'performance': {'auc': 0.58, 'accuracy': 0.56}
}
with open('metadata.json', 'w') as f:
    json.dump(metadata, f)

# Load model
model = lgb.Booster(model_file='model.txt')

Fast Prediction API

from fastapi import FastAPI
import lightgbm as lgb
import pandas as pd

app = FastAPI()
model = lgb.Booster(model_file='model.txt')

@app.post('/predict')
predict(data: dict):
    """Fast prediction endpoint."""
    # Convert to DataFrame
    df = pd.DataFrame([data])
    
    # Predict
    proba = model.predict(df)[0]
    
    return {
        'probability': float(proba),
        'signal': 'BUY' if proba > 0.58 else ('SELL' if proba < 0.42 else 'HOLD')
    }

Monitoring

import logging
from datetime import datetime

def log_prediction(data, prediction, actual=None):
    """Log predictions for monitoring."""
    log_entry = {
        'timestamp': datetime.now().isoformat(),
        'features': data,
        'prediction': prediction,
        'actual': actual
    }
    logging.info(json.dumps(log_entry))

# Monitor model performance
def calculate_rolling_accuracy(predictions, actuals, window=100):
    """Calculate rolling accuracy."""
    correct = sum(p == a for p, a in zip(predictions[-window:], actuals[-window:]))
    return correct / window

Retraining Pipeline

def retrain_model(new_data, existing_model):
    """Retrain model with new data."""
    # Prepare data
    X_new, y_new = prepare_data(new_data)
    
    # Incremental training
    train_data = lgb.Dataset(X_new, label=y_new)
    model = lgb.train(
        params,
        train_data,
        num_boost_round=50,
        init_model=existing_model
    )
    
    # Validate
    auc = evaluate_model(model, X_val, y_val)
    
    # Replace if better
    if auc > current_auc:
        model.save_model('model.txt')
        return model
    return existing_model

Best Practices

  • Version models: Keep history of all model versions
  • A/B test: Run new model alongside old one
  • Monitor drift: Track feature importance and prediction distribution
  • Set alerts: Alert if performance degrades

Feature Pipeline Versioning

Most production failures in ML trading are feature failures, not model failures. The discipline that prevents them:

  • Version every feature alongside the data snapshot that produced it: a hash of input data, transform code and config.
  • Recompute features from raw data on every retrain, never from a cached frame that may have been patched.
  • Diff the feature distributions between training and the live feed weekly; a subtly shifted rolling window silently changes predictions.

Champion-Challenger: The Only Honest A/B

Replacing a live model with a new one after one backtest is how production capital dies. Run a champion-challenger instead:

  1. Shadow-confirm the challenger against the same feed as the champion for a defined period, without letting it trade.
  2. Compare on two axes: signal quality (forecast error) and trade performance (P&L per rupee of risk, using identical fills).
  3. Promote the challenger only if it wins on both axes across two market regimes; if it wins each axis in different regimes, keep both.

Latency: Making LightGBM Fast Enough for a Session

LightGBM models are famously fast, but the plumbing around them decides whether you can act on the signal:

  • Serve the model through a compiled artifact (daal4py-style or pure LightGBM model files) so the scoring overhead is microseconds, not round trips to a Python process.
  • Predict in batch on the same timeframe once per minute rather than ad hoc per tick for daily models.
  • Keep the feature fetch on-memory: a model that waits on a database SELECT before every prediction is only as fast as the slowest JOIN.

Drift Detection That Acts

Monitoring loss curves is reactive; drift detection is predictive. Watch these three:

  • Feature drift via PSI or KS per feature versus the training window, actioned when the top-decile drifters flip meaning (sign of alpha moves negative).
  • Prediction drift: the model's output distribution shifting without market justification flags data or theatre changes.
  • Concept drift: realised target versus predicted target, the ground truth of whether the model still understands the game.

Rollback That Works in Minutes

Every production ML system needs a rollback that does not depend on git archaeology:

  1. Ship models as immutable, dated artifacts in object storage rather than files living inside the app.
  2. Keep the last N models warm and switchable by config flag; rolling back is changing a pointer, not redeploying a service.
  3. Automate the "roll back and flatten" drill: the runbook that, on any model-health alarm, switches to the previous artifact and disables new signals, not just logs the alarm.

The research-to-live bridge is an operations discipline before it is a data science one. Feature versioning, champion-challenger evaluation, fast serving and instant rollback are what turn a model that works once in a notebook into a system that works every day without drama.