Early Stopping in LightGBM

LightGBM's early stopping monitors validation performance and stops training when it stops improving. This prevents overfitting and saves compute time.

import lightgbm as lgb

# Define parameters
params = {
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 15,
    'learning_rate': 0.02
}

# Prepare data
train_data = lgb.Dataset(X_train, label=y_train)
val_data = lgb.Dataset(X_val, label=y_val)

# Train with early stopping
callbacks = [
    lgb.early_stopping(stopping_rounds=50),
    lgb.log_evaluation(period=100)
]

model = lgb.train(
    params,
    train_data,
    num_boost_round=1000,
    valid_sets=[val_data],
    callbacks=callbacks
)

print(f'Best iteration: {model.best_iteration}')
print(f'Best score: {model.best_score}')

Regularization Parameters

L1 Regularization (lambda_l1)

Penalizes leaf weights. Higher values make model more conservative.

params = {
    'lambda_l1': 0.1,  # L1 regularization
    'lambda_l2': 1.0,  # L2 regularization
    'min_gain_to_split': 0.01  # Minimum gain for split
}

L2 Regularization (lambda_l2)

Prevents extreme leaf weights. Critical for noisy financial data.

min_child_samples

Minimum samples in leaf node. Higher values prevent overfitting to noise.

# Conservative settings for financial data
params = {
    'num_leaves': 15,
    'min_child_samples': 30,  # At least 30 samples per leaf
    'lambda_l1': 0.1,
    'lambda_l2': 1.0,
    'min_gain_to_split': 0.01
}

Feature and Bagging Fraction

Stochastic gradient boosting reduces variance:

params = {
    'feature_fraction': 0.8,  # Use 80% of features per tree
    'bagging_fraction': 0.8,  # Use 80% of data per tree
    'bagging_freq': 5  # Perform bagging every 5 iterations
}

Walk-Forward Regularization

def walk_forward_with_regularization(X, y, n_splits=20):
    """Walk-forward validation with regularization tuning."""
    scores = []
    
    for i in range(252, len(X) - 1):
        # Training data: expanding window
        X_train = X[:i]
        y_train = y[:i]
        
        # Validation data: most recent 20 days
        X_val = X[i-20:i]
        y_val = y[i-20:i]
        
        # Test data: next day
        X_test = X[i:i+1]
        y_test = y[i:i+1]
        
        params = {
            'objective': 'binary',
            'metric': 'auc',
            'num_leaves': 15,
            'min_child_samples': 30,
            'lambda_l1': 0.1,
            'lambda_l2': 1.0,
            'feature_fraction': 0.8,
            'bagging_fraction': 0.8
        }
        
        train_data = lgb.Dataset(X_train, label=y_train)
        val_data = lgb.Dataset(X_val, label=y_val)
        
        model = lgb.train(
            params,
            train_data,
            num_boost_round=1000,
            valid_sets=[val_data],
            callbacks=[lgb.early_stopping(50), lgb.log_evaluation(0)]
        )
        
        pred = model.predict(X_test)
        scores.append(pred[0])
    
    return scores

Monitoring Training

# Plot training history
import matplotlib.pyplot as plt

def plot_training_history(model):
    """Plot training history."""
    ax = model.plot_importance(model, max_num_features=15)
    plt.title('Feature Importance')
    plt.show()

Best Practices

  • Always use early stopping: Set stopping_rounds=50-100
  • Use both L1 and L2: lambda_l1=0.1, lambda_l2=1.0 for financial data
  • Set min_child_samples: 20-50 to prevent noise overfitting
  • Use feature/bagging fraction: 0.8 for stochastic gradient boosting
  • Monitor best_iteration: If it keeps increasing, model is overfitting

The Lambda Sweep, With Nifty-Meaningful Results

Regularisation in LightGBM is anchored by lambda_l1 and lambda_l2, and their effect on financial data is measurable:

  • lambda_l1 (L1) shrinks weights toward sparse solutions; on wide feature sets it visibly trims noisy momentum variants.
  • lambda_l2 (L2) shrinks all weights smoothly; on correlated moneyness/IV twins it stabilises without fully dropping either.
  • A practical sweep on daily Nifty features in the 0.1-10.0 range usually finds validation log-loss improving 5-15% versus the zero-regularisation baseline, and the improvement is greatest in small-sample regime windows where the model would otherwise memorise.

min_data_in_leaf Side Effects

min_data_in_leaf is the overlooked workhorse of LightGBM regularisation:

  • Set it near 5-10% of the training rows to stop leaves forming on noise-chunks; a leaf holding 40 samples on a 600-row window is just a memento of one Wednesday.
  • Raising it late in a sweep trims the deepest interaction paths and reliably flattens the train-test divergence curve.
  • The tradeoff with max_depth is asymmetric: raising min_data_in_leaf caps leaf legitimacy without killing genuine short-leaf interactions the way depth limits do.

Feature and Bagging Fractions: The Randomisation Budget

Randomness is regularisation in LightGBM, and the two dials deserve a deliberate budget:

  • feature_fraction 0.7-0.9 exposes each tree to a different column recipe, forcing the ensemble to rely on robust signal rather than a single lucky feature.
  • bagging_fraction 0.7-0.8 with bagging_freq matching the bag size creates (row, column) double-perturbation that dominates single-axis tuning.
  • Reduce boosting rounds by 30-50% and raise both fractions; the speed saved is a second-order benefit, the variance reduction is the first.

A Structured Pruning Schedule

Regularisation works best as a procedure, not a one-shot:

  1. Stage 1: fix min_data_in_leaf and bagging, sweep lambda_l1/l2 across 0.1-10.
  2. Stage 2: with tuned lambdas, sweep feature_fraction 0.6-0.9 and bagging_fraction 0.7-0.9.
  3. Stage 3: re-run early stopping with patience dialled to the new surface; the stopping round and the regulariser budgets must be tuned together, never independently.

Each stage evaluates on the same sequential validation fold; cross-stage coupling (lambdas changing the optimal bagging) is exactly why the schedule, rather than isolated tuners, earns the gain.

A Cookbook Config for Daily Indian Options Data

As a documented starting point for daily Nifty option-chain features (10k-100k rows):

  • num_leaves 31, max_depth 8-10, learning_rate 0.03-0.05.
  • lambda_l1 1.0, lambda_l2 2.0-4.0, min_data_in_leaf 50-200.
  • feature_fraction 0.8, bagging_fraction 0.75, bagging_freq 5.
  • Early stopping with patience 150-300 on a sequential 15% tail validation.

Regularisation is the process of making LightGBM's sharpness affordable. Order the lambdas, leaf-size floors and randomness budgets into a sweep, tune them together against one honest validation fold, and the result is a model that stops memorising the last calendar quarter and starts representing the regime that is actually ahead.