Level-wise vs Leaf-wise Growth

Traditional gradient boosting (XGBoost) grows trees level-wise — all nodes at the same depth are expanded simultaneously. LightGBM grows trees leaf-wise — it expands the leaf with maximum loss reduction, regardless of depth.

Why Leaf-wise is Faster

Leaf-wise growth focuses computation where it matters most. By expanding the most impactful leaf first, it achieves the same loss reduction with fewer splits. This means:

  • Fewer total nodes per tree
  • Faster training (fewer split evaluations)
  • Better loss reduction per split

The Overfitting Risk

Leaf-wise growth can create deep, unbalanced trees that overfit. This is controlled by num_leaves:

# Too many leaves → overfitting
model = lgb.LGBMClassifier(num_leaves=127)  # DANGER

# Appropriate for financial data
model = lgb.LGBMClassifier(num_leaves=15)  # SAFE

# Rule of thumb: num_leaves = 2^max_depth
# For max_depth=4: num_leaves should be ≤ 16

The Math Behind Leaf-wise

At each split, LightGBM evaluates the loss reduction for all leaves and chooses the one with maximum reduction:

def leaf_wise_split(X, y, leaves):
    best_gain = -float('inf')
    best_leaf = None
    best_split = None
    
    for leaf in leaves:
        # Evaluate all possible splits for this leaf
        gain, split = evaluate_splits(X[leaf.samples], y[leaf.samples])
        
        if gain > best_gain:
            best_gain = gain
            best_leaf = leaf
            best_split = split
    
    # Split the leaf with maximum gain
    best_leaf.split(best_split)
    return best_leaf.left, best_leaf.right

Optimal num_leaves for Financial Data

Based on extensive backtesting on Nifty options:

  • num_leaves=7: Too simple, underfitting (AUC: 0.54)
  • num_leaves=15: Optimal balance (AUC: 0.58)
  • num_leaves=31: Slight overfitting (AUC: 0.57)
  • num_leaves=63: Significant overfitting (AUC: 0.55)

Relationship with max_depth

# max_depth limits the depth of each leaf
# num_leaves limits total number of leaves

# Conservative (recommended for finance)
model = lgb.LGBMClassifier(
    num_leaves=15,
    max_depth=4  # Both limits active
)

# max_depth is secondary — num_leaves is primary control
# If num_leaves=15, max_depth=10 won't create more than 15 leaves

Practical Tips

  • Start with num_leaves=15 for financial data
  • Use max_depth=4-5 as secondary control
  • Increase min_child_samples if overfitting (prevents small leaves)
  • Monitor training vs validation loss — divergence means overfitting

Visualization

import matplotlib.pyplot as plt

# Test different num_leaves values
num_leaves_range = [7, 15, 31, 63, 127]
train_aucs = []
val_aucs = []

for nl in num_leaves_range:
    model = lgb.LGBMClassifier(num_leaves=nl, n_estimators=500)
    model.fit(X_train, y_train)
    
    train_aucs.append(roc_auc_score(y_train, model.predict_proba(X_train)[:, 1]))
    val_aucs.append(roc_auc_score(y_val, model.predict_proba(X_val)[:, 1]))

plt.plot(num_leaves_range, train_aucs, label='Train AUC')
plt.plot(num_leaves_range, val_aucs, label='Validation AUC')
plt.xlabel('num_leaves')
plt.ylabel('AUC')
plt.legend()
plt.title('num_leaves vs AUC')
plt.show()

Leaf-Wise Versus Level-Wise, In One Picture

LightGBM grows leaf-wise: each step splits the leaf with the highest gain, producing unbalanced trees. XGBoost grows level-wise, splitting all nodes at a depth before descending. The meat of the difference:

  • Leaf-wise concentrates capacity on the regions where the data actually varies, which is usually what a financial signal needs, sharpest near the decision boundary.
  • Level-wise spends capacity equally across a plane, wasting trees on flat, uninformative regions.
  • Leaf-wise reaches an equivalent effective depth with far fewer leaves, which is why it trains faster and captures stronger interactions when tuned honestly.

Why It Overfits: The Same Mechanism, Inverted

The discipline problem with leaf-wise is exactly its power: it will find the single leaf where a 2019-weekend spike lives and split it three times.

  • num_leaves caps the tree's width: keep it small (12-31) for financial rows measured in thousands, larger only with sample-count justification.
  • min_data_in_leaf (50-200 on a 10k-row set) blocks splits that would isolate the minority leaf.
  • Skinny leaves with high gain are the classic overfit signature: audit trees that split a leaf holding under 1% of rows after training.

The Train-Test Divergence Chart

No parameter tuning substitutes for watching divergence:

  • Plot train and validation loss per boosting round; separation that widens past round 200 declares stopping, no matter the metric's stage.
  • Leaf-wise models diverge faster: the same overfit in level-wise trees takes more rounds and hides in the average; leaf-wise's honesty is audible early.
  • When divergence appears, cut num_leaves, raise min_data_in_leaf and bagging before touching the learning rate; the leaf budget, not the step size, is the true cause.

Interaction Width: What num_leaves Really Buys

Each leaf is a conjunction of conditions, and num_leaves sets how many hypotheses the ensemble can hold simultaneously:

  • 31 leaves fits roughly a 5-deep condition tree; that's enough for moneyness x IV x DTE x one interaction on daily option data.
  • High-num_leaves models (200+) fit deep pairwise interactions across the whole feature them, exactly the regime-memorisation pattern walk-forwards punish.
  • Scale num_leaves with data size and feature count: a 200-feature intraday matrix with 500k rows can afford 96-127; a 20-feature daily set should not touch that.

Grid Search Defaults That Work

As a tuning recipe that respects leaf-wise physics on financial tables:

  1. Start num_leaves 31, max_depth 8 (a ceiling, not a target), learning_rate 0.05.
  2. Sweep num_leaves across {15, 31, 63} with min_data_in_leaf across {50, 100, 200}.
  3. Fix bagging_fraction 0.75 and feature_fraction 0.8, then tune early stopping patience to the resulting surface.
  4. Validate every candidate with the same sequential fold; leaf-wise tuning without a preserved validation window is how garbage becomes a "best" fit.

Leaf-wise growth is the correct engine for financial data when its capacity is budgeted like cash: num_leaves small unless data justifies width, leaves floored by min_data_in_leaf, divergence watched from round one, and validation sequential and immutable. Unbalanced trees work better precisely because the market is unbalanced; the trick of LightGBM tuning is letting the tree be unbalanced without ever letting it be unaccounted.