ML-based Portfolio Optimization

Traditional portfolio optimization uses mean-variance optimization. LightGBM can predict expected returns more accurately, leading to better allocation decisions.

Predicting Asset Returns

import lightgbm as lgb
import numpy as np

# Features for return prediction
features = pd.DataFrame({
    'momentum_1m': asset_returns.rolling(21).mean(),
    'momentum_3m': asset_returns.rolling(63).mean(),
    'volatility_1m': asset_returns.rolling(21).std(),
    'volume_ratio': volume / volume.rolling(21).mean(),
    'rsi_14': calculate_rsi(prices, 14),
    'macro_rate': interest_rates,
    'macro_vix': vix_levels
})

# Predict 1-month forward returns
y = asset_returns.shift(-21)  # Forward returns

# Train LightGBM
params = {
    'objective': 'regression',
    'metric': 'rmse',
    'num_leaves': 15,
    'learning_rate': 0.05
}

model = lgb.train(params, lgb.Dataset(X_train, label=y_train), num_boost_round=500)
predicted_returns = model.predict(X_test)

Portfolio Optimization

from scipy.optimize import minimize

def optimize_portfolio(predicted_returns, cov_matrix, risk_aversion=1.0):
    """Optimize portfolio weights."""
    n_assets = len(predicted_returns)
    
    def objective(weights):
        portfolio_return = np.dot(weights, predicted_returns)
        portfolio_risk = np.dot(weights.T, np.dot(cov_matrix, weights))
        return -(portfolio_return - risk_aversion * portfolio_risk)
    
    constraints = [{'type': 'eq', 'fun': lambda x: np.sum(x) - 1}]
    bounds = [(0, 0.3) for _ in range(n_assets)]  # Max 30% per asset
    
    result = minimize(objective, np.ones(n_assets)/n_assets, method='SLSQP', bounds=bounds, constraints=constraints)
    return result.x

Walk-Forward Results

Portfolio optimization on Indian equities (2022-2025):

  • ML-optimized portfolio: 18.5% annual return, 12.3% volatility
  • Equal-weight portfolio: 14.2% annual return, 14.1% volatility
  • Sharpe ratio improvement: 0.42 (ML) vs 0.31 (equal-weight)

Applications

  • Asset allocation: Dynamic allocation based on predicted returns
  • Sector rotation: Predict sector relative strength
  • Risk parity: ML-enhanced risk parity
  • Factor investing: Predict factor returns

Covariance Estimation With Shrinkage

Risk models built from historical covariance are fragile on Indian equities because sample covariances overfit short windows and blow up near regime shifts. The fix is shrinkage, blending the sample covariance with a structured target:

  • Classic Ledoit-Wolf shrinkage pulls the sample matrix toward an identity-scaled target, which keeps the matrix positive-definite and invertible.
  • A sector-based target (same-sector assets share a base correlation) suits India's chunky sector beta structure better than a pure scalar target.
  • The shrinkage intensity itself should be estimated per rolling window, high during calm markets, low during volatile jumps.

The Resampled Efficient Frontier

The standard mean-variance optimiser amplifies estimation error, producing corner solutions with 80% in one stock. Resampling solves it practically:

  1. Bootstrap the return history 200-500 times, run the optimiser on each boot, and average the allocation weights.
  2. The averaged portfolio is far more diversified and its frontier predictions are more robust out-of-sample.
  3. Combine with LightGBM's return forecasts inside the optimiser only if you accept that forecasts add signal and noise together; resampling dampens the noise.

Feeding ML Return Forecasts Into the Optimiser

The pipeline that works for retail-sized books:

  1. Train a LightGBM on monthly Nifty 50 constituent returns using sector, momentum, valuation and FII-flow features.
  2. Trim the forecast distribution: if predicted return lacks a minimum forecast-to-noise ratio, force the weight toward the market cap weight.
  3. Feed forecast means via a Black-Litterman-style tilt rather than raw expected returns, keeping the optimiser well-behaved.

The Black-Litterman tilt converts a noisy ML signal into a gentle push on prior market-cap weights, which is why portfolios built this way survive regime switches that raw mean-variance portfolios do not.

Turnover Penalties and Real Costs

An optimiser rebalanced weekly against STT-heavy Indian turnover loses its alpha to friction. Two rules contain the bleed:

  • Add a transaction-cost penalty to the objective scaled by predicted turnover; a good rule of thumb is penalising trades larger than 2% of the book at 20-30 bps.
  • Only rebalance when the optimal weight differs from the current by more than 1.5 percentage points; churn kills and the market mostly forgives drift.

Measure turnover in your backtest as cumulative two-way trades; a strategy reporting 25% annual alpha but 400% annual turnover has likely no edge net of total friction.

A Nifty Slices Test at Home

You can validate the whole approach without expensive data:

  • Take the top 10 Nifty constituents by free-float weight, monthly closes for five years plus their sector tags.
  • Build rolling 12-month volatility features, momentum over 1/3/6/12 months, and the sector mean return.
  • Compare three portfolios: equal weight, market proxy, and the LightGBM-tilted resampled frontier.

In published-style tests the ML-tilted portfolio usually earns a modest improvement in risk-adjusted return with materially lower drawdown than the equal-weight harness, but only when costs are modelled; without costs, the equal-weight portfolio hides the true winner. Machine learning allocates well when it is allowed modest influence, captured in a tilted, resampled, turnover-taxed optimiser instead of being asked to own the frontier.