Feature Engineering for Financial ML Models: The Complete Toolkit
Feature engineering is where quant models are won or lost. The best model in the world cannot extract signal from a feature set that encodes none, while a modest model on carefully built features routinely outperforms. This article catalogues the feature families that matter for Indian market data, shows how to build them leakage-free, and explains how to let validation decide what stays.
Why Features Matter More Than Models
Cross-model comparisons from Kaggle's finance competitions (Optiver, Jane Street) repeatedly show that the winning teams shared an obsession with features and validation, while model choice mattered less. Financial features are noisy and collinear; the art is to encode market structure so trees or linear models can find the weak signal without drowning in noise.
Feature Families That Actually Work
1. Return and Momentum Features
- Simple and log returns at 5/10/20/60-day horizons
- Momentum = current price vs price N days ago (normalised)
- Rolling mean-reversion distance (price vs 20-day mean)
2. Volatility Features
- Rolling standard deviation of returns (20-day, annualised)
- ATR over 14 days; Parkinson range-based vol
- GARCH-style EWMA vol with fixed half-life
- Realised vol of intraday ranges when intraday data exists
3. Volume Features
- Volume ratio: current volume / 20-day average
- On-balance volume slope; money-flow index
- VWAP distance: price / daily VWAP - 1
4. Market-Structure and Cross-Asset Features
- Index return (NIFTY) for a stock; sector return
- India VIX level and change (volatility regime indicator)
- USD/INR change for export/import-heavy sectors
5. Calendar Features
- Day-of-week, month, quarter-to-date return
- Days to F&O expiry, days to earnings, post-holiday flag
Writing Them Leakage-Free
import pandas as pd, numpy as np
def build_feats(df, price_col="Close"):
df = df.copy()
df["ret_5"] = df[price_col].pct_change(5)
df["ret_20"] = df[price_col].pct_change(20)
df["vol_20"] = df[price_col].pct_change().rolling(20).std()*np.sqrt(252)
df["vr"] = df["Volume"]/df["Volume"].rolling(20).mean()
df["vwap_dist"] = df["Close"]/df["Close"].rolling(20).mean()-1
return df
# NEVER: df["signal"] = df["Close"] > df["Close"].shift(-1) # leaks futureThe cardinal rule: every feature must be a function of information available at or before the prediction time. Shift everything that could accidentally look ahead.
Filtering Features by Validation, Not Intuition
- Build a broad candidate set
- Run a quick XGBoost with feature importances on a TimeSeriesSplit
- Drop features with near-zero importance
- Check SHAP values for sign stability across folds
- Keep the reduction only if validation improves
Time-Specific Versus Cross-Sectional Features
Time-specific features (this stock's 20-day vol) are the workhorse for a single-instrument model. Cross-sectional features (stock rank within its sector, relative momentum) shine when predicting across many stocks, because they encode relative attractiveness that a single stock cannot. Use both when your dataset allows.
Common Feature Flaws
- Extreme outliers from stock splits or data glitches - winsorise or floor
- Multicollinearity bloating feature importances - drop one of a correlated pair
- Leaking the label through a variable computed on the full window
- Testing features on the data you later test the model on - divide data once, keep a final untouched holdout
The Workflow Summary
1) Gather adjusted data. 2) Build features backwards-only. 3) Split strictly by time with a gap. 4) Filter by validation importance. 5) Validate the final set on a last untouched period. 6) Monitor live drift. The features are 80% of the game - spend your hours there.
SEBI Disclaimer
Algorithmic trading involves substantial risk. This article is educational and is not investment advice.
Feature Versioning: The Silent Backtest Killer
The feature set is a moving asset, and its versioning is the discipline that keeps every backtest reproducible. Each feature family should carry a name, a definition, and the exact data snapshot it was computed from; a change to a window length or a normalisation method without a version bump silently re-writes history and makes every older result non-comparable. The practical habit is to store the computed feature frame with the model's training run, so the model file, the feature file, and the parameters live as one artifact. When the next-quarter backtest drops in accuracy, the first question is "which feature version?", not "what did the model learn?".
Rolling Windows: Lookbacks and Their Costs
Rolling statistics - the 20-day mean, the 50-day standard deviation - create the backbone of most tables, but every window choice carries a second-order cost: chart edges. At the left of the sample the features are sparse or undefined, shrinking the trainable region, and the same hits silently during regime gaps like market closures. Handle the warm-up explicitly by aligning every feature's effective start date, and prefer lookbacks that are at least as long as the events you measure. A 20-day feature computed on a 15-day history is a padded prediction at best, a leaked one at worst.
Normalisation Traps: z-Score vs Rank vs Pct Change
How a feature is scaled changes what the tree can learn. z-score rescaling centres a level feature around its mean, which is defensible for stationary series and misleading for trending ones; percentile ranking converts the feature into its position in the recent distribution, which helps across regimes; the percentage-change feature is the least ambitious and most robust of the three. The mistake is mixing conventions silently across the table: a level feature z-scored against today's distribution and a level feature e of its own mean behave differently under the same model. Choose one convention per feature family, document it, and never let a normaliser walk into the wrong column.
A Ten-Feature Starter Bake
A compact daily table that reliably informs a Nifty model: 1-day, 5-day, and 20-day returns; 20-day realised volatility; 50-day moving average distance; rolling volume ratio; the day-of-week one-hot; the distance to the 20-day high; the session's range as a ratio of the 20-day mean range; and the change in open interest weighted by proximity to spot. This starter ten beats more features additively only when each earns its keep on the walk-forward folds. Add others - funding proxies, cross-asset signals - only when the baseline stops improving; the discipline of starting with ten and adding on evidence is the whole game.
Interaction Features That Survive Financial Noise
Pairwise interactions help trees express the positions where a joint signal beats each feature alone - momentum multiplied by a low-vol state, or range ratio interacted with the day-of-week. The tree can discover the interaction itself given depth, but seeding a few designed ones speeds the learning and shrinks depth need. Keep the interaction count below the patience of your pruning step, because ten designed interactions on the right logic beat a hundred auto-milled crosses that the validation window cannot tell apart from noise.
Target Encoding Dangers for Intraday Data
Target encoding - replacing a categorical with the mean outcome of that category - leaks catastrophically when the mean is computed across time. The intraday version computes a level's average next-move from data that includes the very move the model will predict, converting a 52 percent model into a 90 percent backtest and a live disaster. The safe variant applies the encoding inside each fold's training segment only, recomputed per fold, and never once the row belongs to the validation window. If encoding complexity arrives cheaper than a weekday one-hot on your actual table, use the one-hot.
- Version every feature family from day one.
- Align warm-up edges across rolling windows.
- Pick one scaling convention per family and stick to it.
- Start with the ten-feature baseline; add on evidence.
- Encourage interactions, but prune the auto-milled marriages.
