Why Feature Selection Matters in Finance
Financial datasets often have hundreds of features, many of which are noise. Including noisy features causes overfitting, slows training, and reduces model interpretability. Feature selection is essential for building robust trading models.
Feature Selection Methods
1. Importance-based Selection
Use XGBoost's built-in feature importance to remove low-importance features:
import pandas as pd
import xgboost as xgb
# Train model with all features
model = xgb.XGBClassifier(n_estimators=500, max_depth=4)
model.fit(X_train, y_train)
# Get feature importance
importance = pd.DataFrame({
'feature': X.columns,
'importance': model.feature_importances_
}).sort_values('importance', ascending=False)
# Select top N features
top_features = importance.head(20)['feature'].tolist()
X_selected = X[top_features]2. Recursive Feature Elimination
from sklearn.feature_selection import RFE
model = xgb.XGBClassifier(n_estimators=500, max_depth=4)
rfe = RFE(model, n_features_to_select=20, step=5)
X_selected = rfe.fit_transform(X_train, y_train)
# Get selected features
selected_features = X.columns[rfe.support_].tolist()3. SHAP-based Selection
import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
# Mean absolute SHAP values
shap_importance = pd.DataFrame({
'feature': X.columns,
'importance': np.abs(shap_values).mean(axis=0)
}).sort_values('importance', ascending=False)
# Select features with SHAP > threshold
top_features = shap_importance[shap_importance['importance'] > 0.01]['feature'].tolist()4. Permutation Importance
from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)
perm_importance = pd.DataFrame({
'feature': X.columns,
'importance': result.importances_mean
}).sort_values('importance', ascending=False)Feature Selection Pipeline
def select_features(X, y, n_features=20):
"""Complete feature selection pipeline."""
# Step 1: Remove zero-variance features
from sklearn.feature_selection import VarianceThreshold
selector = VarianceThreshold(threshold=0.01)
X_filtered = selector.fit_transform(X)
filtered_features = X.columns[selector.get_support()]
# Step 2: Remove highly correlated features
corr_matrix = pd.DataFrame(X_filtered, columns=filtered_features).corr().abs()
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
to_drop = [col for col in upper.columns if any(upper[col] > 0.95)]
X_filtered = pd.DataFrame(X_filtered, columns=filtered_features).drop(columns=to_drop)
# Step 3: Importance-based selection
model = xgb.XGBClassifier(n_estimators=500, max_depth=4)
model.fit(X_filtered, y)
importance = pd.Series(model.feature_importances_, index=X_filtered.columns)
top_features = importance.nlargest(n_features).index.tolist()
return top_featuresResults on Nifty Data
Feature selection impact on walk-forward AUC:
- All 50 features: 0.56 AUC
- Top 30 features: 0.58 AUC
- Top 20 features: 0.59 AUC
- Top 15 features: 0.58 AUC
Optimal: 15-20 features for Nifty trading
Common Mistakes
- Removing features based on training importance only — validate on test data
- Ignoring feature correlations — correlated features inflate importance
- Over-selecting — more features ≠ better performance
- Not re-validating after selection — always re-run walk-forward
Stability Ranking First, Not Just Importance
Importance tells you what the model used; stability tells you whether it can be trusted to keep using it. Practical protocol:
- Run the feature-selection importance across five bootstrapped resamples of the data.
- Score each feature by how often it appears in the top decile of importance across the resamples; a feature that flips between top-10 and bottom-half is noise wearing a promotion.
- Select on the stable high-frequency subset for the final model; the surviving list is far smaller and far more honest.
Multicollinearity Versus Actually-Predictive Twins
Correlated features both help and harm, and the correct response is not automatic deletion:
- Two features correlating at 0.98 often both earn importance in a boosting model because each shadows the other's noise differently; deletion rarely changes the objective.
- Delete when the twin's correlations flip sign of economic meaning (log-price and raw-price), which corrupts interpretation, not just variance.
- Use the partial dependence after deletion to confirm the surviving feature's story changed only trivially.
The Causality Check: Lagged Response Discipline
Feature selection must respect time order, or your selected set is a hindsight palace:
- Reject any feature whose value is unobservable at prediction time, even a single bar late.
- Prefer lagged inputs (yesterday's close, 2-day returns) over same-bar values for daily models; the decay is more honest under real deployment.
- Run a leak audit after selection: hold out one future month and re-trace selected features; if the model scores suspiciously well in the "leaky month", reselect with the audit on.
Nested Resampling for Final Validation
Choosing features on the same fold you validate on leaks at double speed. Nested structure works:
- Outer loop: walk-forward slices where the strategy trades; the model is frozen per slice.
- Inner loop: selection and its hyperparameter tuning happen on earlier slices only.
- Report only the outer-loop performance; any strategy that cannot survive this structure is a misfit model, not a promising signal.
A Purging Checklist Before Production
Before any curated feature set reaches capital, run the final gate:
- Every feature survives the five-resample stability test.
- No feature shows an unlagged response violation under audit.
- The set's expected behaviour in crash, bull and sideways regimes is documented, not assumed.
- Deletion of any one feature changes model output by less than 2%: the edge lives in the set, not in the single golden column.
Feature selection done properly is an elimination tournament, not a popularity contest. XGBoost's built-in importance is the arena, but stability, causality and nested validation are who actually gets to hold capital.
