Classification Metrics

  • Accuracy: Overall correct predictions
  • Precision: True positives / all predicted positives
  • Recall: True positives / all actual positives
  • F1-Score: Harmonic mean of precision and recall

Regression Metrics

  • MAE: Mean Absolute Error
  • RMSE: Root Mean Squared Error
  • R-squared: Variance explained

Financial Metrics

  • Sharpe Ratio: Risk-adjusted returns
  • Sortino Ratio: Downside risk-adjusted
  • Maximum Drawdown: Worst peak-to-trough
  • Calmar Ratio: Return / Max Drawdown

Choosing Metrics

For trading: Sharpe ratio and maximum drawdown matter more than raw accuracy.

Choosing the Right Metric for the Right Problem

A machine learning model is only as good as the metric used to judge it, and in finance the wrong metric can show a model that fails live while shining in the backtest. Evaluation starts by matching the metric to the problem type: classification metrics judge categorical predictions, regression metrics judge continuous values, and financial metrics add the costs and asymmetries that markets impose. Selecting the wrong family produces a model optimised for the wrong thing.

The deeper principle is that a metric must reflect the real consequence of being wrong. In trading, a missed trade and a bad trade are not equally costly, and a metric that weights them equally misleads. Metrics such as accuracy, which treats every correct answer the same, frequently overstate the value of a model on imbalanced financial data where profitable moves are rare.

Classification Metrics Explained

  • Accuracy: the fraction of correct predictions, misleading on imbalanced classes.
  • Precision: of predicted positives, how many were actually right, controlling false alarms.
  • Recall: of actual positives, how many were caught, controlling missed trades.
  • F1 score: the harmonic mean of precision and recall, balancing the two.
  • AUC-ROC: ranks the model's ability to separate classes across thresholds.

Regression and Time-Series Metrics

For continuous predictions such as future returns or volatility, mean squared error penalises large errors heavily, while mean absolute error is more robust to outliers. In financial series, however, the direction of a prediction often matters more than its exact size, so additional metrics such as directional accuracy, the fraction of time the sign of the forecast matches reality, become the real measure of value. Out-of-sample testing on a strict holdout set is essential, because in-sample fit on noisy markets is meaningless.

Financial Metrics That Capture Real Costs

Beyond standard metrics, financial evaluation must add simulated P&L. Turn the model's predictions into trades, charge realistic transaction costs and slippage, and measure the resulting Sharpe ratio, maximum drawdown and win rate. Two models with identical accuracy can have wildly different trading value once costs and drawdowns are applied. A slightly less accurate model that makes fewer, higher-conviction trades can vastly outperform a busy model that churns the account.

The Traps of Evaluation on Financial Data

  1. Evaluating on the same data used for training, producing optimistic but useless scores.
  2. Ignoring look-ahead bias, where future information leaks into the features.
  3. Using accuracy on imbalanced data where a model that always predicts "no move" looks great.
  4. Optimising a metric that rewards churn without charging realistic frictional costs.

Walk-Forward Evaluation as the Gold Standard

The most trustworthy financial evaluation is walk-forward testing. Train the model on a rolling window, predict only the next unseen period, then advance the window and repeat. This simulates how the model would behave in production, where it predicts data it has never seen. Comparing walk-forward performance with in-sample performance reveals overfitting: if the model shines during training but collapses out-of-sample, the edge was memorised rather than learned.

Combining Metrics Into a Decision

No single metric tells the whole story. A complete evaluation pairs a broad metric such as AUC with a financial metric such as net Sharpe, and layers on stability tests across regimes. Look for a model whose edge persists out-of-sample on the financial metric, not merely one that scores well on a convenient statistic. The final question is always practical: after costs and drawdowns, does the model make money reliably, not just before, but after surviving the reality of the market?

Choosing Metrics for Trading Decisions

Match the metric to the decision: accuracy for balanced classification, recall for catching true signals, PPV for surgical entries, and MSE purely to monitor model drift rather than to size positions. Plot the ROC curve alongside the gain chart to judge whether the ranking is useful at the operating threshold. Log every metric historically so model decay shows up as a trend, not a surprise.

Calibration Before The Trade

Check whether the model's 60% probability scores are actually realised 60% of the time by bucketing predictions and comparing to observed outcomes. Ignore a high AUC if the operating region near 0.5 is miscalibrated, because that is precisely where trading decisions live. Calibration, not raw discrimination, decides whether a model's confidence is actionable in the option chain.