How We Backtest an NBA Model Without Fooling Ourselves

Almost every sports model looks brilliant on the games it was built from. The only accuracy number worth publishing is one measured on games the model could not have seen. That is what a walk-forward backtest does, and it is the standard every number on HoopVantage has to meet.

Rule 1: only the past

We replay every NBA game since 2015-16 in date order. Before a game is projected, the model's state contains only games played on earlier dates: player box-score totals, minutes, on/off margins and team efficiency. The result of the game, its box score and the minutes each player actually played are folded in only after every game that day has been projected.

Rule 2: fit on earlier seasons only

The weights that turn ratings into a projected margin are refit before each test season using only earlier seasons. The 2025-26 projections were produced with weights that had never seen a 2025-26 game.

Rule 3: choose settings before looking at the test seasons

Choices such as how fast old games fade or how strongly small samples are shrunk were made using 2016-17 through 2021-22 only, then frozen. The four test seasons (2022-23 to 2025-26) were scored once with those settings. When a tweak helped the test seasons but not the training seasons, we did not keep it.

What we measure

• Straight-up accuracy: how often the projected winner won. Around 66-70% is realistic in the NBA; anything claiming 80% is not measuring honestly.
• Margin error (MAE): the average miss on the final margin in points.
• Brier score and log loss: whether the probabilities themselves are good, not just the picks.
• Calibration: when we say 70%, does that team win about 70% of the time?

Rule 4: no hindsight about who plays

An earlier version of this backtest let the model see which players actually played in each game — information nobody has before tip-off. The headline numbers now use only what was public beforehand: the official NBA injury report posted before each game (archived by the league since 2020-21) and how many games each player had just missed, with each status's real chance of playing learned from earlier games only. It turned out to be almost as good as knowing the lineup in advance, and clearly better than ignoring availability. We still publish both of those versions next to the headline so the size of the effect is visible. The full results are on the backtest page, and every live projection is graded on the track record.

📚 More Articles