The problem
Most public "betting model" projects report a profitable backtest. Almost none of them are trustworthy, because the same handful of mistakes inflates every result: the model peeks at data it wouldn't have had at bet time, the strategy's parameters are tuned on the same seasons used to score it, and the sportsbook's commission gets quietly ignored. The interesting engineering problem isn't building a model — it's building a harness where those mistakes are structurally impossible, and then reporting whatever it says.
The approach
- Point-in-time data, enforced by the type system, not by discipline. Every price is served through a single as-of query over append-only snapshots, so the engine can only ever see what existed before tip-off. Final scores live behind a settlement-only interface that the bet-decision code physically cannot reach — lookahead bias is prevented by architecture rather than by remembering not to do it.
- Models written by hand. Logistic regression for win probability and ridge regression for point margin and game total, implemented directly in NumPy, with a hand-rolled normal CDF turning point forecasts into cover and over probabilities. Evaluation is on Brier score and reliability curves rather than raw accuracy, because bet sizing consumes probabilities — a model that is right often but confident wrongly is worse than useless.
- Devigging: recovering the fair price. A sportsbook's quoted odds include its margin on both sides of the market. A devigging module strips that margin out to recover the implied fair probability — the same idea as removing the bid-ask spread to get a fair mid-price — and bets are placed only where the model's edge over that fair price clears a noise threshold.
- Walk-forward validation with one shot at the holdout. Expanding-window walk-forward across 17 test seasons, retraining at every step and refitting the edge threshold on prior seasons only. The final holdout season was touched exactly once, after every parameter was locked, so the reported record is genuinely out-of-sample.
Architecture
- Store: append-only SQLite snapshots behind a single as-of query; settlement data isolated behind its own interface.
- Features: player-availability signals derived from 513K player-game rows ingested from the NBA stats API.
- Models: logistic and ridge regression implemented in NumPy, plus a hand-written normal CDF for cover and over probabilities.
- Execution: devigging, an edge threshold refit on prior seasons only, and flat or capped fractional Kelly staking.
- Evaluation: ROI against break-even, closing line value, a Sharpe-like ratio, max drawdown, and bootstrap confidence intervals — all built by hand.
What it found
On the single-shot holdout season, both markets lost money: −5.64% ROI across 923 spread bets and −8.61% across 954 totals, with 90% bootstrap intervals lying entirely below zero. That is a falsifiable negative result, and it is the honest headline: the NBA closing line is efficient against public box-score information.
Three follow-ups made the negative result mean something:
- Benchmarked head-to-head against the closing line on RMSE and log loss, then regressed realized outcomes on both forecasts with standard errors computed via Newton's method — isolating how much information, if any, the model added on top of the market price.
- Bounded the value of information the model could never have. A deliberately lookahead "oracle" feature established the ceiling: even perfect foreknowledge of nightly player availability would fall 0.08 points short of the 0.74 needed to break even. The strategy doesn't fail from a fixable data gap.
- Tested the market itself, with no model involved. Measuring favorite-longshot bias across underdog buckets with bootstrap intervals found it monotone in the textbook direction — but consistently smaller than the sportsbook's commission, so it isn't exploitable either.
Making it fast
- Replaced a per-game scan of the player-availability table with a binary-search as-of lookup, cutting feature construction from 328s to 2.2s per season — a 150× speedup.
- Vectorized the bootstrap resampler from 3.4 minutes to 9 seconds (23×) by recognizing both statistics as ratios of per-bet sums and drawing 10,000 resample indices in blocked NumPy passes. Full-run time dropped from 7 minutes to 2.
Proving it correct
- An end-to-end test suite over a synthetic fixture with known team strengths asserts as-of query behavior, devigging, settlement, Kelly caps, and a sign-convention check that requires the fixture's near-oracle line to beat the model.
- The engine was extended to live forward paper trading through the identical decision path as the backtest — live bookmaker prices are written into the same point-in-time store, and tests assert both paths produce identical ledgers.
- The hand-written logistic regression was cross-checked against scikit-learn (mean absolute probability difference 0.0012, correlation 1.0000). Platt scaling was skipped after reliability curves showed no material miscalibration.
- A data-integrity guard layer — duplicate detection, snapshot checks, per-season sanity bounds — surfaced a corrupted final score in the source feed.
Stack
- Python
- NumPy
- SQLite
- Logistic regression
- Ridge regression
- Walk-forward validation
- Bootstrap CIs
- Fractional Kelly