acceptodds
Under review as a conference paper at ICLR 2027

Scalar Recalibration Can Overstate Contextual Gains in Pairwise Prediction

Abstract

Does a contextual predictor add value if its advantage survives recalibration of a simpler model? In pairwise prediction, this conclusion can depend on whether the calibrator observes which competitor occupies a known side. An ordinary Platt intercept in blue-first or white-first coordinates is exactly a side-dependent intercept in randomly ordered coordinates. We use this identity to compare ordinary and side-aware Platt scaling while holding each model's forecast stream fixed. Across seven annual League of Legends splits and nine weekly Lichess splits, covering 70,670 and 1,340,775 evaluation matches, respectively, an online side term improves a calendar rating model's log loss after ordinary Platt scaling by 0.001144 and 0.000119. After both readouts receive side information, the differences reverse to +0.000139 and +0.000013, with 95% paired calendar-block intervals of [-0.000087,0.000325] and [0.000004,0.000024]. A four-arm League of Legends ablation also finds no proper-score benefit from the existing frozen rank-five interaction term; shared and independent validation selection yield identical forecasts. These results do not establish equivalence or rule out richer contextual models. They show that survival under scalar recalibration is insufficient evidence of contextual value beyond a simpler predictor with the same side information.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
probability calibration, pairwise prediction, skill rating, contextual modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ad8UOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

From authors: "Survival under scalar recalibration is insufficient evidence of contextual value beyond a simpler predictor with the same side information." "The contribution is a documented empirical failure of an attribution criterion, rather than a new calibration algorithm."

The authors argue that calibration depends on the information available to the recalibration step (e.g., the context, the order, etc.). They should this impact with examples in two real-world datasets.

Critical strengths
  • The authors explore calibration and the impact of certain system characteristics on its calibration, which is a relevant topic.
  • The authors test it on real datasets where such a challenge might exist.
  • The authors apply some data augmentations (i.e., transformations, sampling).
Critical weaknesses
  • The main claims, goals, and guarantees are unclear. The paper reads more as a high-level suggestion to be careful about the context used in calibration, based on empirical results from two datasets. It offers little substance on what to do, how to measure the effect, or how to identify it, how to improve it.

  • In particular, the conclusion that calibration depends on the information given seems somewhat obvious.

Questions for authors
  • Have the authors considered using synthetic datasets to evaluate this challenge more broadly?
  • Isn't the canonical swap a type of data augmentation? Is this already used by literature? What is the difference?
  • Regarding the weakness above: the conclusion that calibration depends on the information given seems somewhat obvious. Could you explain, concisely and in plain English, what I might be missing here?
Reviewer_6x79OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the gains from adding context to a prediction model could be had just by adjusting the probabilities of a simpler model. It looks at pairwise outcomes in League of Legends and chess, where the context is which competitor plays the blue or white side. The authors show that ordinary Platt scaling can capture this information when the competitor on that side is always listed first, but not when the order is randomized. So they compare ordinary and side-aware calibration while keeping each model's underlying predictions fixed. In both datasets, the apparent benefit of explicitly modeling side goes away (in average score) once both models get side-aware calibration. A further experiment finds no benefit from the specific frozen interaction model they tested. The main point is that an improvement that survives ordinary recalibration isn't enough to show that a contextual model adds anything beyond a simpler model given the same information.

Critical strengths

The paper points out an issue that's easy to miss: how the competitors are ordered changes what information the calibrator has. The authors keep the prediction streams fixed and compare calibration methods with the same number of parameters, so changes in the results can be traced to the calibration setup rather than to retraining or extra parameters. The equivalence between canonical ordering and side-aware calibration is also explained clearly.

The authors test the finding in two domains, include an Elo comparison, and separate side effects from the frozen interaction component. The main experiments use separate periods for tuning and calibration, and the results come with paired uncertainty estimates and sensitivity checks.

Critical weaknesses
  1. The paper says outright that it offers an evaluation lesson, not a new calibration method. Both domains test a single binary side variable, and the main mathematical result shows how that variable can be captured by shifting the intercept. The experiments do support the stated conclusion, but it would be nice if the authors can say more clearly which current evaluation practices or published claims this finding calls into question, beyond the historical archive. That would make it easier to see what the paper adds on top of the discussion of conditioning information already in the related work.

  2. The chess experiment uses a short, subsampled history, drops draws, and doesn't use site ratings. The calibrated calendar model gets about 52.3% accuracy. A simple baseline that only uses side would show how much of the predictive signal comes from player histories versus the general white-side advantage. Separately, the interaction component is trained without the skill model and then frozen, and some of the chosen configurations shrink its contribution to almost nothing. So its negative result tells us little about stronger contextual models. I see that the authors do acknowledge these limits, but the question of when side-aware calibration would stop being enough is left mostly open.

Questions for authors
  1. See weaknesses
  2. All seven League of Legends folds pick the smallest tested period scale and side learning rate. Could widening these ranges change how the side model compares to the backbone?
  3. The contextual model updates its side coefficient over time, but the calibration coefficients stay fixed during evaluation. What do the estimated side trajectories say about why a fixed calibration correction is enough on average? I was wondering do periods where the side advantage changes more show different results?
Reviewer_2ZJ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether contextual gains in pairwise prediction remain after both models receive the same side information during calibration. Across LoL and Lichess, the apparent advantage mostly disappears or reverses after side-aware recalibration. The paper argues that gains surviving ordinary calibration alone are not enough to establish contextual value.

Critical strengths

Important and easy-to-miss evaluation issue. The fixed-stream setup gives a clean comparison without retraining the underlying models. Evaluation covers two domains and several robustness checks. Limitations are discussed clearly.

Critical weaknesses

Conclusions depend on the specific forecast streams and calibration models tested. The uncertainty intervals do not include training or model-selection uncertainty. The interaction ablation is fairly narrow, so it does not rule out useful interaction effects more generally.

Questions for authors

Do the conclusions hold with other reasonable calibration functions? How sensitive are results to the calibration penalty and constraints? Do conclusions change when model selection and calibration are repeated during resampling? Could the interaction result change if the interaction were jointly trained rather than frozen? How sensitive are the Lichess results to excluding draws?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.