The reward does not lower the Martingale Score it is built on, and Algorithm 1 does not optimize the objective the paper states, so nothing in the paper shows that the proposed idea causes the reported gain.
1. The paper's one new idea, training on the Martingale Score, is not shown to work, so the paper adds little beyond [1].
[1] introduces the score and its interpretation and already proposes training on it as future work, with a test: such training should lower both the Martingale Score and the Brier score.
On the distilled model, the runs without labels lower only the Brier score.
The Martingale Score barely changes on Llama-3.1-8B and rises on Qwen3-32B (Table 1), and it falls only in runs that use the supervised Brier reward.
So the training fails the test of [1]: the reward does not lower the score it is built on.
No run removes the martingale reward or replaces it with a shuffled or random reward while keeping the information, KL and length terms, so those terms may cause the Brier gain instead.
No run compares the product reward with a simpler form, such as a direct penalty on the predicted part of the update, and the paper does not say why it chose the product over such a form.
Without these controls, the results cannot be attributed to the martingale reward and are hard to trust.
2. The training objective is not what the paper says it is, and the manuscript was not edited for outside readers.
Section 3.3 states that maximizing the reward "is equivalent to minimizing MS", the slope $\hat{\beta}_1$ of the regression of belief updates on priors.
Minimizing MS is an objective on $\hat{\beta}_1$ alone, and Algorithm 1 optimizes a different quantity.
Its reward is minus the product of each update $\Delta_i$ and the update $\hat{\Delta}_i$ that the regression predicts from its prior.
Over a batch these products sum to $\sum_i \hat{\Delta}_i^2$, because a regression's residuals are uncorrelated with its predictions.
Each $\hat{\Delta}_i$ is the mean update plus $\hat{\beta}_1$ times the prior's distance from the mean prior, so the batch reward is $-n(\bar{\Delta}^2 + \hat{\beta}_1^2 \operatorname{Var}(b_{\text{prior}}))$, where $\bar{\Delta}$ is the mean update.
Unlike an objective on $\hat{\beta}_1$ alone, this reward has three effects:
- It penalizes a negative $\hat{\beta}_1$ as much as a positive one, so maximizing it moves MS toward zero, although Section 5.1 says that rational updating by a systematically overconfident agent gives a negative MS, and footnote 1 describes the distilled model as overconfident.
- It penalizes the mean update $\bar{\Delta}$, which MS does not involve.
- It increases when the priors are less spread out, even if $\hat{\beta}_1$ stays the same.
Section 3.3 adds the information term so that the model cannot satisfy the objective "by never updating its belief from the prior", but no term in the paper guards against the route that the third effect opens: the model can raise the reward by stating less spread-out priors, with MS unchanged.
The information term is not defined consistently.
Algorithm 1 adds it to the loss as a function of the sampled updates, which do not depend differentiably on the model, so it contributes no gradient and does not affect training.
Appendix A instead includes it in the reward as a penalty, where it does affect training, so the two descriptions train different objectives.
The KL reference model for the distilled runs is not named.
If it is the undistilled model, the KL term pulls the distilled model back toward it, which alone could restore calibration.
Algorithm 1 also marks PPO clipping and a length penalty as optional, and apart from the length penalty in the two-agent training (Section 3.4), the paper does not say which each run used, so the objective each run trained is unknown.
The manuscript was also not edited for outside readers:
- The data splits are given as "held-out splits as noted per run (run-ids in the project log)" (Appendix A).
- Code names appear beside or in place of definitions, such as BRIER_FROM_STATED (Appendix A) and "INSTR_SYC instruction" (Table 2 caption).
- Labels appear without introduction: "C1 base models" and "C3 instructor" with no C2, "the s0 mean stated-p" (footnote 1), and "(training seed)" attached to both a judge model and a prompt.
- The top panel of Figure 2 is titled "(Mistral-7B base)", a model used in no experiment the paper describes.
- The bottom row of Figure 2 is a raster image of four raw diagnostic plots (MS_coef, meanP, options[0]) whose text cannot be read at print size.
- The same row calls the distilled model "B", the letter Table 1 uses for the Brier reward.
- Section 3.4 gives settings as command-line defaults, "(default: deepseek)" and "(default: 3)".
- One model is called "deepseek", "deepseek-chat", "deepseek-chat-v3" and "DeepSeek".
- Text in Figure 1 is also too small to read at print size.
- The lower quarter of page 6 is blank.
The related work cites [2] for "under what conditions sycophantic behavior arises" (page 2), but [2] studies models' reluctance to express uncertainty and does not mention sycophancy.
The AI use statement says that "references were compiled and checked manually, and all text underwent extensive manual editing and review" (page 8).
The listed items and this citation error are each minor, but together they contradict that statement and show that the reported numbers were not checked closely enough to be trusted.
References
[1] Z. He, T. Qiu, H. Shirado, and M. Sap. Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning. NeurIPS 2025. arXiv:2512.02914.
[2] K. Zhou, J. D. Hwang, X. Ren, and M. Sap. Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty. ACL 2024.