- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a test-time adaption method for improved numerical predictions. The authors first present the numerical manifold hypothesis which describes the angular relationship between numeric output directions, and empirically validate it across multiple models. Based this hypothesis, the authors propose to use the angular dispersion of the numerical prediction as a test-time objective for improving numerical predictions. An intuitive explanation for the gradient update is provided through the relation between change of output token probabilities and change of angular dispersion. The authors compare their method to alternative decoding and test-time adaptation strategies across both VLMs and LMs on tasks focussing on number understanding and mathematical reasoning and show its empirical performance.
Critical strengths
- The paper introduces an interesting test-time adaptation method that goes beyond entropy signals by incorporating the geometric properties of number representations instead. The proposed method thus also goes beyond simple distance-based priors often enforced in previous work, which allows to capture a richer structure in the representations.
- Empirical evaluation is shown across a variety of numerical, mathematical reasoning, and visual tasks and the proposed method seems to consistently improve over most baselines.
- The authors critically discuss the method's limitations in the form of potential miscorrections (Figure 7).
Critical weaknesses
- The method is not guaranteed to improve prediction accuracy, as it may also enforce overconfidence for some experiments. While the method seems to improve overall prediction quality as empirically shown, it would be insightful to also understand to which degree the overall predictions change, e.g. by quantifying how often the described failure modes occur.
- The need for multiple forward and backward passes during the weight updates appears to be the main computational bottleneck of the proposed method. Consequently, the latency of the method does not change much even for very sparse weight updates. While the latency is lower than full chain-of-thought reasoning, it still remains behind alternative inference methods and it remains unclear if it can be improved further.
Questions for authors
- The paper discusses the possibility of miscorrections as a potential failure mode, where correct predictions are changed to incorrect ones. How often does this failure mode occur in the presented setups?
- The statement that numerical structure is not only found in input embeddings, as often described in previous work, but also contained in numeric output directions appears of particular importance for models using weight sharing between input embeddings and output weights. Does the method show performance differences between models using weight-sharing and other models?
- The method's performance seems to depend highly on the temperature parameter. What is a practical way of choosing this hyperparameter?
- To clearly highlight the difference between entropy minimisation and the proposed angular dispersion objective, it would be helpful to apply the entropy minimisation method only on the numerical tokens in a further ablation.
4 comments
Response to Reviewer 8VDe (Part 2/2)
Q3 The method's performance seems to depend highly on the temperature parameter. What is a practical way of choosing this hyperparameter?
We appreciate your attention to this practical aspect of the method and agree that guidance on choosing the temperature would make NuRA easier to use across models and tasks. NuRA uses the same temperature transformation as softmax sampling, with γ = 1/T, but applies it to the numerical distribution used for adaptation without rescaling the final decoding logits. We believe one practical guide is to consider how concentrated this distribution is before adaptation, using a smaller γ when one numerical token dominates and a larger γ when probability is spread too evenly across numerical tokens.
When the distribution is very sharp, other numerical tokens receive little weight, so their geometric relationships have limited influence on the objective. A smaller γ gives these alternatives more weight, allowing adaptation to make greater use of the numerical geometry. Too much smoothing, however, weakens the model's preferences among numerical tokens, and a very small γ also reduces how strongly the adaptation weights respond to changes in the logits. We find it helpful to view the choice of γ as a balance between preserving these preferences and allowing plausible alternatives to contribute to the update.
This perspective is consistent with our observations across models and tasks. Numerical predictions in mathematical reasoning were often more concentrated, and we use γ = 0.3 for Qwen3 on the reasoning tasks, compared with 0.6–0.8 on direct numerical prediction. We also observed sharper answer distributions in DeepSeek-R1-Distill-Qwen-1.5B, for which we use γ = 0.125 for reasoning and 0.25 for direct numerical prediction. These observations suggest that models within the same family and related tasks can share patterns in prediction concentration, which may provide a useful starting point when choosing γ for a new setting. We have added this practical guidance and the complete configurations to Appendix C, together with sensitivity results under the main adaptation protocol in Appendix F.5.
Q4 To clearly highlight the difference between entropy minimisation and the proposed angular dispersion objective, it would be helpful to apply the entropy minimisation method only on the numerical tokens in a further ablation.
Thank you for this helpful suggestion. We compared the two objectives on the same numerical tokens with the adaptation settings fixed, and NuRA achieved 0.75–2.71 percentage points higher answer accuracy across all four settings. We have added this ablation to Appendix C of the revised manuscript.
W2 The need for multiple forward and backward passes during the weight updates appears to be the main computational bottleneck of the proposed method. Consequently, the latency of the method does not change much even for very sparse weight updates. While the latency is lower than full chain-of-thought reasoning, it still remains behind alternative inference methods and it remains unclear if it can be improved further.
Thank you for highlighting this limitation. We agree that sparse updates provide only modest latency savings because forward and backward computation is still required. Our sparsity experiments examine how few parameters need to change to improve numerical prediction, and we find that updating a small, gradient-selected subset retains most of the token-accuracy gain from full updates, suggesting that existing numerical representations can be adjusted while leaving almost all pretrained weights unchanged.
We appreciate your emphasis on efficiency and have explored several alternatives, including updating only the final MLP layers, directly optimizing the final hidden state, and updating normalization parameters following the approach of TENT (Wang et al., 2021). Table 4 reports the final-four-layer comparison, and Appendix F.1 reports the normalization-only results. Although these alternatives did not match the accuracy gains of our selected MLP updates, improving efficiency remains an important goal. We believe that stopping adaptation once the numerical prediction distribution stabilizes is a promising direction, and we will explore the resulting accuracy–latency trade-off in future work. We have expanded the discussion of computational cost in Appendix H.
References
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. (2021). Tent: Fully Test-Time Adaptation by Entropy Minimization. ICLR.
Thank you for your elaborate rebuttal and clarifications, particularly regarding the amount of miscorrections as a potential failure mode. I believe this adds a valuable analysis of the method to the paper.
Thank you for your feedback
Thanks again for taking another look and for your support of our work. We're really glad the miscorrection analysis was useful, and your feedback helped us strengthen the paper.
Response to Reviewer 8VDe (Part 1/2)
We sincerely appreciate the time and thought you have devoted to reviewing our work. Your recognition of the numerical geometry and empirical results is encouraging, and your questions have helped us examine the method’s behavior and practical use more closely. We address each point below with additional experiments and clarification.
We welcome the opportunity to discuss this practical question in more detail. We compared unadapted greedy decoding with NuRA on the same questions from Tables 1–3. Across the three task groups, 77 of 4,050 initially correct answers became incorrect (1.90%), while 293 of 5,436 initially incorrect answers became correct (5.39%). The table reports both directions of change, with each rate calculated over the corresponding initially correct or incorrect predictions. Net gain is the number of corrections minus miscorrections; the token row covers aligned final-answer token positions for direct numerical prediction only.
Your suggestion helped us assess the gains alongside the risk of changing correct answers. NuRA corrected 5.39% of initially incorrect answers while changing 1.90% of initially correct answers to incorrect ones, with corrections outnumbering miscorrections in every task group. These rates show how adaptation recovers errors while largely preserving correct predictions. We have added this paired analysis to Appendix D.4.
Thank you for raising this distinction and suggesting an interesting direction to explore. We compared tied and untied models within the Qwen3 and Llama families, and the table reports their gains over greedy decoding in average answer accuracy (percentage points) from Table 1 and Appendices D.1–D.2.
The similar gains on Qwen3-4B and Qwen3-8B in direct numerical prediction (+2.04 and +1.99 points) suggest that NuRA's benefits extend across both tied and untied models.
We also explored input embeddings as geometric priors in the two untied models. The table reports Ans. Acc. / Token Acc. (%), averaged across the six direct numerical prediction datasets, with the adaptation columns indicating the source of the geometric prior.
Both sources improve numerical prediction, with output head weights giving slightly higher answer and token accuracy in both models. This suggests that useful numerical geometry is present on both sides, while output head weights provide a prior directly connected to the logits used for prediction. We have added these comparisons to Appendix F.4.
Our response continues in Part 2.