- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how local arithmetic errors arise, propagate, and can be corrected in multi-operand addition by LLMs. Using carry-preserving counterfactuals and causal interventions, the authors identify three stages of an error. First, weakened carry-sensitive computation can lead the model to generate a wrong digit. Second, once that digit is fed back into the model, Query-Key interactions in specific attention heads route the subsequent computation along an incorrect carry path. Third, even after following the wrong path, the model's downstream hidden states retain signals of arithmetic inconsistency.
Based on these findings, the authors propose Route+Entropy, a training-free correction method that uses internal carry routing to select candidate digits and late-layer predictive entropy to decide whether to replace the original output. On Qwen3-4B, the method improves exact-answer accuracy from 60.9% to 81.7%. The authors also examine the mechanisms and correction performance on other Qwen and Gemma models.
Critical strengths
1. Strong causal analysis of the error mechanism. The paper goes beyond observing correlations in hidden states and predictive entropy. The authors construct controlled counterfactual examples and use activation patching, attention-head analysis, and Query-Key edge interventions to identify components involved in error propagation. In particular, intervening on just two QK edges nearly reproduces the effect of replacing the full attention patterns. This provides convincing causal evidence for the proposed carry-routing mechanism.
2. A useful delayed error signal. An interesting finding is that the model retains information about whether a generated digit is correct, even after it has committed to an incorrect continuation. This helps explain why errors can propagate despite the presence of internal signals that distinguish correct and incorrect computation paths. The finding is also practically useful: the proposed training-free correction method improves exact-answer accuracy from 60.9% to 81.7% on the main model.
Critical weaknesses
1. Limited generality of the mechanism. The analysis focuses mainly on off-by-one carry errors in multi-operand addition, with the most detailed causal experiments conducted on specific attention heads in Qwen3-4B. Although the authors include experiments on other models and identify analogous mechanisms across architectures, it remains unclear whether the findings extend to reasoning errors beyond arithmetic. There is also a potential confounding factor in the construction of Case C. The authors require the model to naturally generate the correct digit on the modified problem. This selection criterion may introduce differences in the model's original prediction confidence, making it harder to determine whether the observed compatibility signals are independent of confidence differences.
2. Limited additional benefit and unclear computational efficiency of the correction method. Although Route+Entropy substantially improves over the original model, its advantage over Entropy-only is less consistent. The gain in exact-answer accuracy is 3.8 percentage points on Qwen3-4B and 7.9 points on Gemma-3-4B, but only 0.4 points on Qwen3-30B and 0.1 points on Qwen3.5, with no improvement on Qwen3.8. These results raise questions about how much the routing mechanism contributes to correction performance across different models. The method also requires additional forward computation for multiple candidate digits. Without comparisons under matched inference budgets, it is difficult to assess its efficiency relative to other inference-time correction methods.
Questions for authors
1. Case C requires the model to naturally generate the correct digit on the modified problem. Could the authors control for prediction confidence or logit margins at the target position to check whether the downstream entropy differences are partly explained by this selection criterion?
2. Route+Entropy provides only small improvements over Entropy-only on several models. Could the authors explain why the benefit of routing varies across models? A comparison under matched inference time or compute budgets would also help clarify the practical contribution of the routing mechanism.
5 comments
Q3. Model-Dependent Correction Gains
Entropy-only already shares the mechanism-guided entropy readout layers (Sec. 6 and Appendix E.4), and routing/recurrent proposals additionally restrict the candidate set. We analyze this contribution on matched decoding states from five 1,000-question test cohorts.
Table R8. Proposal quality and incremental gains over Entropy-only.
Model Ques Acc Qwen3-4B +3.8% 95.74% 2.62% +23 +25 Qwen3-30B-A3B +0.4% 80.41% 0.76% −2 +3 Gemma-3-4B +7.9% 94.77% 0.87% +63 +20 Qwen3.5-4B +0.1% 88.73% 11.27% 0 −1 Qwen3.8-27B 0.0% 90.18% 9.82% −12 +11 Ques Acc is the end-to-end question-accuracy gain from Table 11. On aligned natural off-by-one first errors, is the fraction proposing the GT neighbor (unresolved counted as misses), while is the fraction where candidate restriction excludes GT. At matched decoding states, counts additional first-error repairs (Route minus Entropy-only), and counts harmful edits avoided on originally correct digits (Entropy-only minus Route).
The key factor is the reliability and coverage of candidate restriction. Reliable routing can help entropy-based correction both recover errors and preserve correct digits. However, incorrect restrictions may exclude GT, while unresolved routing limits how often the mechanism can contribute.
Gemma and Qwen3-4B improve error recovery and correct-digit preservation simultaneously (Table R8). Qwen3-30B is mainly limited by unresolved routing, whereas Qwen3.5 shows nearly offsetting gains and losses, and Qwen3.8 trades fewer repairs for better preservation.
Architecturally, Qwen3-30B's MoE pathways may make fixed-head routing preferences less consistently resolved, suggesting a coverage bottleneck. In contrast, Gated DeltaNet derives directional proposals indirectly from recurrent innovation energy rather than explicit QK preferences; our implementation always issues a direction, so unreliable proposals may exclude GT. These architectural explanations remain hypotheses, but suggest that routing-signal reliability and coverage govern its incremental value over entropy selection.
Q4. Computational Overhead
As stated in Appendix E, our correction experiments serve as functional validation of the identified mechanism, rather than a practical arithmetic-solving method. For real-world usage, using an external calculator is clearly preferable. We nevertheless report latency and FLOPs in the following table for completeness, without claiming computational efficiency or deployment advantages.
Table R9. Inference-time accuracy and computational cost.
Method Latency (ms) GFLOPs Ques Acc (%) Original 450.08 747.94 60.9 Re-prompting 490.32 1436.82 61.4 Steering 473.57 756.50 63.5 Target-state 465.00 748.00 66.4 Dual-stream 483.74 748.01 59.3 Next-state 599.11 759.75 84.4 Entropy 1395.11 935.78 77.9 Route+Entropy 1390.90 935.78 81.7
Dear Reviewer,
Thank you again for your constructive feedback. We have updated our manuscript to incorporate additional analyses including L24 carry-sensitive heads, causal connection between generation and routing, cross-operation generalization and so on, as discussed in our rebuttal. We would greatly appreciate it if you could take another look.
If you feel that these revisions have adequately addressed your concerns, we would be very grateful if you would consider updating your assessment and score. Should you have any further questions or concerns, we are happy to address them and remain available for further discussion.
Thank you once again for your time and thoughtful consideration!
Thank you for your time and constructive feedback! We have carefully addressed your comments below, and are incorporating some of these new results into our manuscript, which we look forward to sharing soon.
All newly added figures are prefixed with "R" (e.g., Fig. R1) for distinction, which can be viewed in the Supplementary Material.
Q1. Generalization Beyond Arithmetic Reasoning
Q2. Target-Position Confidence Confound