- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how autoregressive models represent numbers split across multiple tokens. It proposes a Place-Scaled Helix, where the period of each token’s helical representation scales with its numerical place. Experiments across LLMs, VLMs, and different tokenizers, together with activation interventions, support the proposed structure and its causal relevance.
Critical strengths
- Important research question. The paper extends the study of numerical representations from single-token to multi-token numbers, making the analysis more relevant to modern language models.
- Causal evaluation. Beyond measuring correlations, the authors intervene on model activations to test whether the proposed representations influence numerical outputs.
Critical weaknesses
- The scope of generalization needs clarification. The study focuses on arithmetic tasks, so it remains unclear whether the same structure applies to decimals, negative numbers, or numbers in broader language contexts.
- The causal interpretation needs more support. A last-token intervention failing to change other digits does not necessarily show that the last token has no causal role, especially if those digits were generated earlier.
- The manuscript lacks an AI Use Statement. The authors should add this section to disclose any use of AI tools, following the venue’s requirements.
- More related-work discussion. It would be better to discuss relevant work on interpreting arithmetic computation, e.g., ''Interpreting and Improving Large Language Models in Arithmetic Calculation'' and ''Case-based or Rule-based: How Do Transformers Do the Math?'' Comparing the findings with the proposed place-scaled representation would help clarify the paper’s novelty and its connection to arithmetic mechanisms.
Questions for authors
Refer to weakness.
3 comments
Gentle Reminder
Thank you for your constructive feedback. We have addressed all the concerns raised in your review. We would appreciate it if you could review our rebuttal and consider revising your score if you find our concerns satisfactorily addressed. Just a gentle reminder that the Author-Reviewer discussion deadline is in 2 days (18th Nov AoE).
Thank you for the clarifications. I will keep my current score for two reasons. First, focusing only on natural numbers in arithmetic narrows the paper's overall contribution. Second, the related-work discussion remains somewhat brief and would benefit from a more thorough comparison with both suggested papers to better clarify this work's novelty.
Rebuttal Response 1
(1) “scope of generalization needs clarification”:
We thank the reviewer for raising this point. As mentioned in
Section 3.1 [Line 145], we analyze only natural numbers and extend prior work to multi-token numerical representations, which remain largely underexplored and are crucial. Other number settings are an important direction for future work, as acknowledged inSection 7 (Future Work)(2) “causal interpretation needs more support”:
We apologize for the confusion. As the reviewer correctly points out, intervening on output tokens cannot affect digits that have already been generated. This is exactly why we perform causal steering on the input operand (a) and measure its effect on the subsequently generated output digits, as shown in
Fig. 2 (right)andTable 5.To further emphasize that we need to use all tokens instead of the last token alone, we performed linear probing on the last token to recover the entire number, but we can't steer all digits using the last token, as shown below (results are with Molmo2 in addition task)
This is shown in
Fig. 2 (Right): steering the last token in the operand only changes the corresponding digits (1 digit in the single-digit tokenizer and 3 digits in the multi-digit tokenizer). We can add 1 more digit if there’s a carryover.(3) “manuscript lacks an AI Use Statement” :
We apologize for the omission and thank the reviewer for highlighting this requirement. We will add it in the revised manuscript. We generated and verified the evaluation datasets and carefully designed the experiments. AI was used only minimally to rephrase the manuscript text.
(4) “More related-work discussion”:
We thank the reviewer for suggesting these relevant works on arithmetic computation. Our work provides complementary representation-level explanations to their findings.
As shown in
Fig. 4, numerical value representations become increasingly monotonic in later layers, coinciding with the emergence of the Place-Scaled Helix structure.Fig. 7 (bottom)shows high intervention accuracy (IIA) when the helix forms.This explains the finding in ''Interpreting and Improving Large Language Models in Arithmetic Calculation'', where answers are processed in later layers, with earlier layers processing the operands and operator with the help of formation of place-scaled helix structures at respective layers.