- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies improvements in LLM reasoning obtained by inserting repeated, semantically uninformative tokens between the system prompt and the question. It approximates their first-layer attention effect using scaling and additive contributions and associates this intervention with fewer near-zero MLP activations. Motivated by these observations, ARM directly applies bounded, sign-preserving outward shifts to selected post-nonlinearity activations in the first-layer MLP. The paper reports improvements across six models and mathematical, scientific-question-answering, and coding benchmarks.
Critical strengths
- Interesting research question. Using training-free methods like inserting uninformative tokens / ARM to redistribute activation / attention is a interesting trial.
- Extensive experiments. Table 2, Table 13 and Table 15 have demonstrate the effectiveness of method across different models, benchmarks and pass@k.
- Well-motivated method design. The observed activation redistribution motivates ARM as a direct internal intervention, providing a useful connection between the token-insertion phenomenon and training-free reasoning improvement.
Critical weaknesses
- The performance degradation with overly long meaningless-token sequences remains insufficiently explained. The authors attribute it to accumulated perturbations in Section F.1 and by Eq 3, but provide no comparison of attention outputs across insertion lengths to support this hypothesis.
- The evaluation of ARM focuses on relatively older model families, leaving its effectiveness on more recent, stronger reasoning models unclear.
Questions for authors
- While ARM improves performance, how it changes generation behavior and how these changes relate to the gains remain unclear. Although the authors have discussed mless's influence on termination rate in F.1 and ARM's influence on 2-gram diversity in Fig 13, I am still curious about the influence on output length.
3 comments
Thank you for your response. I will keep my score.
Dear Reviewer mYrz,
Thank you for your positive feedback and for acknowledging our efforts to address your concerns. We are glad that our responses have adequately resolved your major concerns about generalizability of new model family, output length and further mechanism explaination. Given that your main concerns have been addressed, we would sincerely appreciate it if you could consider updating your rating accordingly.
Thank you again for your time and constructive feedback.
Reply(1/1)
We thank the reviewer for their insightful questions and constructive feedback.
As the number of inserted tokens increases, these tokens receive a larger share of attention, while the original input receives a smaller share. With a moderate number of inserted tokens, this change can produce the beneficial activation redistribution discussed in Section 3. However, with too many inserted tokens, the attention output may deviate significantly from its original representation, making information from the original question less influential. Consequently, excessive insertion may interfere with the model's reasoning process rather than improve it.
We evaluated ARM on Qwen3.5-4B using MATH-500 and AIME 2025, with a maximum output length of 32,768 tokens. For each benchmark, we conducted three runs with different random seeds and report the mean accuracy and standard deviation. The results show that ARM also improves performance on Qwen3.5-4B, extending our evaluation to a more recent model.
We report the average output length for Qwen2.5-Math-1.5B and Qwen2.5-7B-Instruct on MATH-500 and AIME 2025. ARM produces outputs of similar average length to the vanilla models, suggesting that its performance gains are not primarily driven by generating more tokens.