- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether language models can perform effective computation without interpretable CoT. The author evaluates fixed, task-agnostic "filler sequences" applied to tasks involving synthetic multiplication, arithmetic operations, and variable counting, examining how their effectiveness is influenced by token type, few-shot context and the model.
Critical strengths
The paper addresses a perspective previously unexplored in the literature, investigating the impact of filler words on performance across diverse models and tasks through a comprehensive set of experiments. It also reports on the effects and implications of training methods such as SFT and RL.
Critical weaknesses
While existing research has examined various "filling sequences," most analyses have been limited to performance comparisons at the sequence level, failing to fully elucidate the impact of factors such as semantics, regularity, tokenization structure, and prompt format. Consequently, no interpretable or predictable link has been established between the characteristics of the inserted content and model behavior; performance fluctuations are often simply attributed to an "opaque inference process." Although this paper demonstrates that the effects of filling sequences depend heavily on the model, task, and context, it fails to establish principles to explain or predict these variations. Thus, readers struggle to determine whether the results apply under untested conditions or how they might be leveraged to enhance inference. Regarding Chain-of-Thought (CoT) monitoring, experiments have shown that readable inference traces are absent for certain computations, yet the impact of this phenomenon on monitoring effectiveness has not been directly quantified. The paper needs to more clearly define its scientific contributions and scope of applicability.
Questions for authors
-
I want to know how manys GPUs were used for the model training of Qwen3-235B (SFT and RL stage)
-
Is there further evidence to distinguish the "task-specific computations" arising from the "fill position" from positional effects or attention-redistribution effects? In particular, how should the proposed diagnostic criteria be interpreted regarding the performance gains observed alongside "context-blocking"?
-
For models that do not support the "assistant-prefill" function, please report the number of retained samples, the exclusion rates for both conditions, and the comparison results based on the shared subset of retained samples. How would the conclusions change if "Chain-of-Thought" (CoT) outputs were taken into account or treated as protocol violations?