- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how auxiliary format rewards affect representations during RLVR. Starting from a shared format-only training checkpoint for each seed, the authors compare correctness-only training with training that retains a fixed-format reward. They analyze teacher-forced solutions rendered in four formats using variance decomposition and cross-format retrieval. The main finding is a consistent directional difference across three seeds at a late-middle layer and an answer-boundary position, with much smaller differences in pooled reasoning-body representations. An exploratory condition combines randomized formats with additional reasoning supervision.
Critical strengths
-
The matched C–CF comparison may be a useful part of the design. Branching from the same Phase-1 checkpoint within each seed makes the effect of retaining format reward easier to interpret than a comparison between unrelated checkpoints.
-
The paired-format evaluation keeps the problem and solution content fixed. Reporting both variance decomposition and within-model cross-format retrieval, together with per-seed results, provides useful evidence for the observed representation difference.
Critical weaknesses
-
I think this paper is closely related to the paper published by Shao et al. 2024 and subsequent work examining how reward design affects RLVR. It already includes format-only reward experiments. Although the present paper cites that work, it discusses the general spurious-reward findings without explicitly addressing this directly relevant setting, I wonder if the author intentionally avoided addressing this issue. Therefore, I find its novelty limited, and the experimental setup fairly conventional.
-
I also find the introductory statement that “the internal changes induced by RLVR remain poorly understood” too broad. There is already a substantial body of work examining how RLVR and reasoning-oriented fine-tuning affect model behavior such as [1,2,3]. The paper does not sufficiently engage with this literature, making the research gap appear broader than it is and leaving its incremental contribution less clear.
[1]. Ruan Z, Chen Y, Hou Y, et al. Unveiling over-memorization in finetuning llms for reasoning tasks[J]. arXiv preprint arXiv:2508.04117, 2025.
[2]. Wu M, Zhang Z, Dong Q, et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(40): 33944-33952.
[3]. Yan L, Li R, Chen G, et al. Spurious rewards paradox: Mechanistically understanding how RLVR activates memorization shortcuts in LLMs[J]. arXiv preprint arXiv:2601.11061, 2026.
Questions for authors
-
Regardless of whether the author can clarify the differences between this work and previous ones, the matched format-reward comparison maybe a useful distinction, but its main effect is concentrated at the answer boundary and nearly disappears in pooled reasoning-body representations. It remains unclear what additional understanding this shift provides beyond a localized change in format sensitivity. Could the author provide a more insightful analysis?
-
As I mentioned in the Weaknesses, please have the author clarify this distinction.