- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper ask whether RLVR with a format-specific answer reward (final answer after “####” or inside \boxed{}) reduces a model's adherence to a different answer-format request that it previously followed. The authors propose a crossed design experimental setup: they train under each format, evaluate each trained policy under both requests, and use the initial policy as a reference.
The experiments find that boxed-format training lowers adherence to the hash format in four of five seeds, with the model reverting to \boxed{} instead. Changing only the final answer marker in SFT is enough to reverse which format the model prefers. Strict-reader score gains substantially overstate gains in the correctness of the model's committed answers. Producing the requested marker does not guarantee that it carries the model’s actual answer. The effect is small when the trained and requested formats are both already familiar to the model.
Critical strengths
-
The research questions are very well designed, also the problem is interesting from an engineering and practical perspective: Training under one answer format can quietly reduce adherence to another format the model previously followed, and evaluation under the trained format would not detect this. I encountered this problem during training LLM several times but have not seen any paper analyzed it this deeply, which I really appreciate.
-
The experimental setup and design is very comprehensive, careful, well thought-out: The crossed training and request design is sophisticated, with reference to a shared initial policy. Experiments are validated through multiple seeds, including one seed which runs against the main trend, so I really admire this since many other LLM works only report single seed results. Frozen detectors and readers are applied to the same saved responses, and human annotation is blinded and two-stage. This matters because the headline numbers can be trusted as stated, and the authors avoid overclaiming from individual analyses.
Critical weaknesses
- The paper is very hard to follow. The contribution is primarily an evaluation insight, so it has to be communicated clearly to be useful. However, the writing is absolutely awful.
- The abstract relies on terms that are not explained in plain language (“current request”, “payload”, “convention-sensitive scorer”, “committed-answer correctness”), overall the writing is full of ambiguous terms and jargons, makes it an absolutely chore to read through
- Line 125: P_{t,r} = E[d_r(y)] has t on the left-hand side but not the right-hand side, because t enters only through y ∼ π_t(·|p_r(x)). The sampling distribution should be written explicitly. Eq. (2) has the same issue with π.
- E_len and E_str (line ~163) are used before they are defined. The strict and lenient readers are only described in Section 3, under different names.
- The rationale for Eq. (1) is not explained. “Advantage” here means the hash-trained minus boxed-trained payload rate, which is confusing in a paper that also uses GRPO advantages. In addition, I_P does not involve the initial policy, so it cannot show which arm changed. The quantity reported in the abstract is actually ΔP_{B,h}.
- Sections 2.3–2.4 are dense and is not well organized, some places even over-complicated. For instance, the I4 contrast written “relative to the shared initial policy” reduces to D(π_H) − D(π_B), because D(π₀) cancels. ρ and S_k are introduced without explaining why they are needed.
- The result seems to be in a very narrow scope, and appears inconsistent between different pairs of convention. Also the novelty of the work is limited:
- The loss of convention-adherence effect is shown for one model (Qwen2.5-7B-Instruct), one format pair, one training task, and in one direction only. Boxed training reduces hash adherence, but hash-trained policies keep at least 99.55% boxed adherence. Moreover, the only other format pair tested, JSON/XML on OLMo-2 (App. D.3), shows no effect. So the effect is asymmetric, and also does not generalize well. The author present no explanation or analysis regarding this.
- Regarding novelty, existing works already shows that RLVR degrades instruction following, including GRPO experiments on Qwen2.5 [1]. It also shows that models struggle to follow instructions that conflict with patterns learned in training [2], and that format learning accounts for much of the measured RLVR gain [3]. Coupled with the fact that the evaluation and analysis of the author seems to be relevant on only a limited set of convention and direction (box and hash), I suspect that this is simply a problem with a training configuration (the choice of answer convention) rather than an actual RLVR failure mode. The author presents no explanation or analysis on this point.
NOTE: the paper has no limitations section.
References: [1] Fu, Tingchen, Yafu Li, Jiawei Gu, Xiaoye Qu, and Yu Cheng. "Scaling reasoning, losing control: Evaluating instruction following in large reasoning models." In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 40445-40463. 2026.
[2] Zhang, Qinyan, Xinping Lei, Ruijie Miao, FU YU, Haojie Fan, Le Chang, Jiafan Hou et al. "Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?." In International Conference on Learning Representations, vol. 2026, pp. 129040-129063. 2026.
[3] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. Advances in Neural Information Processing Systems, 38:122721–122764, 2026b.
Questions for authors
- Since KL reward and regularization are disabled, I am curious whether boxed-format RLVR still reduce hash adherence (a) with KL regularization to the initial policy, or (b) with training prompts that randomly mix both requests, each paired with its matching reward? If either removes the effect, the failure mode follows from a specific training configuration rather than from RLVR itself.
- How do you explain the asymmetry, where hash training preserves boxed adherence? Is there evidence that the effect occurs when the trained format is not the model’s pre-existing dominant format, for example hash/boxed crossing on OLMo-2 or another model family?
- Why do the 7B runs use the frozen stratified prompt proposal with importance correction, and does the crossed result hold under uniform prompt sampling?
- At what training step does hash adherence diverge between seed 85 and the other seeds, and does any training statistic predict it?