acceptodds
Under review as a conference paper at ICLR 2027

Learned Reporting Preferences in RLVR Can Conflict with the Current Request

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using automatically checked answers. Yet convention-matched evaluation cannot reveal whether reinforcing one reporting convention reduces adherence to a different request that the initial policy already follows. To test this, we train matched policies under two reporting conventions and evaluate each policy under both current requests, using the same initial policy as a shared reference. We complement this crossed design with controlled interventions and independent human calibration. On GSM8K, boxed-format RLVR reduces the fraction of Qwen2.5-7B responses containing the requested hash-format payload by 35.33–74.37 percentage points relative to a 95.45% initial baseline in four of five training seeds; the fifth improves by 2.50 points. In the four deteriorating runs, almost every response that omits the requested payload instead retains the trained boxed convention, and the same four seeds deteriorate under two fixed paraphrases. Changing only the final-answer marker in supervised targets reverses which reporting convention the model prefers across three seeds, providing controlled evidence that this preference is learnable. Across three settings with independent human calibration, gains under a convention-sensitive scorer exceed the corresponding gains in committed-answer correctness, i.e., the correctness of the answer the model actually commits to. Together, these results separate three distinct post-training outcomes: learned reporting preference, current-request adherence, and committed-answer correctness. They show that convention-matched accuracy alone does not fully characterize post-training behavior and motivate evaluating current-request adherence alongside convention-matched task accuracy. Code: https://anonymous.4open.science/r/rlvr-reporting-conventions-8114/

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards (RLVR), reasoning evaluation, instruction following, reporting conventions, evaluation reliability, human calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Mfc2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper ask whether RLVR with a format-specific answer reward (final answer after “####” or inside \boxed{}) reduces a model's adherence to a different answer-format request that it previously followed. The authors propose a crossed design experimental setup: they train under each format, evaluate each trained policy under both requests, and use the initial policy as a reference.

The experiments find that boxed-format training lowers adherence to the hash format in four of five seeds, with the model reverting to \boxed{} instead. Changing only the final answer marker in SFT is enough to reverse which format the model prefers. Strict-reader score gains substantially overstate gains in the correctness of the model's committed answers. Producing the requested marker does not guarantee that it carries the model’s actual answer. The effect is small when the trained and requested formats are both already familiar to the model.

Critical strengths
  1. The research questions are very well designed, also the problem is interesting from an engineering and practical perspective: Training under one answer format can quietly reduce adherence to another format the model previously followed, and evaluation under the trained format would not detect this. I encountered this problem during training LLM several times but have not seen any paper analyzed it this deeply, which I really appreciate.

  2. The experimental setup and design is very comprehensive, careful, well thought-out: The crossed training and request design is sophisticated, with reference to a shared initial policy. Experiments are validated through multiple seeds, including one seed which runs against the main trend, so I really admire this since many other LLM works only report single seed results. Frozen detectors and readers are applied to the same saved responses, and human annotation is blinded and two-stage. This matters because the headline numbers can be trusted as stated, and the authors avoid overclaiming from individual analyses.

Critical weaknesses
  1. The paper is very hard to follow. The contribution is primarily an evaluation insight, so it has to be communicated clearly to be useful. However, the writing is absolutely awful.
  • The abstract relies on terms that are not explained in plain language (“current request”, “payload”, “convention-sensitive scorer”, “committed-answer correctness”), overall the writing is full of ambiguous terms and jargons, makes it an absolutely chore to read through
  • Line 125: P_{t,r} = E[d_r(y)] has t on the left-hand side but not the right-hand side, because t enters only through y ∼ π_t(·|p_r(x)). The sampling distribution should be written explicitly. Eq. (2) has the same issue with π.
  • E_len and E_str (line ~163) are used before they are defined. The strict and lenient readers are only described in Section 3, under different names.
  • The rationale for Eq. (1) is not explained. “Advantage” here means the hash-trained minus boxed-trained payload rate, which is confusing in a paper that also uses GRPO advantages. In addition, I_P does not involve the initial policy, so it cannot show which arm changed. The quantity reported in the abstract is actually ΔP_{B,h}.
  • Sections 2.3–2.4 are dense and is not well organized, some places even over-complicated. For instance, the I4 contrast written “relative to the shared initial policy” reduces to D(π_H) − D(π_B), because D(π₀) cancels. ρ and S_k are introduced without explaining why they are needed.
  1. The result seems to be in a very narrow scope, and appears inconsistent between different pairs of convention. Also the novelty of the work is limited:
  • The loss of convention-adherence effect is shown for one model (Qwen2.5-7B-Instruct), one format pair, one training task, and in one direction only. Boxed training reduces hash adherence, but hash-trained policies keep at least 99.55% boxed adherence. Moreover, the only other format pair tested, JSON/XML on OLMo-2 (App. D.3), shows no effect. So the effect is asymmetric, and also does not generalize well. The author present no explanation or analysis regarding this.
  • Regarding novelty, existing works already shows that RLVR degrades instruction following, including GRPO experiments on Qwen2.5 [1]. It also shows that models struggle to follow instructions that conflict with patterns learned in training [2], and that format learning accounts for much of the measured RLVR gain [3]. Coupled with the fact that the evaluation and analysis of the author seems to be relevant on only a limited set of convention and direction (box and hash), I suspect that this is simply a problem with a training configuration (the choice of answer convention) rather than an actual RLVR failure mode. The author presents no explanation or analysis on this point.

NOTE: the paper has no limitations section.

References: [1] Fu, Tingchen, Yafu Li, Jiawei Gu, Xiaoye Qu, and Yu Cheng. "Scaling reasoning, losing control: Evaluating instruction following in large reasoning models." In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 40445-40463. 2026.

[2] Zhang, Qinyan, Xinping Lei, Ruijie Miao, FU YU, Haojie Fan, Le Chang, Jiafan Hou et al. "Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?." In International Conference on Learning Representations, vol. 2026, pp. 129040-129063. 2026.

[3] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. Advances in Neural Information Processing Systems, 38:122721–122764, 2026b.

Questions for authors
  1. Since KL reward and regularization are disabled, I am curious whether boxed-format RLVR still reduce hash adherence (a) with KL regularization to the initial policy, or (b) with training prompts that randomly mix both requests, each paired with its matching reward? If either removes the effect, the failure mode follows from a specific training configuration rather than from RLVR itself.
  2. How do you explain the asymmetry, where hash training preserves boxed adherence? Is there evidence that the effect occurs when the trained format is not the model’s pre-existing dominant format, for example hash/boxed crossing on OLMo-2 or another model family?
  3. Why do the 7B runs use the frozen stratified prompt proposal with importance correction, and does the crossed result hold under uniform prompt sampling?
  4. At what training step does hash adherence diverge between seed 85 and the other seeds, and does any training statistic predict it?
Reviewer_fR3wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether reporting conventions reinforced during RLVR can override a different convention explicitly requested at evaluation time. It crosses training convention with current request, anchors comparisons to the initial policy, and distinguishes requested-payload presence, automatic-reader accuracy, and human-assessed committed-answer correctness. In the main Qwen2.5-7B/GSM8K experiment, four of five paired training seeds show substantial loss of a previously supported hash-format request after boxed-format training. Validation includes fixed request paraphrases, reader variants, a marker-only SFT intervention, 625 human-calibration observations across three settings, and boundary cases with weaker or absent effects. The main contribution is a controlled diagnostic and evaluation framework, rather than a universal claim that RLVR harms reasoning or only learns formatting.

Critical strengths
  1. A controlled demonstration of lost request adherence. The shared initial-policy anchor and crossed training-convention/current-request design directly test a behavior that the model could previously perform. In Figure 1 and Tables 11–12, initial hash-payload presence is 95.45%; four of five boxed-training seeds reduce it by 35.33–74.37 percentage points, while one seed improves it by 2.50 points. Retaining the counterexample and testing fixed paraphrases makes the conclusion more credible. This is consequential because the failure is measured as output behavior, rather than inferred solely from a strict parser’s accuracy.

  2. Careful separation of measurement targets and honest treatment of boundaries. The paper distinguishes payload presence from final commitment and correctness. Its human calibration uses independent annotation, blinded adjudication, and explicit handling of unresolved cases. It reports that strict-score gains exceed human-assessed gains in the calibrated settings while retaining the uncertainty in the latter. Negative boundary results are also preserved. Together, these choices support a useful change in how post-training evaluations should be designed and interpreted.

Critical weaknesses
  1. The evidence is stronger for the existence of the conflict than for its general mechanism or semantic consequences. The full Qwen2.5-7B/GSM8K crossed matrix is not covered by the human commitment audit, and the marker-only SFT intervention is a separate experimental setting. Likewise, the interface-familiarity comparisons change other factors, including model or task. These panels therefore cannot be combined into a single causal account of why the core RL effect occurs or how much answer correctness changes when adherence fails. The paper acknowledges several of these boundaries, which keeps this concern nonfatal, but the explanatory reach remains narrower than the overall narrative may suggest.

  2. A secondary boundary experiment has an unresolved model identity. Table 4 on p.9 labels Native MATH as Qwen2.5-7B, whereas Appendix D.3.1 on p.33 identifies Qwen2.5-1.5B-Instruct for the same +2.2/+1.4/+0.0 results. The actual backbone matters for interpreting the claimed boundary across settings. I do not see evidence that this invalidates the core five-seed result, but it needs to be corrected before the auxiliary evidence can be relied upon.

Questions for authors
  1. Clarification: Which model and checkpoint produced the Native MATH results in Table 4 and Appendix D.3.1? Is the inconsistency confined to the model label, or does it change the interpretation of any other comparison?

  2. Clarification: Is the intended central conclusion restricted to a demonstrated conflict in request adherence, with familiarity and semantic effects treated as separate hypotheses or contextual observations? Please identify any stronger claim that relies on combining the different experimental panels.

  3. Additional evidence that could change my assessment of significance: On the existing core crossed responses, how often does a missing requested marker coincide with a correct committed answer, conflicting answers, or no clear commitment? A stratified audit across the four deteriorating seeds and the counterexample seed would directly connect the behavioral finding to its semantic consequences. Its absence does not by itself overturn the narrower existence result.

Reviewer_MjxROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines a model's ability to adhere to a response format requested during an evaluation after training with different response formats. They find that naive post-training of a model to use a different answer format lowers its adherence to previous strategies.

Critical strengths

For the problems this paper is addressing, their experimental framework is generally sound. I appreciate that they use human reviewers for a portion of the evaluation, and they use common math eval answer formats.

Critical weaknesses

The problem this paper is attempting to address is not present in modern models, and has been solved by general work on instruction following abilities more broadly in the models themselves, not on careful management of mixtures of answer formats/etc. I believe this is why all the models in this paper are 1.5 years old or more, as modern models of the same or smaller size don't exhibit this behavior anymore. I'm sorry to say there is nothing new here for the community, and it is not nearly up to the standard of an acceptable paper to ICLR. If the authors are going to continue this line of work, I encourage them to read more of the literature on strict instruction following when deciding what to explore next.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.