acceptodds
Under review as a conference paper at ICLR 2027

When Format Rewards Fade: Layer-Specific Representation Shifts in RLVR

Abstract

Format constraints in reinforcement learning with verifiable rewards (RLVR) are typically treated as output-level scaffolding, but whether they leave persistent traces in a model’s internal representations remains unclear. We study this question through a controlled two-phase training protocol for mathematical reasoning. All models first undergo a shared format-compliance phase and then continue training under three conditions: format reward withdrawal, sustained fixed-format reward, or sustained randomized-format reward. Using paired-format evaluation traces and layer-wise representation analysis relative to the shared Phase-1 checkpoint, we examine how these reward choices reshape format- and problem-related information. We find that withdrawing format reward produces a pronounced increase in problem-related variance at a late-middle layer, while the corresponding changes are not reliably observed at early or final layers. Sustaining fixed-format reward instead leaves representations closer to the shared baseline, revealing a strongly layer-specific response to reward composition. Behavioral evaluation further shows that the randomized-format condition attains the highest observed mean GSM8K accuracy and the lowest cross-seed variability among the three conditions. Together, these results suggest that format-induced specialization is not necessarily permanent: changing the reward signal can selectively reshape internal representations, while format diversity may offer a promising direction for more stable RLVR training.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning from Verifiable Rewards (RLVR), Representation Reallocation, Format Reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2pi6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how auxiliary format rewards affect representations during RLVR. Starting from a shared format-only training checkpoint for each seed, the authors compare correctness-only training with training that retains a fixed-format reward. They analyze teacher-forced solutions rendered in four formats using variance decomposition and cross-format retrieval. The main finding is a consistent directional difference across three seeds at a late-middle layer and an answer-boundary position, with much smaller differences in pooled reasoning-body representations. An exploratory condition combines randomized formats with additional reasoning supervision.

Critical strengths
  1. The matched C–CF comparison may be a useful part of the design. Branching from the same Phase-1 checkpoint within each seed makes the effect of retaining format reward easier to interpret than a comparison between unrelated checkpoints.

  2. The paired-format evaluation keeps the problem and solution content fixed. Reporting both variance decomposition and within-model cross-format retrieval, together with per-seed results, provides useful evidence for the observed representation difference.

Critical weaknesses
  1. I think this paper is closely related to the paper published by Shao et al. 2024 and subsequent work examining how reward design affects RLVR. It already includes format-only reward experiments. Although the present paper cites that work, it discusses the general spurious-reward findings without explicitly addressing this directly relevant setting, I wonder if the author intentionally avoided addressing this issue. Therefore, I find its novelty limited, and the experimental setup fairly conventional.

  2. I also find the introductory statement that “the internal changes induced by RLVR remain poorly understood” too broad. There is already a substantial body of work examining how RLVR and reasoning-oriented fine-tuning affect model behavior such as [1,2,3]. The paper does not sufficiently engage with this literature, making the research gap appear broader than it is and leaving its incremental contribution less clear.

[1]. Ruan Z, Chen Y, Hou Y, et al. Unveiling over-memorization in finetuning llms for reasoning tasks[J]. arXiv preprint arXiv:2508.04117, 2025.

[2]. Wu M, Zhang Z, Dong Q, et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(40): 33944-33952.

[3]. Yan L, Li R, Chen G, et al. Spurious rewards paradox: Mechanistically understanding how RLVR activates memorization shortcuts in LLMs[J]. arXiv preprint arXiv:2601.11061, 2026.

Questions for authors
  1. Regardless of whether the author can clarify the differences between this work and previous ones, the matched format-reward comparison maybe a useful distinction, but its main effect is concentrated at the answer boundary and nearly disappears in pooled reasoning-body representations. It remains unclear what additional understanding this shift provides beyond a localized change in format sensitivity. Could the author provide a more insightful analysis?

  2. As I mentioned in the Weaknesses, please have the author clarify this distinction.

Reviewer_iBqpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how format rewards in RLVR affect internal representation.

The work studies 3 different reward functions: correctness-only, correctness plus fixed format (typically used in GRPO codes), and an exploratory randomized format.

The paper's main contributions is studying the 3 reward functions' effect when doing RLVR for Qwen2.5-7B-Instruct on GSM8K dataset. The core finding is, removing format reward increased problem-associated variance and improves cross-format retrieval performance.

The experiment part uses 3 seeds

Critical strengths
  • Studying the internal representation of RLed checkpoint is interesting.
  • Honestly reporting the random seed results
Critical weaknesses
  • The paper observes the representation diff of different reward functions but does not demonstrate why this difference matters for reasoning or generalization.
  • The testbed is narrow and small. The author only trains one model (Qwen2.5-7B-Instruct) in one domain (GSM8K) with few training data and number of iterations. The generalization of the finding on format reward cannot be supported by the given experiments
  • The claimed stability improvement is not well supported by only three seeds per condition. C alone spans approximately 23 percentage points in accuracy across seeds, indicating substantial training variability. With so few runs, both mean performance and across-seed variance are poorly estimated, and the apparent ranking may depend heavily on individual runs. More independent seeds and uncertainty estimates are needed before concluding that CRF reliably improves accuracy or stability.
Questions for authors

See weaknesses.

Reviewer_zEf9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to determine whether format rewards in RLVR training alter the model’s internal representations. The authors first had the model learn a specific format uniformly, then split it into two groups for further training: one group had the format reward removed, while the other retained it. They then compared the internal states of the two groups of models just before they produced their answers. The conclusion is that, after removing the format reward, the model focuses more on the problem itself in the middle and later layers and becomes less sensitive to the format, while the accuracy of both groups of models in solving problems is roughly the same.

Critical strengths
  1. Pairwise branch design is a concise method for isolating format-specific rewards. For each seed, both C and CF start from the same first-stage checkpoint, share the same correctness reward, and employ a format-agnostic answer extraction method. The use of paired formats and teacher-forced corpora ensures, to a certain extent, that the quality of the generated answers does not influence the representation analysis. Furthermore, the authors candidly point out in the main text that the existing evidence is purely descriptive.
Critical weaknesses
  1. The experimental results are insufficient to support the conclusion. The main results are based on only three seeds, and the core argument is that the three groups show consistent trends; however, even if no effect exists, the probability of such a coincidence occurring is 1 in 4. Furthermore, the paper emphasizes the single layer 21 out of the four layers that happened to exhibit this pattern. The effect sizes vary greatly. Meanwhile, the confidence bands in Figure 2 overlap almost completely. The signs of the sums of squares of variance in the problem section do not even match in Table 3. Therefore, it is impossible to distinguish the differences between the C-CF checkpoints from seed noise. Furthermore, the abstract over claims that there are significant improvements and that format diversity shows promise, yet the latter statement contradicts the narrative in the main text.
  2. The implications are limited, and the training setup is questionable. If a model consistently receives rewards for a specific delimiter format, it will retain more format-related variance at the boundaries of its answers, a finding that is largely as expected. Furthermore, this finding has no measurable impact on performance, and there is essentially no difference in accuracy. Furthermore, the reported GSM8K accuracy (61.5-84.9%) is significantly lower than the commonly reported values for the not fine-tuned Qwen2.5-7B-Instruct (approximately 90%), yet the paper reports neither the accuracy of the base model nor that of the before fine-tuning checkpoint. If the training caused the model to perform worse, then the observed changes in representations might reflect optimization instability rather than the nature of the RLVR reward composition itself.
Questions for authors
  1. Please report GSM8K accuracy, using the same evaluation pipeline, for the base model and each Phase-1 checkpoint. Why are Phase-2 accuracies (e.g., 61.5% for C, seed 50) so far below the base model?
  2. How large is the layer-21 difference between two independent Phase-2 runs of the same condition ? Without this null, the C-CF contrast cannot be separated from run-to-run variability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.