acceptodds
Under review as a conference paper at ICLR 2027

Decodable Need Not Be Reward-Reachable: A Controlled Study of Direct-Answer Learning in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) depends on both the answers a policy samples and the update that uses their rewards. Guaranteeing correct answers in training removes a coverage problem, but leaves open whether the update can learn an answer that supervision can teach. We isolate this distinction with execution-labeled Yes/No questions about runtime errors in short Python programs. On a list-index task, Qwen3-8B representations support linear label decoding, and same-data supervised fine-tuning (SFT) reaches margin AUC 0.927. Forcing both final answers into every group restores reward variation and a nonzero answer-logit signal, yet the tested centered update reaches only 0.558; the same intervention succeeds on dictionary lookup. Changing the loss or weighting rule improves index readouts under matched sampling. The primary exact-gradient comparison remains inconclusive, while a separate pre-specified error-focused rule learns the direct index answer in all three seeds and meets its local-sufficiency criterion. Reasoning provides a distinct route: strong Qwen answers are already expressed before task-specific training, while direct answers remain weak after reasoning-channel training. The results separate recoverable label information, answers supervision can teach, and answers a specified reward update learns.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning with verifiable rewards, Answer-readout learning, Final-answer coverage, Credit assignment, Representation probing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hRweOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes why GRPO-based updates can fail on single-token binary classification tasks even if they modify the sampling procedure to contain at least one correct and incorrect answer in each group, while SFT succeeds. They show that changing the update weighting (removing zero-centering, allocating more weight to questions the model gets wrong) substantially improves learning. Additionally, they show that linear probes can recover label information despite near-chance direct answers.

Critical strengths

S1: The paper distinguishes decodability (whether a linear classifier can recover label information from hidden states) from learnability (the model can learn to answer correctly) S2: The paper compares group-relative advantage formulations and update weighting rules under matched sampling procedures

Critical weaknesses

The paper is difficult to follow due to insufficient motivation and dense, imprecisely defined terminology. The implications of the single-token binary classification setting and its differences from standard generative GRPO could be explained more clearly.

The controlled experiments use a fixed training budget without demonstrating convergence. Longer runs and learning-rate sensitivity checks are needed to distinguish a persistent limitation from slower optimization.

The broader significance for RLVR remains unclear. Although the paper includes standard GRPO comparisons, its strongest controlled evidence concerns single-token binary classification under forced sampling. Further evidence is needed to establish whether the identified update sensitivity extends to learning multi-step reasoning.

Questions for authors

Do longer runs or learning-rate sweeps close the gap between the centered update and SFT? How do training and held-out correctness probabilities evolve for the freely sampled answers? Which conclusions do the authors expect to extend beyond single-token binary classification to multi-token reasoning?

Reviewer_dZo8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To investigate whether RLVR parameter updates can learn answers that supervised training can teach, the paper conducts experiments on binary yes/no questions involving short Python programs with runtime errors and execution annotations. The experiments distinguish three concepts: label information that can be recovered and extracted, answers that supervised training can teach the model to produce, and answers that the specified reward-update rule can actually learn. Specifically, the experiments show that, on the list-indexing task, the representations of Qwen3-8B support linear decoding of the labels. Even when both possible final answers are forced to appear in every group, restoring reward variance and a non-zero answer log-probability signal, the centered update algorithm evaluated in the paper achieves only a marginal AUC of 0.558. In contrast, the exact same intervention succeeds on the dictionary-lookup task. Under matched sampling conditions, changing the loss function or weighting rule improves the model's output on the indexing task.

Critical strengths
  1. The paper independently evaluates three dimensions—decodability, teachability through supervised training, and learnability through reward updates—and provides a joint decision rule. This multidimensional, hierarchical evaluation framework offers greater interpretability.

  2. The paper compares and analyzes multiple failure cases in detail, including inconclusive results, failure to meet the strict criterion, failure of the Llama replication to meet the installation rule, and GSM8K/MBPP failing to meet the inclusion criteria.

Critical weaknesses
  1. The experiments focus primarily on the indexing task. The authors explicitly position key lookup as a positive control for the intervention rather than a difficulty-matched task. GSM8K and MBPP fail to meet the inclusion criteria because their baseline margins are already 0.77, while SFT on Llama is unstable. It therefore remains unclear whether the conclusions generalize to natural language tasks.

  2. The error-focused weighting rule uses unnormalized weights, and greedy accuracy varies substantially across the three random seeds. Furthermore, all conclusions are based on Qwen3-8B and a single task, leaving insufficient experimental evidence to establish the stability and generalizability of the findings.

Questions for authors

Does the success of the error-focused weighting rule stem from relative weight allocation or the total update magnitude? can the authors report results for a normalized version of the weighting rule and a continuous sweep over alpha?

Reviewer_K4FJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether answers that are decodable from LLM representations and learnable through supervised fine-tuning can also be learned through reinforcement learning with verifiable rewards (RLVR). Using controlled Python runtime-error prediction tasks, the authors show that GRPO can fail to learn correct answers even when both correct and incorrect responses are guaranteed during training. They further examine alternative loss functions, reward weighting strategies, and reasoning-based responses to better understand this limitation. Experiments across multiple tasks and models reveal a gap between answer decodability, supervised learnability, and reward-based learning, although the underlying optimization mechanism remains inconclusive.

Critical strengths

The paper addresses an interesting and meaningful research question: whether knowledge accessible in model representations can be effectively learned through reward-based optimization. This distinction provides a valuable perspective for understanding the limitations of current RLVR methods.

Critical weaknesses
  1. The main findings rely heavily on controlled, synthetic Python tasks, with insufficient evidence that the observed phenomenon extends to real-world reasoning tasks.
  2. Although the paper identifies a learning gap between SFT and RLVR, it does not clearly establish why GRPO fails. The key experiments investigating the optimization mechanism remain inconclusive, limiting the strength of the paper's conclusions.
Questions for authors
  1. Given that the primary exact-gradient comparison remains inconclusive, what evidence supports the proposed explanation for why GRPO fails to learn certain answers despite guaranteed correct-answer coverage?
  2. Would error-focused weighting still outperform standard reward weighting if the total update magnitude were kept constant?
  3. Under what conditions would the proposed weighting strategies be expected to improve RLVR performance on more complex reasoning tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.