acceptodds
Desk rejected at ICLR 2027

Reflection Rewards Pay for Deferral: Capability-Aware Rewards for Reflective RL

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-reflective reinforcement learning trains models to revise their answers and is typically judged by final-answer accuracy. However, a reward for correcting mistakes also shapes the answer being corrected. We identify deferral, a failure mode in which a model postpones its real answer until after reflection, so that competitive final accuracy conceals a degraded first answer. In a controlled setting, a reflection-trained model's first answer falls below chance while its final answer remains accurate. We trace this to the reward: when a trajectory-level reward values a correction above a preserved correct answer, it introduces a bias toward deferral that grows with the model's capability. Natural remedies do not resolve this. Rewarding the reflection or answer changes can also pay for deferral, and since a deferred first answer is predictably wrong, rewarding calibrated confidence reinforces it too. We therefore propose Uncertainty-Aware Self-reflective Reinforcement Learning (UASRL), which uses a per-prompt capability proxy estimated once before RL, beyond the policy's control, and favors correct first answers where capability is high while still rewarding correction. Added to an existing reflection reward, UASRL substantially improves the first-answer accuracy of that reward and reduces deferral while maintaining competitive final accuracy. Across two backbones and three benchmarks, including a held-out one, it outperforms all tested reflection methods on first answers. Our results expose first-answer degradation as a hidden cost of reflection rewards, derive design criteria for such rewards, and show that self-reflection should be evaluated by both its first and final answers.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, self-reflection, reward function, VLM.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BoTkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper identifies deferral in reflective RL: correction rewards can encourage poor first answers while preserving final accuracy. It analyzes this incentive and proposes UASRL, which uses per-prompt success rates fixed before RL to reward correct first answers. Experiments on two vision-language models and three main benchmarks, supplemented by ablations, multi-seed runs, and a non-templated experiment, show reduced deferral and improved first-answer accuracy with competitive final accuracy. Most evidence comes from controlled, templated initializations.

Critical strengths
  1. A consequential failure mode with a clear mechanism. The paper shows how competitive final accuracy can conceal severe first-answer degradation, and connects this behavior to correction incentives through local theoretical analysis and answer-transition measurements. This provides a concrete reason to evaluate both first and final answers in reflective RL.
  2. An effective mitigation in the evaluated settings. UASRL improves first-answer accuracy over the tested reflection baselines across two backbones and three main benchmarks while maintaining competitive final accuracy. Reward ablations support the usefulness of the fixed capability signal, and the method requires no change to the inference procedure.
Critical weaknesses
  1. Limited evidence beyond the controlled initialization. The main experiments use SFT data containing 68% deferred trajectories. The non-templated experiment is encouraging, but its labeling rule may conflate deferral with ordinary errors that disappear under a different prompt, leaving the broader applicability of the diagnosis uncertain.
  2. Table 5 compares UASRL with the strongest uniform baseline, “SRPO as specified,” using single runs. The multi-seed results in Table 11 instead use SRPO†, a different reward variant, leaving the robustness of UASRL’s advantage over the stronger baseline unverified.
Questions for authors
  1. In Table 6 and Appendix B.6, how was the deferral labeling rule validated against genuine correction or prompt-dependent changes in correctness?
  2. Can you provide matched multi-seed comparisons between UASRL and the strongest uniform-protection baseline, “SRPO as specified” in Table 5?
3 comments

We thank the reviewer for the careful review, the encouraging assessment of our non-templated experiment, and the constructive questions.

W1: Evidence beyond the controlled initialization

The templated initialization is a deliberate controlled setting: the placeholder makes deferral directly measurable, so that we can quantify how each reward amplifies or suppresses it. Our non-templated experiment (Table 6) indicates that deferral also emerges without templates; to further demonstrate this, we have extended it to another backbone and another benchmark.

All settings start from the base model with the reflection prompt and no SFT, using the same GRPO configuration as the main experiments. For UASRL, cec_e is estimated before RL from eight rollouts of this initialization. Values are percentages from single runs; the deferral proxy is discussed in Q1.

BackboneBenchmarkMethodA1A_1 ↑A2A_2 ↑Deferral proxy ↓
Qwen2.5-VL-7BMathVistaInitialization57.357.60.3
SRPO†46.663.117.6
UASRL61.964.82.4
InternVL3-8BMathVistaInitialization70.471.20.8
SRPO†59.275.216.0
UASRL74.577.42.7
Qwen2.5-VL-7BScienceQAInitialization77.378.00.7
SRPO†66.983.118.0
UASRL82.984.01.2

In all three settings, SRPO† lowers first-answer accuracy by 10.4–11.2 points while increasing final-answer accuracy by 4.0–5.5 points, whereas UASRL improves both relative to initialization. Because this pattern uses only accuracies, it does not depend on the deferral labeling rule. It shows that first-answer degradation concealed by improved final accuracy is not confined to templated SFT; the general prevalence of deferral remains open. We will add these results to §5.6 and clarify the scope of the controlled initialization in the introduction and limitations.

Q1: Validation of the deferral labeling rule

The rule in Appendix B.6 combines three conditions: the first answer is wrong, differs from the final answer, and the same checkpoint answers correctly under a single-pass prompt. The third condition provides a capability check: it identifies cases where an incorrect first answer coexists with demonstrated ability to answer the question directly. We assessed the resulting pattern across initialization and the trained checkpoints. Across the three non-templated settings, the proxy rate is 0.3–0.8% at initialization, 1.2–2.7% under UASRL, and 16.0–18.0% under SRPO†. The concentration of this behavior under the positive-gap reward is consistent with our incentive analysis. Independently of the labeling rule, SRPO† reduces first-answer accuracy by 10.4–11.2 points while improving final-answer accuracy, whereas UASRL improves both. These comparisons provide supporting evidence, rather than an independent validation of individual labels. In particular, single-pass success does not exclude a genuine error in another trajectory, and prompt sensitivity may change during training. We will make this distinction explicit in Appendix B.6: the non-templated results directly establish first-answer degradation, while the operational proxy supports its interpretation as deferral without uniquely identifying that mechanism.

W2 / Q2: Multi-seed comparison with SRPO as specified

Table 11 reports multiple seeds for SRPO†, whereas the comparison with SRPO as specified in Table 5 uses a single run. We have now evaluated SRPO as specified over three training seeds under the same initialization, G=2G=2, training budget, and evaluation protocol as UASRL. We also include matched SRPO† for context. Three-seed comparison (MathVista, Qwen2.5-VL-7B, shared templated SFT initialization). Values are percentages, reported as mean ± sample standard deviation across three training seeds. The UASRL row uses a seed set separate from that of Table 11.

RewardA1A_1 ↑A2A_2 ↑Deferral ↓
Matched SRPO† (g=+0.25g=+0.25)52.7 ± 0.865.8 ± 0.428.5 ± 1.0
SRPO as specified (g=−0.25g=-0.25)58.2 ± 0.767.0 ± 0.421.9 ± 0.2
UASRL61.3 ± 0.667.8 ± 0.413.5 ± 0.7

Compared with SRPO as specified, UASRL improves mean first-answer accuracy by 3.1 percentage points and reduces mean deferral by 8.4 points. These results provide additional evidence for the advantage observed in the earlier single-run comparison on this backbone and benchmark. Mean final-answer accuracy is also 0.8 points higher; we interpret this smaller difference cautiously given the limited number of seeds. SRPO as specified remains a strong simple baseline: its negative gap already removes the positive deferral bias under our local model. We will add this comparison to the paper, foreground SRPO as specified as the primary uniform baseline, and distinguish UASRL’s incremental gains over it from the larger improvements over positive-gap SRPO†.

Thanks. I will keep my score and raise my soundness accordingly.

Thank you for the positive assessment and for raising the soundness score!

We’ve revised the paper based on our responses above, with text changes marked in blue. In particular, we updated the non-templated experiment table (Table 6) with the two new settings, and added a new table in Appendix C.2 with the matched three-seed comparison, including SRPO as specified and selective masking.

Thanks again for the constructive feedback. It really helped improve the paper.

Reviewer_SNwmOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the deferral problem in self-reflective RL. The problem is that, for higher correction rewards, the model tends to lower the first-answer accuracy and postpone the correct answer after reflection. The author analyzes this reward incentive and proposes UASRL. By utilizing the anticipated per-prompt capability to adjust the reward and encourage the model to directly give the correct answer.

Critical strengths
  1. The paper reveals one important but easily ignored failure model in self-reflective RL. The paper points out that corrective rewards will lead the model to postpone the correct answer to make the final answer accuracy to degrade due to first-answer degradation. Future, the auther theortically anayze the relationship between reward gap and deferral incentive . The paper provides a good theoretical framework for reflection reward design.
  2. The experimental results fully show the deferral impacts and the effectiveness of UASRL. The author validate their idea under two VLM backbones (Qwen2.5-VL-7B, InternVL3-8B) and 3 benchmarks (MathVista, MMMU-Pro, ScienceQA). UASRL substantially improves first-answer accuracy with competitive final-answer accuracy.
Critical weaknesses
  1. The core experiment relies on human-constructed SFT initialization, which limits the paper's generalization ability. The authors intentionally inserted incorrect initial answers into 68% of the SFT trajectories, causing the model to exhibit clear delayed-response behavior even before the RL phase began. Therefore, the primary experiments demonstrate, more directly, that UASRL can successfully handle pre-injected delays, rather than establishing the prevalence of such issues in natural, self-reflective RL. While Table 6 provides experiments using non-templated initializations, it covers only a single backbone model and one benchmark.
Questions for authors

NA

3 comments

We thank the reviewer for the careful assessment and constructive suggestions, which helped us clarify the scope of the controlled initialization and strengthen the evidence beyond it.

W1: Templated SFT initialization and generality

The templated initialization is a deliberate controlled setting: the placeholder makes deferral directly measurable, allowing us to quantify how each reward amplifies or suppresses it. We agree that this setting alone does not show the prevalence of deferral in naturally initialized self-reflective RL. The original non-templated experiment (Table 6) provides evidence beyond this construction, but covers only one backbone and one benchmark. We have therefore extended it to two additional settings.

All settings start from the base model with the reflection prompt and no SFT, following the GRPO configuration of the original non-templated experiment. Values are percentages from single runs; the first block reproduces the corresponding results from Table 6. Deferral is measured using the operational proxy in Appendix B.6.

BackboneBenchmarkMethodA1A_1 ↑A2A_2 ↑Deferral proxy ↓
Qwen2.5-VL-7BMathVistaInitialization57.357.60.3
SRPO†46.663.117.6
UASRL61.964.82.4
InternVL3-8BMathVistaInitialization70.471.20.8
SRPO†59.275.216.0
UASRL74.577.42.7
Qwen2.5-VL-7BScienceQAInitialization77.378.00.7
SRPO†66.983.118.0
UASRL82.984.01.2

Across all three settings, SRPO† lowers first-answer accuracy by 10.4–11.2 points while increasing final-answer accuracy by 4.0–5.5 points. UASRL instead improves both accuracies relative to initialization. These directly measured changes establish that first-answer degradation concealed by improved final accuracy is not confined to templated SFT.

The deferral proxy also rises from 0.3–0.8% at initialization to 16.0–18.0% under SRPO†, compared with 1.2–2.7% under UASRL. This pattern is consistent with deferral emerging without templates, although the proxy may also capture prompt-sensitive errors. The expanded results strengthen the evidence beyond our controlled initialization without establishing general prevalence. We will add these experiments to §5.6 and clarify this scope in the introduction and limitations.

Thanks! I would like to keep my score!

Thank you for the positive assessment.

We’ve revised the paper based on our responses above, with text changes marked in blue. In particular, we updated the non-templated experiment table (Table 6) with the two new settings, and added a new table in Appendix C.2 with the matched three-seed comparison, including SRPO as specified and selective masking.

Thanks again for the constructive feedback. It really helped improve the paper.

Reviewer_rGjUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RL training of vision-language models to reflect on and revise their answers. The model gives a first answer, a reflection, and a final answer, and is rewarded on both answers. The authors point out a failure they call deferral: the model learns to give a throwaway first answer and save its real answer for after the reflection. Final accuracy looks fine while the first answer quietly degrades.

They trace this to the reward. If correcting a mistake pays more than getting the answer right the first time (a positive "correction-incentive gap"), the model is rewarded for deferring, and the incentive grows with how capable the model is on a given question. They note that SRPO's released code has such a positive gap, even though the SRPO paper specifies a negative one. They also argue that rewarding the reflection itself, answer changes, or calibrated confidence can all end up paying for deferral too.

Their fix, UASRL, estimates how often the SFT model answers each training question correctly before RL starts. It uses that fixed estimate to make preserving a correct first answer worth more than a correction on questions the model can handle, while still rewarding correction elsewhere.

Experiments use Qwen2.5-VL-7B and InternVL3-8B, trained on MathVista and ScienceQA and tested on those plus held-out MMMU-Pro. They start from an SFT model whose training data deliberately includes placeholder first answers. UASRL gives the best first-answer accuracy of the methods tested in every setting, with similar or slightly better final accuracy than other methods

Critical strengths
  1. Formal presentation of why reflection rewards pay for deferral with broadly applicable design rules: Earlier work provided evidence that models had reward hacking behavior of deliberately providing wrong first answer and correcting it later but nobody had formal explanation or study. The paper shows that the extra reward for deferring is simply the final-accuracy effect plus a bias that grows with the model's capability . So whenever correction pays more than getting it right the first time, a capable model is better off just postponing its answer. It also explains why the obvious fixes don't work. Rewarding calibrated confidence, or using the policy's own success rate, both end up rewarding deferral, because a deferred first answer is predictably wrong.

  2. Careful measurement that makes the failure concrete. The authors show how much of each change in first-answer accuracy comes from deferring versus genuinely attempting (Table 4). They track exactly how answers move between the first and final attempt. They check that UASRL isn't just attempting easier questions, and they audit how each reward treats deferral before training even starts. The experiment without templates is particularly convincing: SRPO still pushes deferral up to 17.6% and costs over ten points of first-answer accuracy, while final accuracy goes up.

This matters because it turns an anecdote into a measurable effect, and it gives the field a simple evaluation habit worth adopting: report first-answer accuracy and answer transitions, not just final accuracy. It's also why I'm convinced the problem is real, even though I'm less convinced UASRL is the right fix.

Critical weaknesses
  1. I don't think the experiments clearly show that UASRL works for the stated reason in the paper. The paper explains UASRL's effect through the correction-incentive gap: it makes the gap negative where capability is high, and predictions are stated in terms of the gap's sign (table 5). In the G = 2 regime used for all experiments, Theorem 1(ii) says the gap matters only through its sign, that is, through the ranking of outcomes. The binary r_cal control is hard to reconcile with this account under either reading of the theory. It keeps SRPO's outcome ranking on every prompt, so the G = 2 analysis predicts the same behavior as SRPO. It also raises the average (0,1) − (1,1) reward difference at initialization (+0.104), so the expected-reward analysis of Theorem 1(i) predicts more deferral. Yet it cuts deferral from 32.5% to 17.8% and lifts A1 from 52.4 to 59.3 (Table 4 vs. the matched run in Table 5). That's about seven of UASRL's nine points over matched SRPO. Continuous r_cal, the variant that actually flips the gap on capable prompts, adds less than one more point (60.1).

  2. The improvement over the simplest fix is small: The alternative that is presented is to use the reward as specified in the SRPO paper itself which has a negative gap. This lands the results at 58.4 against USARL's 61.3 which is a much smaller win.

  3. Paper positions deferral as a problem it identifies. Based on literature survey, I see that at least couple of other papers have identified this phenomenon. These are 'Self-rewarding correction for mathematical reasoning' and 'Learning Self-Correction in Vision–Language Models via Rollout Augmentation' (ICML '26).

Second paper reports the same behavior in a single-pass VLM setting with a shaped reward attributed to SRPO that has the same positive gap: the model deliberately produces an incorrect first response despite correct reasoning, followed by a trivial correction after It also proposes a remedy: it masks the first response and unmasks it only when correctness is unchanged, so corrected trajectories never reinforce the wrong first answer. Since this removes the deferral incentive without a capability proxy, it would be a natural baseline.

I would honestly find the paper stronger if it framed its contribution as the first formal analysis of this incentive.

Questions for authors
  1. What advantage normalization, loss aggregation and KL settings were actually used? Binary r_cal keeps SRPO's ranking and raises the average reward for deferring, so what explains its effect? How many seeds were run for it ?

  2. How were UASRL's reward coefficients chosen and what explains relatively smaller win ?

2 comments

We thank the reviewer for the careful review and constructive suggestions.

W1: Binary control and mechanistic attribution

Under Theorem 1(ii)’s assumptions, unchanged outcome rankings imply unchanged local G=2G=2 updates in the vanishing-stabilizer limit. Binary’s improvement is therefore not explained by this ranking-only analysis.

For Theorem 1(i), Table 14’s +0.104 is a pooled, outcome-conditioned shaping difference, not a uniform increase in the within-prompt gap. Binary changes g/hg/h from 0.25 to 0.125 on high-cec_e prompts and to 0.321 on low-cec_e prompts. The pooled statistic therefore does not imply increased deferral under the per-prompt criterion hΔ+sgh\Delta+sg.

We extended the single-run comparisons to three seeds under a matched configuration (MathVista, Qwen2.5-VL-7B, G=2G=2):

RewardA1A_1 ↑A2A_2 ↑Deferral ↓
Matched SRPO†52.7 ± 0.865.8 ± 0.428.5 ± 1.0
Binary rcalr_{\mathrm{cal}}57.9 ± 0.767.0 ± 0.419.8 ± 0.8
Continuous rcalr_{\mathrm{cal}}60.0 ± 0.667.6 ± 0.316.6 ± 0.6
SRPO as specified (g=−0.25g=-0.25)58.2 ± 0.767.0 ± 0.421.9 ± 0.2
UASRL61.3 ± 0.667.8 ± 0.413.5 ± 0.7

Values are percentages, mean ± sample SD across seeds 147469, 271483, and 176219. These exclude the original single runs; UASRL uses different seeds from Table 11.

Binary improves mean A1A_1 by 5.2 points over matched SRPO†, versus 8.6 for UASRL. Continuous adds 2.1 points over binary but changes both reward magnitudes and gap signs. These results support effectiveness without establishing sign reversal as the dominant mechanism. We will clarify this distinction in the introduction and §5.5 and label Table 5’s predictions as local-model predictions.

W2: Margin over the simplest fix

SRPO as specified is a strong simple baseline. UASRL improves mean A1A_1 by 3.1 points and reduces deferral by 8.4 points (38% relative); we interpret the smaller 0.8-point A2A_2 gain cautiously with three seeds.

The smaller margin is consistent with the negative-gap baseline already removing the positive deferral bias under our local model. We will foreground this baseline and distinguish the incremental gain from the larger improvement over SRPO†.

W3: Prior observations and response masking

We agree that [1] and [2] previously observed reward-induced wrong-first behavior; our discussion understated this overlap. Following the reviewer’s suggestion, we will credit both explicitly and frame our contribution around the formal analysis of this incentive.

We added an SRPO† + selective-masking control under the matched configuration, retaining first-response policy loss only when first- and final-answer correctness agree. This adapts the masking mechanism rather than reproducing the full Octopus framework.

MathVista, Qwen2.5-VL-7B. Values are percentages, reported as mean ± sample SD.

MethodA1A_1 ↑A2A_2 ↑Deferral ↓
Matched SRPO†52.7 ± 0.865.8 ± 0.428.5 ± 1.0
SRPO† + selective masking56.4 ± 1.267.4 ± 0.714.9 ± 1.0
UASRL61.3 ± 0.667.8 ± 0.413.5 ± 0.7

Masking substantially reduces deferral, while UASRL achieves 4.9 points higher mean first-answer accuracy with comparable final accuracy. We will add this comparison and clarify the distinction between reward shaping and selective masking.

Q1: Training configuration and seeds

Advantages are normalized within each prompt’s rollout group as (ri−rˉ)/(σ+ϵ)(r_i-\bar r)/(\sigma+\epsilon), with ϵ=10−4\epsilon=10^{-4}. Matched runs use G=2G=2, DAPO-style token-level loss aggregation, and no KL penalty. The main-table MathVista SRPO† run instead uses G=4G=4, KL coefficient 0.04, and one epoch; the self-consistency variant also uses G=4G=4. Neither exception applies above. Binary originally used one seed; the new comparison uses three.

Q2: Reward coefficients

We manually compared a small number of settings using MathVista testmini accuracy, without a separate validation split; the MathVista margins should be interpreted accordingly. ScienceQA and MMMU-Pro were not used for selection.

The defaults give geff(ce)=0.25−0.55ce−sgrdg_{\mathrm{eff}}(c_e)=0.25-0.55c_e-s_{\mathrm{grd}} and h(ce)=1.15−0.05ceh(c_e)=1.15-0.05c_e, implementing §3’s design criteria. At zero grounding, the gap is negative for ce≥1/2c_e\geq1/2 on the eight-rollout grid, while hh remains positive. These constraints motivate but do not uniquely determine the coefficients. Table 12 reports sensitivity to the penalty-to-reward ratio, shaping scale, and rollout budget; we do not claim optimality.

[1] Learning Self-Correction in Vision–Language Models via Rollout Augmentation. ICML 2026.

[2] Self-rewarding correction for mathematical reasoning. ArXiv:2502.19613.

Thank you again for your review. We’ve revised the paper based on our responses above, with text changes marked in blue. In particular:

Non-templated experiments: Table 6 now includes the two new settings (InternVL3-8B on MathVista and Qwen2.5-VL-7B on ScienceQA), and §5.6, the Introduction, and the Limitations now state the scope of these results more clearly. Deferral proxy: Appendix B.6 now explains how cec_e is computed in this setting, and clarifies what the deferral proxy does and does not establish. Multi-seed comparison: Table 12 (Appendix C.2) reports the matched three-seed comparison, including SRPO as specified, and §5.4 discusses it.

We hope these changes address your concerns. If anything remains unclear, we’d be happy to discuss further before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.