- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper identifies deferral in reflective RL: correction rewards can encourage poor first answers while preserving final accuracy. It analyzes this incentive and proposes UASRL, which uses per-prompt success rates fixed before RL to reward correct first answers. Experiments on two vision-language models and three main benchmarks, supplemented by ablations, multi-seed runs, and a non-templated experiment, show reduced deferral and improved first-answer accuracy with competitive final accuracy. Most evidence comes from controlled, templated initializations.
Critical strengths
- A consequential failure mode with a clear mechanism. The paper shows how competitive final accuracy can conceal severe first-answer degradation, and connects this behavior to correction incentives through local theoretical analysis and answer-transition measurements. This provides a concrete reason to evaluate both first and final answers in reflective RL.
- An effective mitigation in the evaluated settings. UASRL improves first-answer accuracy over the tested reflection baselines across two backbones and three main benchmarks while maintaining competitive final accuracy. Reward ablations support the usefulness of the fixed capability signal, and the method requires no change to the inference procedure.
Critical weaknesses
- Limited evidence beyond the controlled initialization. The main experiments use SFT data containing 68% deferred trajectories. The non-templated experiment is encouraging, but its labeling rule may conflate deferral with ordinary errors that disappear under a different prompt, leaving the broader applicability of the diagnosis uncertain.
- Table 5 compares UASRL with the strongest uniform baseline, “SRPO as specified,” using single runs. The multi-seed results in Table 11 instead use SRPO†, a different reward variant, leaving the robustness of UASRL’s advantage over the stronger baseline unverified.
Questions for authors
- In Table 6 and Appendix B.6, how was the deferral labeling rule validated against genuine correction or prompt-dependent changes in correctness?
- Can you provide matched multi-seed comparisons between UASRL and the strongest uniform-protection baseline, “SRPO as specified” in Table 5?
3 comments
Thanks. I will keep my score and raise my soundness accordingly.
Thank you for the positive assessment and for raising the soundness score!
We’ve revised the paper based on our responses above, with text changes marked in blue. In particular, we updated the non-templated experiment table (Table 6) with the two new settings, and added a new table in Appendix C.2 with the matched three-seed comparison, including SRPO as specified and selective masking.
Thanks again for the constructive feedback. It really helped improve the paper.
We thank the reviewer for the careful review, the encouraging assessment of our non-templated experiment, and the constructive questions.
W1: Evidence beyond the controlled initialization
The templated initialization is a deliberate controlled setting: the placeholder makes deferral directly measurable, so that we can quantify how each reward amplifies or suppresses it. Our non-templated experiment (Table 6) indicates that deferral also emerges without templates; to further demonstrate this, we have extended it to another backbone and another benchmark.
All settings start from the base model with the reflection prompt and no SFT, using the same GRPO configuration as the main experiments. For UASRL, ce is estimated before RL from eight rollouts of this initialization. Values are percentages from single runs; the deferral proxy is discussed in Q1.
In all three settings, SRPO† lowers first-answer accuracy by 10.4–11.2 points while increasing final-answer accuracy by 4.0–5.5 points, whereas UASRL improves both relative to initialization. Because this pattern uses only accuracies, it does not depend on the deferral labeling rule. It shows that first-answer degradation concealed by improved final accuracy is not confined to templated SFT; the general prevalence of deferral remains open. We will add these results to §5.6 and clarify the scope of the controlled initialization in the introduction and limitations.
Q1: Validation of the deferral labeling rule
The rule in Appendix B.6 combines three conditions: the first answer is wrong, differs from the final answer, and the same checkpoint answers correctly under a single-pass prompt. The third condition provides a capability check: it identifies cases where an incorrect first answer coexists with demonstrated ability to answer the question directly. We assessed the resulting pattern across initialization and the trained checkpoints. Across the three non-templated settings, the proxy rate is 0.3–0.8% at initialization, 1.2–2.7% under UASRL, and 16.0–18.0% under SRPO†. The concentration of this behavior under the positive-gap reward is consistent with our incentive analysis. Independently of the labeling rule, SRPO† reduces first-answer accuracy by 10.4–11.2 points while improving final-answer accuracy, whereas UASRL improves both. These comparisons provide supporting evidence, rather than an independent validation of individual labels. In particular, single-pass success does not exclude a genuine error in another trajectory, and prompt sensitivity may change during training. We will make this distinction explicit in Appendix B.6: the non-templated results directly establish first-answer degradation, while the operational proxy supports its interpretation as deferral without uniquely identifying that mechanism.
W2 / Q2: Multi-seed comparison with SRPO as specified
Table 11 reports multiple seeds for SRPO†, whereas the comparison with SRPO as specified in Table 5 uses a single run. We have now evaluated SRPO as specified over three training seeds under the same initialization, G=2, training budget, and evaluation protocol as UASRL. We also include matched SRPO† for context. Three-seed comparison (MathVista, Qwen2.5-VL-7B, shared templated SFT initialization). Values are percentages, reported as mean ± sample standard deviation across three training seeds. The UASRL row uses a seed set separate from that of Table 11.
Compared with SRPO as specified, UASRL improves mean first-answer accuracy by 3.1 percentage points and reduces mean deferral by 8.4 points. These results provide additional evidence for the advantage observed in the earlier single-run comparison on this backbone and benchmark. Mean final-answer accuracy is also 0.8 points higher; we interpret this smaller difference cautiously given the limited number of seeds. SRPO as specified remains a strong simple baseline: its negative gap already removes the positive deferral bias under our local model. We will add this comparison to the paper, foreground SRPO as specified as the primary uniform baseline, and distinguish UASRL’s incremental gains over it from the larger improvements over positive-gap SRPO†.