- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines whether reasoning skills from RLVR training in math and coding transfer to GQA. The authors find a significant gap: thinking traces help with formal tasks like MATH, but provide little benefit for GQA. To address this, they propose START, a two-stage approach that masks answers during training to force better thinking in phase 1, then optimizes thinking and answering together in phase 2. Experiments show START eliminates thinking stagnation and outperforms standard joint RL training.
Critical strengths
S1. START enables robust thinking evolution through a simple token-level gradient mask during backpropagation. It works well with existing open-source RL frameworks like GRPO and DAPO.
Critical weaknesses
W1. The title claims findings about RLVR, but the GQA experiments use learned reward models (ArmoRM), not verifiable rewards. The MATH comparison changes both task domain and reward mechanism simultaneously, making it unclear which causes the observed thinking stagnation. Please clarify this distinction and limit claims to what the experiments actually demonstrate.
W2. Masking answer-token gradients does not freeze answering behavior. Because thinking and answering share model parameters, updates from thinking tokens still affect the answer distribution. Table 5 confirms this: answering improves by +0.0055 during Phase I, surpassing the +0.0046 gain from updated thinking alone. This contradicts START's central claim that answering becomes stationary and that rewards improve solely through better thinking.
Questions for authors
Q1.How sensitive is START’s performance to the transition point between Phase 1 and 2? Could the authors provide an ablation varying Phase 1 duration under a fixed total training budget, and guidance on selecting the transition point for new model families or task distributions?
6 comments
Thanks for the detailed explanation. RLVR is inherently designed for verifiable tasks, whereas adapting it to GQA relies on a model-based scorer. This raises two lingering questions:
-
Comparison with non-RL baselines: RLVR has gained popularity for its effectiveness in enhancing LLM reasoning and achieving superior performance over non-RL methods like SFT. Without comparing against non-RL baselines on GQA, it remains unverified whether explicit RL-based post-training is necessary for general QA tasks.
-
Reward hacking and upper bound: Unlike verifiers in mathematics, a learned reward model is inherently subjective and susceptible to reward hacking. Also, does the quality of the reward model act as an upper bound on START's performance? If so, why don't we just use the reward model instead?
Q1: Comparison with Non-RL Baselines
R: We thank the reviewer and would like to clarify the scope of our claim.
We do not claim that RL post-training is necessary for GQA final performance, nor that it outperforms SFT. Our claim is: (i) in Section 2.1, we show that GQA thinking efficacy is low and not saturated (ii) our goal is to improve thinking itself; (iii) improving thinking typically requires RL, because SFT and distillation can only imitate an existing answer or thinking distribution, and open-ended GQA provides no gold thinking traces.
A non-RL baseline would test a different question, namely whether SFT can reach a similar final GQA score, and would require oracle supervision unavailable in most open-ended GQA settings. We therefore do not include it.
Q2: Reward Hacking and the Upper Bound of the Reward Model
R:
Reward hacking. We agree that using ArmoRM for both training and evaluation cannot by itself exclude reward over-optimization. We therefore re-evaluated the main comparisons with the independent Llama3-70B judge.
For Table 3, the win rates against the post-trained GRPO baseline are:
| Config. | Post-trained | |
|---|---|---|
| GRPO | --- | --- |
| GRPO-MA | 45.22% | 49.68% |
| +START | 52.23% | 59.23% |
For Table 4, START's win rates against the respective vanilla post-trained baselines are:
| Setting | START Win Rate |
|---|---|
| ExpertQA / Qwen3-1.7B / DAPO | 56.41% |
| UltraFeedback / Qwen3-1.7B / GRPO | 53.75% |
| AlpacaEval / Qwen3-1.7B / GRPO | 52.05% |
| ExpertQA / Hunyuan-1.8B / GRPO | 56.05% |
| ExpertQA / Qwen3-4B / GRPO | 54.14% |
The independent-judge margins are smaller than the ArmoRM-based ones (e.g., 59.23% vs. 68.15% for ), but START remains above 50% in all five settings and the ranking is preserved.
Upper bound. We agree that the fidelity of the reward model bounds how faithful the training signal is, which is a general limitation of learned-reward RL. This bounds signal quality, not the answer quality the policy can reach: the RM is a scoring function rather than a generator, and its preferences define an ordering over responses rather than a ceiling on achievable quality. START is reward-agnostic in this sense—it changes how a given signal is allocated between thinking and answering. Therefore we cannot "just use the reward model instead": scoring answers is not producing them.
We sincerely thank you for your careful review and valuable comments. Below, we provide point-by-point responses to each comment.
W1: Regarding GQA with Learned Reward Models
R: We thank the reviewer for highlighting this distinction. RLVR applies to tasks such as MATH, where correctness can be deterministically verified. When transferring to open-ended GQA, such verifiable rewards are no longer available. We therefore use ArmoRM, a multi-objective reward model that evaluates response quality across multiple preference dimensions (correctness, truthfulness, instruction following), to provide comprehensive feedback for open-ended answers. Thus, the change in reward type follows inherently from the change in task domain, rather than representing an independently varied experimental factor.
To directly evaluate the title claim, we further tested the Qwen3-1.7B model trained on MATH with GRPO on ExpertQA. Its post-RLVR thinking traces, paired with the base answerer, achieve a 45.54% win rate against the base configuration, while the full post-trained model achieves 49.17%. These results provide direct evidence that, in this setting, the thinking improvements obtained through RLVR do not transfer to GQA. We have clarified the distinction between verifiable rewards and learned rewards in the revision.
W2: Regarding Answering Behavior during Phase I
R: We thank the reviewer for identifying this overstatement. Masking answer tokens removes their direct contribution to the policy-gradient objective, but shared-parameter updates still change the answer distribution. The +0.0055 gain for Tpre+Apost shows that Phase I improves both answering behavior and thinking utility; the reward gain should not be attributed solely to thinking. We have therefore replaced “fixed,” “stationary,” and “solely” with the more precise claim that Phase I focuses direct optimization on thinking tokens. Importantly, Tpost+Apre still achieves a +0.0046 gain and a 69.11% win rate when evaluated with the unchanged base answerer, demonstrating improved functional utility of the learned thinking traces.
Q1: Regarding Phase Transition Sensitivity
R: START does not use a predetermined transition step. Phase II begins when the validation Tpost+Apre reward shows no improvement for three consecutive evaluations (approximately 600 steps in our main setting). We also forced an earlier transition at 300 steps while keeping the total training budget fixed:
The early transition mainly weakens standalone thinking utility (52.55% vs. 68.15%). For new models or datasets, we recommend monitoring Tpost+Apre on the validation set and transitioning after three evaluations without improvement. We have added this ablation to the revision.