- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes J_Q, a family of losses that interpolates between RLVR (q=0) and log-marginal-likelihood over latent rationales (q=1), where all members differ only by a per-example gradient factor P^(-q). A gradient-flow analysis shows that higher q escapes cold start much faster but also fits label noise faster, which explains the SFT-then-RLVR pipeline. The authors derive two estimators, GARL and PAFT, that require no annotated rationales or verifiers. On three QA benchmarks, GARL with high enough q escapes cold start where GRPO scores zero, and intermediate q is more robust to label noise.
Critical strengths
The paper unifies SFT and RLVR under one theoretical framework, in which they are two ends of a single loss family, and this view explains why SFT is usually done before RLVR. The framework also leads to new training methods (GARL and PAFT) that need no annotated rationales and can escape cold start where GRPO fails.
Critical weaknesses
- Missing baselines. The experiments compare only with GRPO and the two endpoints of GARL (q=0 and q=1). Directly relevant methods discussed in the related work, such as MaxRL, STaR, TRICE, and SPG, are not evaluated, which makes it hard to judge the method’s advantage in a crowded area.
- Limited evaluation and applicability. The experiments use only three QA-style benchmarks with short exact-match answers, and do not include standard RLVR benchmarks such as math reasoning (e.g., GSM8K, MATH)。
- The theory does not guide the choice of q. The theory explains qualitatively why higher q escapes cold start and lower q resists noise, but it does not tell how to set q in practice. The critical q differs across models (q ≥ 0.75 for Qwen 3 0.6B, q ≥ 0.85 for Qwen 3 8B, q ≥ 0.5 for Gemma 4 E4B) and is found by sweeping and validation, so the practical value of the theory is limited.
Questions for authors
- Why were FinQA, HotPotQA, and MuSiQue chosen? Can the methods be applied to standard RLVR benchmarks such as math reasoning?
- Can the authors compare with directly relevant methods discussed in the related work, such as MaxRL, STaR, and TRICE, in both the cold-start and prompted settings?
2 comments
Response (2/2)
Q2: "compare with … MaxRL, STaR, and TRICE, in both the cold-start and prompted settings"
The baselines behave differently in the two settings (Table A.1; GSM8K in Table B.1):
- Cold start. MaxRL, STaR and original TRICE get no learning signal. The cause is not their objectives (original TRICE targets ) but their estimators, which update only on sampled successes: MaxRL's rollouts succeeded on 3 of 786 training steps, and STaR and original TRICE find no verified rationale for any training question on any benchmark. With success probabilities of about to per rollout before training (geometric means for Qwen 3 0.6B), these estimators are almost always zero: a variance problem. Scoring samples by the teacher-forced answer likelihood fixes this: continuous-likelihood TRICE escapes cold start, but still ends 5–10 m@16 below GARL at under either rollout allocation.
- Prompted. With task prompts, successes are common and all three train. However, with Qwen 3 0.6B and Gemma 4 E4B, cold-start GARL, without task prompts, still beats every new baseline in either setting on every metric and benchmark.
Response (1/n)
We thank Reviewer Ehrv for the careful review and for recognizing that the framework unifies SFT and RLVR, explains why SFT usually precedes RLVR, and yields methods that escape cold start without annotated rationales.
W1: "MaxRL, STaR, TRICE, and SPG, are not evaluated"
We ran MaxRL and STaR, each from a cold start and with task prompts; original TRICE with task prompts; and continuous-likelihood TRICE (Phan et al., 2023, Appendix A) from a cold start, with 32 proposals per question (as for the other methods) and with one proposal per visit (as in the TRICE paper). All run on FinQA, HotPotQA, MuSiQue and GSM8K (original and Platinum test sets; one-proposal TRICE not on GSM8K).
With Qwen 3 0.6B and Gemma 4 E4B, cold-start GARL outperforms every one of them on all four benchmarks (Tables A.1, A.2, and B.1 in the general response above).
SPG tilts each token by its reward increment, so with our answer-only reward it samples rationales from the model unchanged and fine-tunes on the gold answer. With a log-likelihood reward, its sequence-level target is the q=1 posterior, whose gradient PAFT at q=1 estimates; we report cold-start PAFT at q=1 in its place (Table A.1), which consistently underperforms cold-start GARL.
W2: "do not include standard RLVR benchmarks such as math reasoning (e.g., GSM8K, MATH)" / Q1: "Why were FinQA, HotPotQA, and MuSiQue chosen? Can the methods be applied to standard RLVR benchmarks such as math reasoning?"
We chose FinQA, HotPotQA and MuSiQue because they put the models in cold start, the regime the theory addresses: without a task prompt, every base model scores 0 on all three (Table C). Math starts near the opposite end: Qwen 3 0.6B already reaches 77.6% pass@1 on MATH-500 in thinking mode. Following the suggestion, we added GSM8K and GSM8K-Platinum (Table B.1). With Qwen 3 0.6B, cold-start GARL outperforms all baseline methods on both GSM8K and GSM8K-Platinum. As cautioned in our general response, gains on popular math benchmarks are hard to attribute for Qwen-family models, whose pre-training data may contain them Wu et al., 2025.
W3: "it does not tell how to set q in practice"
First, q=1 always escapes cold start (Theorem 3.2), in all nine settings of the paper's Table 1, at most 5.1 m@16 below the best q: a safe sweep-free default. Second, a lower q, more robust to label noise, can be fixed before training: a run escapes within its step budget when (1−q)log(p01)+log1−q1≤C, so q∗≈1−C/log(p01), where p0 is estimated from one batch of rollouts before training and C depends on the step budget and learning rate.
We calibrated C=13.3 once, conservatively, on Qwen 3 0.6B FinQA (log(p01)=53.3; escaped at q=0.75, and failed to escape at 0.5, gives 13.3≤C<26.7). On GSM8K this fixed q at 0.74, 0.87 and 0.57 for Qwen 3 0.6B, Qwen 3 8B and Gemma 4 E4B. All three escape cold start (Tables B.1–B.3), and with Qwen 3 0.6B, GARL at the predicted q outperforms all baselines on both GSM8K test sets.
The critical q differs across models because p0 does: on FinQA, log(p01) is 53.3, 95.9 and 38.0 for Qwen 3 0.6B, Qwen 3 8B and Gemma 4 E4B, so the predicted thresholds are 0.75, 0.86 and 0.65, each at or above a q that escapes in the paper's Table 1 (0.75, 0.85 and 0.5). This holds in all 8 model × benchmark settings (using Qwen 3 0.6B FinQA is the calibration point). We separately evaluated the only model × benchmark setting whose predicted q lies below the coarse grids we tested (Gemma x MuSiQue, predicted q=0.72). It achieves p@1 / p@16 / m@16 of 55.6 / 77.4 / 59.3 respectively, outperforming all cold-start and prompted baselines.