acceptodds
Under review as a conference paper at ICLR 2027

How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

Abstract

The standard recipe for training reasoning models — SFT on rationale annotations followed by RLVR — relies on rationale annotations being available and clean. When no annotated rationales exist, SFT cannot start at all, and when the model is too unaligned, RLVR alone stalls at cold start; when annotations are weak, SFT memorizes the errors. Using the Tsallis -logarithm, we define a loss family that interpolates between RLVR (at , the _exploitation pole_) and the log-marginal-likelihood over latent trajectories (at , the _density-estimation pole_). The standard pipeline corresponds to a stepwise schedule, and intermediate enables training in regimes where the two-step pipeline cannot apply. All members share the same gradient direction, differing only by a per-instance amplification that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires time to escape cold start but is robust to label noise, while the density-estimation pole escapes in but memorizes label noise. This separation explains why SFT () precedes RLVR () in the standard pipeline. We further derive two Monte Carlo estimators that directly optimize fixed- on the continuum, without annotated rationales or external verifiers: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high escapes cold start where GRPO fails entirely under Qwen 3 and Gemma 4 architectures. Without task prompts, cold-start GARL on Qwen 3 0.6B also matches or exceeds prompted GRPO with reshaped reward on majority vote and best-of- on all three benchmarks.

Primary area:
generative models
Keywords:
RLVR, reasoning models, reinforcement learning, latent variable models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EhrvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes J_Q, a family of losses that interpolates between RLVR (q=0) and log-marginal-likelihood over latent rationales (q=1), where all members differ only by a per-example gradient factor P^(-q). A gradient-flow analysis shows that higher q escapes cold start much faster but also fits label noise faster, which explains the SFT-then-RLVR pipeline. The authors derive two estimators, GARL and PAFT, that require no annotated rationales or verifiers. On three QA benchmarks, GARL with high enough q escapes cold start where GRPO scores zero, and intermediate q is more robust to label noise.

Critical strengths

The paper unifies SFT and RLVR under one theoretical framework, in which they are two ends of a single loss family, and this view explains why SFT is usually done before RLVR. The framework also leads to new training methods (GARL and PAFT) that need no annotated rationales and can escape cold start where GRPO fails.

Critical weaknesses
  1. Missing baselines. The experiments compare only with GRPO and the two endpoints of GARL (q=0 and q=1). Directly relevant methods discussed in the related work, such as MaxRL, STaR, TRICE, and SPG, are not evaluated, which makes it hard to judge the method’s advantage in a crowded area.
  2. Limited evaluation and applicability. The experiments use only three QA-style benchmarks with short exact-match answers, and do not include standard RLVR benchmarks such as math reasoning (e.g., GSM8K, MATH)。
  3. The theory does not guide the choice of q. The theory explains qualitatively why higher q escapes cold start and lower q resists noise, but it does not tell how to set q in practice. The critical q differs across models (q ≥ 0.75 for Qwen 3 0.6B, q ≥ 0.85 for Qwen 3 8B, q ≥ 0.5 for Gemma 4 E4B) and is found by sweeping and validation, so the practical value of the theory is limited.
Questions for authors
  1. Why were FinQA, HotPotQA, and MuSiQue chosen? Can the methods be applied to standard RLVR benchmarks such as math reasoning?
  2. Can the authors compare with directly relevant methods discussed in the related work, such as MaxRL, STaR, and TRICE, in both the cold-start and prompted settings?
2 comments
AuthorsOpenReview ↗

Response (1/n)

We thank Reviewer Ehrv for the careful review and for recognizing that the framework unifies SFT and RLVR, explains why SFT usually precedes RLVR, and yields methods that escape cold start without annotated rationales.


W1: "MaxRL, STaR, TRICE, and SPG, are not evaluated"

We ran MaxRL and STaR, each from a cold start and with task prompts; original TRICE with task prompts; and continuous-likelihood TRICE (Phan et al., 2023, Appendix A) from a cold start, with 32 proposals per question (as for the other methods) and with one proposal per visit (as in the TRICE paper). All run on FinQA, HotPotQA, MuSiQue and GSM8K (original and Platinum test sets; one-proposal TRICE not on GSM8K).

With Qwen 3 0.6B and Gemma 4 E4B, cold-start GARL outperforms every one of them on all four benchmarks (Tables A.1, A.2, and B.1 in the general response above).

SPG tilts each token by its reward increment, so with our answer-only reward it samples rationales from the model unchanged and fine-tunes on the gold answer. With a log-likelihood reward, its sequence-level target is the q=1q=1 posterior, whose gradient PAFT at q=1q=1 estimates; we report cold-start PAFT at q=1q=1 in its place (Table A.1), which consistently underperforms cold-start GARL.


W2: "do not include standard RLVR benchmarks such as math reasoning (e.g., GSM8K, MATH)" / Q1: "Why were FinQA, HotPotQA, and MuSiQue chosen? Can the methods be applied to standard RLVR benchmarks such as math reasoning?"

We chose FinQA, HotPotQA and MuSiQue because they put the models in cold start, the regime the theory addresses: without a task prompt, every base model scores 0 on all three (Table C). Math starts near the opposite end: Qwen 3 0.6B already reaches 77.6% pass@1 on MATH-500 in thinking mode. Following the suggestion, we added GSM8K and GSM8K-Platinum (Table B.1). With Qwen 3 0.6B, cold-start GARL outperforms all baseline methods on both GSM8K and GSM8K-Platinum. As cautioned in our general response, gains on popular math benchmarks are hard to attribute for Qwen-family models, whose pre-training data may contain them Wu et al., 2025.


W3: "it does not tell how to set q in practice"

First, q=1q=1 always escapes cold start (Theorem 3.2), in all nine settings of the paper's Table 1, at most 5.1 m@16 below the best qq: a safe sweep-free default. Second, a lower qq, more robust to label noise, can be fixed before training: a run escapes within its step budget when (1−q)log⁡(1p0)+log⁡11−q≤C(1-q)\log(\frac{1}{p_0}) + \log\frac{1}{1-q} \le C, so q∗≈1−C/log⁡(1p0)q^\ast \approx 1 - C/\log(\frac{1}{p_0}), where p0p_0 is estimated from one batch of rollouts before training and CC depends on the step budget and learning rate.

We calibrated C=13.3C = 13.3 once, conservatively, on Qwen 3 0.6B FinQA (log⁡(1p0)=53.3\log(\frac{1}{p_0}) = 53.3; escaped at q=0.75q=0.75, and failed to escape at 0.50.5, gives 13.3≤C<26.713.3 \le C < 26.7). On GSM8K this fixed qq at 0.740.74, 0.870.87 and 0.570.57 for Qwen 3 0.6B, Qwen 3 8B and Gemma 4 E4B. All three escape cold start (Tables B.1–B.3), and with Qwen 3 0.6B, GARL at the predicted qq outperforms all baselines on both GSM8K test sets.

The critical qq differs across models because p0p_0 does: on FinQA, log⁡(1p0)\log(\frac{1}{p_0}) is 53.3, 95.9 and 38.0 for Qwen 3 0.6B, Qwen 3 8B and Gemma 4 E4B, so the predicted thresholds are 0.75, 0.86 and 0.65, each at or above a qq that escapes in the paper's Table 1 (0.75, 0.85 and 0.5). This holds in all 8 model × benchmark settings (using Qwen 3 0.6B FinQA is the calibration point). We separately evaluated the only model × benchmark setting whose predicted qq lies below the coarse grids we tested (Gemma x MuSiQue, predicted q=0.72q = 0.72). It achieves p@1 / p@16 / m@16 of 55.6 / 77.4 / 59.3 respectively, outperforming all cold-start and prompted baselines.


AuthorsOpenReview ↗

Response (2/2)

Q2: "compare with … MaxRL, STaR, and TRICE, in both the cold-start and prompted settings"

The baselines behave differently in the two settings (Table A.1; GSM8K in Table B.1):

  • Cold start. MaxRL, STaR and original TRICE get no learning signal. The cause is not their objectives (original TRICE targets q=1q=1) but their estimators, which update only on sampled successes: MaxRL's rollouts succeeded on 3 of 786 training steps, and STaR and original TRICE find no verified rationale for any training question on any benchmark. With success probabilities of about 10−2610^{-26} to 10−2310^{-23} per rollout before training (geometric means for Qwen 3 0.6B), these estimators are almost always zero: a variance problem. Scoring samples by the teacher-forced answer likelihood fixes this: continuous-likelihood TRICE escapes cold start, but still ends 5–10 m@16 below GARL at q=1q=1 under either rollout allocation.
  • Prompted. With task prompts, successes are common and all three train. However, with Qwen 3 0.6B and Gemma 4 E4B, cold-start GARL, without task prompts, still beats every new baseline in either setting on every metric and benchmark.
Reviewer_oX5SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to tradeoff between RLVR and maximum marginal likelihood for LLM training. Specifically, this paper tries to address the cold-start problem where annotated reasoning trajectories are hard to find and a two-stage SFT then RLVR process is infeasible. They propose a new training objective where a tunable parameter controls the tradeoff between rewarding successes, and increasing probabilities of correct answers. This paper proposes Monte Carlo gradient estimators to optimize this objective. Experiments on a range of datasets show that their proposed objective enables successful cold-start training where naive GRPO fails.

Critical strengths
  1. Principled framework connecting RL and maximum marginal likelihood: the authors show theoretically how their proposed objective trades off between these two paradigms, providing deeper conceptual understanding. They show how their proposed Monte Carlo estimators can be used to optimize this objective.
  2. Good empirical results: the paper shows how their proposed methods can outperform GRPO, allowing a cold start whereas GRPO fails to succeed since it may not have strong enough initial success rate to reward further successes.
Critical weaknesses
  1. Somewhat limited novelty: previous methods have used teacher-forced answer likelihoods to train without external verifiers, and investigated trading off between RL and maximum likelihood. This paper seems to focus on developing a principled framework and combines ideas from various parts of the literature, which is fine. Hence, why I am giving a weak accept for now.

  2. Disentangling improvements over other methods: It would be helpful to compare to other methods like TRICE and MaxRL that utilize similar techniques, rather than mostly comparing against RL-only baselines. It would be helpful for understanding how important each specific part of the pipeline is: for example, whether the proposed Monte Carlo estimators offer much difference, or whether the most important thing is trading off between RL and maximum likelihood.

Questions for authors

How do the authors view how this work fits into the larger literature? Is it to propose a better method, develop a principled framework and justification?

2 comments
AuthorsOpenReview ↗

Response 1/n

We thank Reviewer oX5S for the thoughtful review, and for recognizing both the principled framework connecting RL and maximum marginal likelihood and the empirical results where GRPO fails to start.


W1: "Somewhat limited novelty: previous methods have used teacher-forced answer likelihoods to train without external verifiers, and investigated trading off between RL and maximum likelihood."

VeriFree (RB-RLOO in the paper) removes the zero-gradient problem of REINFORCE/RLOO at cold start by teacher-forcing the answer, a Rao-Blackwellization. Variance is not the only problem, however: the gradient it estimates is the RLVR (q=0q=0) gradient, whose magnitude is too small at cold start (Theorem 3.1), and RB-RLOO stays at 0 accuracy on all three benchmarks (the paper's Table 1). We will make the distinction between variance and magnitude clearer in the revision.

Beyond the prior connections between SFT and RL surveyed in our related work, a major new contribution of this work is the commitment continuum and its analysis.

To the best of our knowledge, the closest work is MaxRL, which bridges RL and marginal maximum likelihood through pass@TT; its continuum is indexed by compute. Both objectives scale the RLVR gradient ∇θP\nabla_\theta P of each question by a weight that depends on its pass rate PP: JQJ_Q by P−qP^{-q} (Proposition 2.2), and MaxRL with TT rollouts by wT(P)=(1−(1−P)T)Pw_T(P) = \frac{(1-(1-P)^T)}{P}, which is about TT at cold start (P≪1TP \ll \frac{1}{T}). Equating the two weights gives the commitment MaxRL applies at a pass rate PP, qeff=log⁡Tlog⁡(1P)q_{\mathrm{eff}} = \log \frac{T}{\log(\frac{1}{P})}. To match q=0.75q=0.75, the escape threshold for Qwen 3 0.6B, requires T≈p0−0.75T \approx p_0^{-0.75}, about 2×10172\times10^{17} to 3×10193\times10^{19} rollouts per prompt, which is obviously infeasible for model training.

In contrast, JQJ_Q is indexed by commitment directly: GARL trains at every qq in [0,1][0,1] with the same 32 rollouts, at the cost of a finite-sample bias that vanishes as the number of rollouts grows (Theorem 4.1).


W2: "compare to other methods like TRICE and MaxRL … whether the proposed Monte Carlo estimators offer much difference, or whether the most important thing is trading off between RL and maximum likelihood"

Both the estimator design and the commitment magnitude have effects:

  • q=1q=1 is competitive against both likelihood-maximizing baselines TRICE and MaxRL: in particular, rollout budget matched continuous-likelihood TRICE optimizes the same q=1q=1 objective as GARL but ends worse on m@16 with Qwen 3 0.6B and Gemma 4 E4B. Our two estimators of the same q=1q=1 objective differ as well: PAFT reaches 19.3 m@16 on cold-start MuSiQue against 32.5 for GARL, consistent with Proposition D.4 (GARL has lower variance than PAFT for the same objective).
  • The trade-off matters as well: by Proposition D.3, high qq fits wrong labels faster, and with 20% of FinQA labels corrupted, q=1q=1 loses 6.2 m@16 while q=0.75q=0.75 loses 1.5 (RQ2; the paper's Table 2). Among runs that escape, an intermediate qq scores higher m@16 than q=1q=1 in 7 of 9 model × benchmark settings (the paper's Table 1).
AuthorsOpenReview ↗

Response 2/2

Q: "How do the authors view how this work fits into the larger literature? Is it to propose a better method, develop a principled framework and justification?"

We thank the reviewer for the important question. A core thesis of this work is that the continuum between RL and marginal maximum likelihood is itself a family of valid training objectives, with a trade-off between its endpoints: towards maximum likelihood, training escapes cold start faster; towards RL, it fits wrong labels more slowly. This differs from prior connections between RL and maximum likelihood, which treat intermediate objectives as devices for reaching one endpoint:

  • Reward-augmented maximum likelihood and RL-as-inference (Norouzi et al., 2016; Levine, 2018) relate the two endpoints but treat them as separate frameworks.
  • Guu et al. (2017) note that the RL and maximum-marginal-likelihood gradients differ only in how they weight correct samples, and interpolate between them to counter spurious programs, again in service of one target.
  • IWAE and Rényi variational inference (Burda et al., 2016; Li & Turner, 2016) form continua of bounds whose purpose is to tighten towards log⁡p\log p, the q=1q = 1 end.
  • MaxRL (Tajwar et al., 2026) moves towards maximum likelihood by spending more samples, so its intermediate points are truncations of log⁡p\log p rather than targets.
  • STaR and TRICE sit at one endpoint each (q=0q = 0 and q=1q = 1).

In JQJ_Q, by contrast, every qq is an objective in its own right:

  • Its own optimum: each member has a distinct minimizer, the escort distribution θ∗∝α1/q\theta^\ast \propto \alpha^{1/q} (Theorem 2.1). This is mode-seeking for q<1q < 1 and calibrated only at q=1q = 1, so qq sets training-time commitment much as temperature sets it at inference.
  • A predicted trade-off:
    • Cold-start escape takes Θ(p0−(1−q)(1−q))\Theta\big(\frac{p_0^{-(1-q)}}{(1-q)}\big) for q<1q < 1 and Θ(log⁡(1p0))\Theta(\log(\frac{1}{p_0})) at q=1q = 1 (Theorems 3.1–3.2).
    • The same factor P−qP^{-q} speeds the fitting of wrong labels (Prop. D.3).
  • Interior optima in practice: the best qq is interior on most settings. An intermediate qq gets higher m@16 than q=1q = 1 in 7 of the 9 model × benchmark settings, and with 20% of FinQA labels corrupted, q=1q = 1 loses 6.2 m@16 while q=0.75q = 0.75 loses 1.5.

To answer the question directly: both. The framework is the main contribution, and the methods follow from it.

Reviewer_Rf26OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a mathematical connection between reward maximization and likelihood-based training, then proposes two ways to train reasoning models using questions and final answers without annotated reasoning traces.

Critical strengths
  • This paper demonstrates a clear motivation.
Critical weaknesses
  • The experimental comparison with closely related approaches is limited. Although Table 1 includes GRPO and RB-RLOO through GARL at q=0q=0, it does not directly evaluate TRICE or MaxRL. The GARL q=1q=1 ablation tests a shared marginal-likelihood objective, but does not reproduce TRICE’s training algorithm. Consequently, the results leave unclear how the proposed approach compares with alternative methods connecting reinforcement learning and marginal-likelihood training.
  • The successful qq threshold varies across models
Questions for authors
  • Could the authors clarify the practical differences that preclude these comparisons, or provide a matched-budget comparison with a closely related method? For MaxRL, the prompted setting would be particularly informative because successful samples are available, avoiding the zero-success limitation already discussed in the paper.
  • Could authors include zero-shot, few-shot performance in the main experinment table?
  • What is reason behind choosing FinQA, HotPotQA and MuSiQue as benchmarking? Any results in some common reasoning benchmarks, e.g., math or coding? I would like to see results on more benchmarks.
2 comments

We thank Reviewer Rf26 for the careful review and for recognizing the clear motivation of the paper.


W1: "it does not directly evaluate TRICE or MaxRL … does not reproduce TRICE's training algorithm"

We thank Reviewer Rf26 for the suggestion. We ran the following compute-matched MaxRL and TRICE variants on FinQA, HotPotQA and MuSiQue, and on GSM8K, which we evaluate on both its original and Platinum test sets (we did not run the one-proposal TRICE variant on GSM8K):

  • MaxRL: cold start and prompted.
  • TRICE, cold start (continuous variant described in Appendix A of Phan et al., 2023) with two different budget allocation variants:
    • 32 rollouts per question (matches the rest of RL-based methods)
    • 2 rollouts, one-proposal-per-visit with controlled total compute budget (matches the original TRICE paper)
  • TRICE, prompted (binary variant described in the main paper).

The results are in Tables A.1 and B.1 of the general response. While prompted MaxRL and continuous cold-start TRICE are competitive against GRPO on HotPotQA, cold-start GARL outperforms all variants of either MaxRL or TRICE on every metric and benchmark with Qwen 3 0.6B and Gemma 4 E4B.


W2: "The successful (q) threshold varies across models"

First, we note q=1q=1 always escapes cold start: by Theorem 3.2, and in all nine settings of the paper's Table 1, where it is the best qq in two and at most 5.1 m@16 below the best qq in the other seven.

Second, a lower threshold, which is more robust to label noise, can be estimated before training from Theorem 3.2. The theorem gives an escape time of cp0−(1−q)(1−q)c \frac{p_0^{-(1-q)}}{(1-q)}, so a run with a budget of BB steps escapes when (1−q)log⁡(1/p0)+log⁡11−q≤C(1-q)\log(1/p_0) + \log\frac{1}{1-q} \le C, where C=log⁡(Bc)C = \log(\frac{B}{c}) depends on the step budget and learning rate. At cold start log⁡(1p0)\log(\frac{1}{p_0}) is large (roughly 30 to 100 in the GSM8K runs below), while log⁡11−q\log\frac{1}{1-q} stays O(1)O(1) for qq bounded away from 1, so q∗≈1−Clog⁡(1p0)q^\ast \approx 1 - \frac{C}{\log(\frac{1}{p_0})}.

To calibrate CC, one coarse sweep suffices: Qwen 3 0.6B on FinQA has log⁡(1p0)=53.3\log(\frac{1}{p_0}) = 53.3 (the geometric mean, over 64 training questions with 32 rollouts each, of the teacher-forced answer probability before training) and escaped at q=0.75q=0.75 but not at q=0.5q=0.5, so 13.3≤C<26.713.3 \le C < 26.7. We take the conservative end, C=13.3C = 13.3, so the predicted qq is sufficient for escape rather than the minimal value (Gemma 4 E4B, for example, already escapes FinQA and HotPotQA at q=0.5q=0.5). On GSM8K, this fixes qq before training at 0.740.74, 0.870.87 and 0.570.57 for Qwen 3 0.6B, Qwen 3 8B and Gemma 4 E4B. All three escape cold start (Tables B.1 - B.3), and with Qwen 3 0.6B, cold-start GARL at the predicted qq outperforms all baselines on both GSM8K test sets.


Q1: "provide a matched-budget comparison with a closely related method? For MaxRL, the prompted setting would be particularly informative"

The results are in Tables A.1 and B.1 of the general response. With the same budget (32 rollouts per question, batch size 64 and the same number of steps), cold-start GARL outperforms prompted MaxRL on FinQA, HotPotQA, MuSiQue and both GSM8K test sets with Qwen 3 0.6B.


Q2: "Could authors include zero-shot, few-shot performance in the main experiment table?"

We agree zero- and few-shot results are useful for measuring in-context learning performance. The results are in Table C. We will incorporate them in the manuscript.


Q3: "Any results in some common reasoning benchmarks, e.g., math or coding? I would like to see results on more benchmarks."

We chose FinQA, HotPotQA and MuSiQue because they put the models in cold start, the regime the theory addresses: without a task prompt, every base model scores 0 on all three (zero-shot in Table C). For math, the results are in Table B.1. With Qwen 3 0.6B, cold-start GARL outperforms all baseline methods on both GSM8K and GSM8K-Platinum. As we cautioned in our general response above, gains on popular math benchmarks might be hard to attribute for Qwen-family models, whose pre-training data may contain them (Wu et al., 2025)

Thank you for adding the TRICE and matched-budget MaxRL comparisons. These address my main concern, and the GSM8K results make the case for the proposed rule for choosing q more convincing. In the revision, I suggest distinguishing a q that gets training off the ground from one that delivers the best final performance. The new evidence moves me toward weak acceptance.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General response (1/n)

We thank the reviewers for their careful reading. We are grateful for the reviewers' suggestions of baselines, datasets and feedback on the choice of qq: the new experiments further strengthen the evidence that GARL, derived from the JQJ_Q continuum, is a competitive training method. Nonetheless, we would like to highlight the commitment notion as a central contribution, which we believe will benefit the community most: the index qq of a continuum between RLVR and marginal maximum likelihood, which sets how strongly supervision acts when it disagrees with the model, is a training-time counterpart of inference temperature. To the best of our knowledge, our work is the first to characterize commitment in training with latent rationales theoretically, as a family of valid objectives that trade off between cold-start escape and robustness to label noise.

We summarize the main points raised in the reviews and our response below:

  1. More baselines (all reviewers). Rf26 and oX5S asked for likelihood maximizing baselines (TRICE and MaxRL); Ehrv for MaxRL, STaR, TRICE and SPG, in both the cold-start and prompted settings.
    1. We ran compute-matched MaxRL, TRICE and STaR on all three benchmarks with Qwen 3 0.6B (Table A.1) and Gemma 4 E4B (Table A.2). Cold-start GARL, trained without task prompts, scores higher than every new baseline on p@1 and m@16 on all three benchmarks, including baselines trained with task prompts.
    2. Whether the estimator or the trade-off matters (oX5S):
      1. With the same # of rollouts, continuous-likelihood TRICE optimizes the same q=1q=1 objective as GARL but has worse m@16 with Qwen 3 0.6B and Gemma 4 E4B (though with many more training epochs TRICE slightly outperforms GARL under some tasks with Gemma 4 E4B), so the estimator matters;
      2. the trade-off matters as well: too low a qq does not escape cold start, and among the values that escape, an intermediate qq scores higher m@16 than q=1q=1 in 7 of 9 model × benchmark settings (the paper's Table 1); with 20% of FinQA labels corrupted, q=1q=1 loses 6.2 m@16 and q=0.75q=0.75 loses 1.5 (RQ2 in the paper). Moreover, these validation-selected interior q's outperform best TRICE and MaxRL baselines.
    3. SPG tilts each token by its reward increment, so with our answer-only reward it samples rationales from the model unchanged and fine-tunes on the gold answer. With a log-likelihood reward, its sequence-level target is the q=1q=1 posterior, whose gradient PAFT at q=1q=1 estimates; we report cold-start PAFT at q=1q=1 in its place.
  2. Math benchmarks (Rf26, Ehrv).
    1. We added GSM8K and GSM8K-Platinum (Table B). From a cold start, GARL reaches 87.287.2 / 89.889.8 m@16 with Qwen 3 0.6B, above every baseline (TRICE/MaxRL/GRPO) trained with task prompts (at most 83.783.7 / 86.686.6); cold-start GRPO and MaxRL stay at 0. The QA results carry over: continuous-likelihood TRICE also escapes cold start but ends at 81.381.3 / 84.884.8 m@16, cold-start STaR again cannot train (no verified rationale among the 69736973 training questions), and with task prompts STaR peaks at step 50 (80.480.4 / 83.683.6) and then degrades as its responses lengthen (Table B.1).
    2. We originally chose FinQA, HotPotQA and MuSiQue because they put the models in cold start, the regime the theory addresses. On math these models start near the opposite end (Qwen 3 0.6B scores 77.6% pass@1 on MATH-500 in thinking mode), and gains on popular math benchmarks are hard to attribute for Qwen-family models, whose pre-training data may contain them (Wu et al., 2025).
  3. Choosing qq (Rf26, Ehrv).
    1. The paper swept qq on a coarse grid (0,0.25,0.5,0.75,0.85,1\\{0, 0.25, 0.5, 0.75, 0.85, 1\\}) to show the cold-start transition. Two sweep-free choices follow from the theory:
      1. First, q=1q=1 escapes cold start in all nine settings; it is the best qq in two of them and lands 0.7–5.1 m@16 below the best qq in the other seven (the paper's Table 1).
      2. Second, Theorems 3.1–3.2 fix the form of the threshold, q∗≈1−Clog⁡(1/p0)q^\ast \approx 1 - \frac{C}{\log(1/p_0)}, with p0p_0 estimated from one batch of rollouts before training. Calibrating CC on a single run (Qwen 3 0.6B, FinQA, q=0.75q=0.75), we fixed qq before training on GSM8K for all three models (0.740.74, 0.870.87 and 0.570.57): all three escape cold start, reaching 86.986.9, 96.496.4 and 93.193.1 m@16 (Tables B.1 - B.3).
  4. Zero- and few-shot results (Rf26). Added for all three models, with and without task prompts (Table C). Zero-shot without a task prompt, every base model scores 0; the best 3-shot setting stays below cold-start GARL in all nine model × benchmark settings, by 7−427 - 42 m@16.
AuthorsOpenReview ↗

General response (2/n)

  1. Novelty (oX5S).
    1. A major new contribution in this work is the commitment continuum and its analysis. Teacher-forced answer likelihoods (VeriFree) and trade-offs between RL and maximum likelihood (MaxRL) exist in prior work, as we discussed in the related work sections. MaxRL's continuum is indexed by compute: the larger the number of rollouts TT, the closer MaxRL's objective is to marginal maximum likelihood. Both objectives scale the RLVR gradient ∇θP\nabla_\theta P of each question by a weight that depends on its pass rate PP: JQJ_Q by P−qP^{-q} (Proposition 2.2), and MaxRL with TT rollouts by wT(P)=1−(1−P)TPw_T(P) = \frac{1-(1-P)^T}{P}, which is about TT at cold start (P≪1TP \ll \frac{1}{T}). Equating the two weights gives the commitment MaxRL applies at a pass rate PP, qeff=log⁡Tlog⁡(1P)q_{\mathrm{eff}} = \log \frac{T}{\log(\frac{1}{P})}. For Qwen 3 0.6B, the measured log⁡(1p0\log(\frac{1}{p_0} before training is 53.3, 55.7 and 59.8 on FinQA, HotPotQA and MuSiQue, so with T=32T=32 rollouts, qeffq_{\mathrm{eff}} is 0.0580.058 - 0.0650.065. Conversely, matching q=0.75q=0.75, the escape threshold for Qwen 3 0.6B, requires T≈p0−0.75T \approx p_0^{-0.75}, about 2×10172\times10^{17} to 3×10193\times10^{19} rollouts per prompt; and once PP rises above 1T\frac{1}{T}, MaxRL's weight approaches 1P\frac{1}{P}, so no fixed TT applies an intermediate qq throughout training. On the other hand, JQJ_Q is indexed by commitment directly: GARL can train for every qq in [0,1][0,1] with the same 32 rollouts, at the cost of a finite-sample bias that vanishes as the number of rollouts grows (Theorem 4.1). We will make the connection clearer in the revision.

New baselines on FinQA, HotPotQA and MuSiQue

All new baseline experiments follow the same setup in Appendix F unless otherwise described (i.e., TRICE under original paper budget allocation).


Table A.1. Qwen 3 0.6B (test p@1 / p@16 / m@16 at the best-validation checkpoint; one seed unless noted). Rows marked paper Table 1 are from the paper.

Bold: best m@16 per benchmark. Italicized: second-best m@16 per benchmark.

Method (p@1 / p@16 / m@16)FinQAHotPotQAMuSiQue
Cold start: GARL, q=0.75q=0.75 (paper Table 1, 3 seeds)30.5 ± 0.3 / 61.5 ± 0.4 / 38.6 ± 0.653.4 ± 0.6 / 74.6 ± 1.1 / 57.4 ± 0.927.5 ± 1.0 / 58.3 ± 0.6 / 35.6 ± 1.4
Cold start: GARL, q=1q=1 (paper Table 1)21.9 / 58.7 / 33.548.7 / 75.5 / 56.621.6 / 58.1 / 32.5
Cold start: PAFT, q=1q=1 (new)16.0 / 48.9 / 25.141.1 / 68.5 / 49.313.5 / 42.3 / 19.3
Cold start: TRICE, continuous likelihood (TRICE App. A), 32 rollouts per question (as other baselines) (3-seed means)15.4 ± 0.5 / 47.5 ± 1.1 / 23.8 ± 0.841.8 ± 1.1 / 68.7 ± 1.1 / 49.4 ± 0.915.8 ± 0.5 / 48.4 ± 1.0 / 23.7 ± 1.3
Cold start: TRICE, continuous likelihood (TRICE App. A), one proposal per visit (as TRICE paper) (3-seed means)15.6 ± 0.4 / 48.2 ± 1.3 / 25.4 ± 0.544.5 ± 0.2 / 73.2 ± 0.1 / 52.0 ± 0.416.9 ± 0.8 / 50.3 ± 0.3 / 25.8 ± 1.0
Cold start: MaxRL (new); GRPO (paper Table 1)0 / 0 / 00 / 0 / 00 / 0 / 0
Cold start: STaR; TRICE, original (binary likelihood)cannot traincannot traincannot train
Prompted: GRPO, matched reward (revised Table 1, 3 seeds)17.6 ± 3.9 / 45.8 ± 4.2 / 24.8 ± 4.348.5 ± 1.5 / 65.7 ± 0.7 / 51.6 ± 0.719.8 ± 4.7 / 43.1 ± 4.9 / 23.4 ± 5.0
Prompted: GRPO, reshaped reward (paper Table 1; 3 seeds on FinQA, 2 on HotPotQA)30.1 ± 0.1 / 57.9 ± 0.1 / 37.1 ± 0.253.2 ± 0.2 / 67.7 ± 1.4 / 55.4 ± 0.228.9 / 51.7 / 34.0
Prompted: MaxRL, matched reward (3-seed means)22.8 ± 1.7 / 53.2 ± 1.4 / 30.7 ± 1.547.9 ± 1.7 / 67.3 ± 1.7 / 52.1 ± 0.822.7 ± 2.3 / 48.6 ± 3.3 / 27.6 ± 2.6
Prompted: TRICE, original (binary likelihood), matched reward20.6 / 50.1 / 27.642.8 / 64.3 / 46.920.3 / 46.2 / 24.8
Prompted: STaR, matched reward16.7 / 42.8 / 22.338.8 / 60.0 / 43.714.7 / 37.9 / 18.4
AuthorsOpenReview ↗

General response (3/3)

Table A.2. Gemma 4 E4B (test p@1 / p@16 / m@16; one seed).

Rows marked paper Table 1 are from the paper; some of their HotPotQA and MuSiQue cells were not finished by the submission deadline, which are now added here and in Tables D.1 and D.2. Bold: best m@16 per benchmark.

Method (p@1 / p@16 / m@16)FinQAHotPotQAMuSiQue
Cold start: GARL, q=0.75q=0.75 (FinQA and HotPotQA from paper Table 1)45.2 / 69.8 / 50.265.5 / 80.1 / 67.450.6 / 76.0 / 57.2
Cold start: GARL, q=1q=1 (FinQA and HotPotQA from paper Table 1)33.8 / 71.2 / 46.160.0 / 80.2 / 64.651.9 / 81.2 / 59.7
Cold start: TRICE, continuous likelihood (TRICE App. A), 32 rollouts per question (as other baselines)2.0 / 15.0 / 0.053.8 / 79.7 / 61.944.9 / 76.0 / 52.4
Cold start: TRICE, continuous likelihood (TRICE App. A), 2 rollouts per question (as in original TRICE paper)35.5 / 72.8 / 48.655.3 / 84.2 / 64.641.5 / 74.4 / 52.8
Cold start: GRPO (FinQA and HotPotQA from paper Table 1)0 / 0 / 00 / 0 / 00 / 0 / 0
Prompted: GRPO, matched reward (FinQA and HotPotQA from paper Table 1)18.8 / 25.3 / 19.861.6 / 69.7 / 62.654.5 / 70.9 / 56.3
Prompted: GRPO, reshaped reward (FinQA from paper Table 1)19.5 / 26.2 / 20.563.0 / 69.6 / 63.443.0 / 65.5 / 50.0
Prompted: MaxRL, matched reward44.3 / 67.4 / 48.458.9 / 68.1 / 61.944.3 / 65.0 / 50.1

GSM8K and GSM8K-Platinum results

The predicted qq is fixed before training from the estimated p0p_0. The base model scores 0 without a task prompt; its zero-shot row is the mean of three evaluations.

The rollout budgets for all methods are identical to those in the paper: 32 rollouts per question in training and 16 at inference time.


Table B.1. Qwen 3 0.6B.

Method (p@1 / p@16 / m@16)GSM8KGSM8K-Platinum
Cold start: GARL, predicted q=0.74q=0.7481.1 / 94.6 / 86.984.3 / 96.5 / 89.8
Cold start: TRICE, continuous likelihood (TRICE App. A), 32 rollouts per question (as other baselines)72.9 / 92.6 / 81.376.1 / 94.8 / 84.8
Cold start: GRPO, matched reward0 / 0 / 00 / 0 / 0
Cold start: MaxRL, matched reward0 / 0 / 00 / 0 / 0
Cold start: STaRcannot traincannot train
Prompted: GRPO, matched reward73.7 / 92.4 / 82.176.6 / 94.6 / 85.0
Prompted: MaxRL, matched reward73.4 / 92.6 / 82.076.6 / 94.2 / 84.7
Prompted: TRICE, original (binary likelihood), matched reward76.4 / 93.5 / 83.779.1 / 95.2 / 86.6
Prompted: STaR, matched reward72.7 / 91.7 / 80.475.9 / 93.9 / 83.6
Base model, zero-shot with task prompt61.3 / 90.5 / 71.464.3 / 92.6 / 75.2

Table B.2. Qwen 3 8B.

Method (p@1 / p@16 / m@16)GSM8KGSM8K-Platinum
Cold start: GARL, predicted q=0.87q=0.8795.6 / 98.1 / 96.498.3 / 99.8 / 99.0

Table B.3. Gemma 4 E4B.

Method (p@1 / p@16 / m@16)GSM8KGSM8K-Platinum
Cold start: GARL, predicted q=0.57q=0.5789.1 / 97.2 / 93.192.8 / 99.3 / 96.6

Zero- and few-shot results

The 3-shot prompts use three fixed examples drawn from the training set.


Table C. Base models versus cold-start GARL (test m@16 on FinQA / HotPotQA / MuSiQue).

m@16 (FinQA / HotPotQA / MuSiQue)Qwen 3 0.6BQwen 3 8BGemma 4 E4B
Base, zero-shot0 / 0 / 00 / 0 / 00 / 0 / 0
Base, 3-shot13.4 / 38.1 / 13.432.4 / 61.9 / 48.918.0 / 46.6 / 15.7
Cold-start GARL (paper Table 1)38.6 / 57.4 / 35.6 (q=0.75q=0.75)52.9 / 68.6 / 62.9 (q=0.85q=0.85)50.2 / 67.4 / 57.2 (q=0.75q=0.75)

TRICE variants. TRICE (Phan et al., 2023) keeps one rationale per question and updates it by Metropolis–Hastings, with the model as the proposal, training on the kept rationale; both variants target marginal likelihood (q=1q=1). Original TRICE (their Algorithm 1) uses a binary likelihood: it accepts a proposal only if it reaches the gold answer, and it starts from rationales sampled with the answer as a hint, so it needs at least one verified rationale. Continuous-likelihood TRICE (their Appendix A) uses the teacher-forced answer likelihood pθ(y∗∣x,z)p_\theta(y^\ast \mid x, z) instead and accepts a proposal z~\tilde z with probability min⁡1,pθ(y∗∣x,z~)/pθ(y∗∣x,z)\min\\{1, p_\theta(y^\ast \mid x, \tilde z) / p_\theta(y^\ast \mid x, z)\\}, so it can train from a cold start.

We have two 2048-rollout-per-step configs:

  • 32 rollouts per question per update, as for the other baselines (31 of them used as sequential proposals) resulting in 64 examples per step and 2 epochs,
  • and 2 rollouts (one stored + one proposal) per question per visit on 32 times as many questions, as in the TRICE paper, resulting in 1024 examples per step and about 32 epochs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.