acceptodds
Under review as a conference paper at ICLR 2027

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

Abstract

Reinforcement learning using verifiable rewards (RLVR) improves LLM reasoning, but the conditions under which it transfers across domains, and why it does so, remain under-explored. We study cross-domain transfer in two model families whose SFT and RL post-training stages use only constraint-satisfaction puzzles, with no mathematics problems in the post-training data, using a reasoning primitive-level analysis framework: a 9-class span classifier that segments chain-of-thought reasoning into primitives and tracks how they are sequenced across recipe stages and domains. We find that puzzle SFT induces a reasoning-primitive vocabulary, and that vanilla GSPO then commits the model to longer uninterrupted chains of reasoning, yielding pass@32 gains on OlymMATH-Hard in both families. But the same training also suppresses backtracking across families and in both domains. To address this, we propose a novelty bonus that rewards diverse correct rollouts using perplexity under the reference model as a signal, designed to preserve the backtracking present in the SFT checkpoint. It brings backtracking back toward its SFT level, and in a controlled A/B test against vanilla GSPO (matched data, algorithm, training budget) it adds a further pass@32 gain on hard mathematics. The end-to-end recipe raises hard-math pass@32 by 20 points on OlymMATH-Hard for OLMo3-7B and 7 points on HMMT for Qwen3-4B without adding any mathematics problems in the post-training stages studied here.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, RLVR, Exploration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TKsWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission studies puzzle-only SFT and RL, evaluates transfer to mathematics, and analyzes generated reasoning through a nine-class primitive classifier. It reports that vanilla GSPO reduces visible backtracking, while a frozen-reference novelty reward restores some backtracking and improves sampled mathematical success.

Critical strengths
  • Useful controlled comparison. Vanilla and novelty GSPO start from the same SFT checkpoint with matched training settings. This meaningfully isolates the reward modification. Substantive behavioral analysis. The taxonomy, blind human annotation, per-problem bootstrap analysis, and worked examples provide more information than accuracy and response length alone.
Critical weaknesses
  1. The distinctive methodological contribution is not sufficiently established. The proposed reward combines several plausible design choices, but the comparison against vanilla GSPO does not clarify which choices are essential. Consequently, it remains unclear whether the gains arise from the proposed frozen-reference formulation or from the more general effect of rewarding uncommon successful trajectories.
  2. The source of cross-domain transfer remains insufficiently disentangled. Although excluding mathematics examples from post-training is a useful experimental choice, it does not distinguish puzzle-specific skill transfer from more general effects of reasoning-trace distillation or improved elicitation of existing knowledge. The experiments therefore establish the effectiveness of the recipe more clearly than they explain why puzzle training transfers to mathematics.
Questions for authors
  1. What is the essential ingredient of the proposed novelty reward? How should we distinguish the benefit of using a frozen reference and top-k NLL from the more general benefit of encouraging exploration among successful trajectories? Which existing evidence best supports these particular design choices?

  2. What does the study establish about the source of puzzle-to-math transfer? How do the authors distinguish the transfer of puzzle-specific problem-solving operations from broader effects of reasoning-trace distillation or improved elicitation of the base model’s existing knowledge?

  3. How should “reasoning diversity” be interpreted in this paper? Does it refer primarily to behavioral variation within individual traces, or to different solution strategies across sampled traces? What evidence connects the reported primitive statistics to the intended notion of diversity?

6 comments
AuthorsOpenReview ↗

Response to Reviewer TKsW (1/3): design ablations and matched exploration baselines (W1, Q1)

Thank you for the constructive review. In our response, we answer W1 together with Q1, W2 together with Q2, and Q3 on its own. To address W1 and Q1, we provide additional ablation experiments and show that each component of our proposed design is essential. To address W2 and Q2, we identify the source of puzzle-to-math transfer as elicitation of mathematical knowledge the base model already holds, through changes in reasoning behaviour at the primitive level. For Q3, we clarify that by reasoning diversity, we mean how differently the model reasons across traces sampled for the same problem, at the primitive vocabulary level.

Additional details for each raised weakness/question are addressed below.

W1 and Q1. Which design choices are essential, and is the gain specific to the frozen-reference top-k formulation?

Yes, in large part: the gain depends on the frozen reference, the normalisation over correct rollouts and the bonus in all-correct groups, and rewarding uncommon successful trajectories in general produces a smaller gain. Each of the four design choices contributes, and no other exploration method we ran matches the gain. The bonus has four design choices: the statistic (mean reference NLL of the 100 highest-NLL tokens of a response), the reference the NLL is computed under (the frozen SFT model), the set the score is normalised over (the correct rollouts of a group), and the treatment of all-correct groups (they keep the bonus, which is then their only source of advantage). We ran one matched run per choice (same recipe, initialisation, data, α = 0.1 and software stack), and evaluated the best checkpoint of each on OlymMATH-Hard (Table T1).

Table T1. OlymMATH-Hard, best checkpoint of each run.

rundesign choice changedpass@1pass@8pass@32
novelty bonus (paper recipe)–4.018.036.3
mean NLL over all tokens instead of the top-100statistic3.616.634.1
current policy instead of the frozen SFT model as referencereference3.013.427.3
bonus on all rollouts instead of correct rollouts onlynormalisation set3.114.230.3
bonus withheld from all-correct groupsgroups kept2.713.831.0
vanilla GSPO (no bonus)–3.213.827.9

Each choice matters:

  • Statistic: the top-100 mean is above the mean over all tokens at every k.
  • Reference: with the current policy as reference, the run still learns the puzzles but its transfer to mathematics stays at the vanilla level.
  • Normalisation set: applying the bonus to correct rollouts only is better than applying it to all rollouts at every k.
  • All-correct groups: in a group where every rollout is correct, the bonus is the only source of advantage. Withholding it from these groups, with everything else unchanged, removes some of the pass@32 gain over vanilla.

Design or general effect. The gain is specific to the design. We trained five other exploration approaches on the same recipe, one per run, and evaluated the best checkpoint of each on OlymMATH-Hard (Table T2).

Table T2. Matched OLMo3-7B runs with one exploration approach each, best checkpoint, OlymMATH-Hard.

runexploration approachpass@1pass@8pass@32
vanilla GSPOnone3.213.827.9
entropy bonusentropy coefficient 0.0012.611.827.0
entropy-advantage (Cheng et al., 2025)α 0.4, κ 22.912.825.7
DIVER (arXiv 2509.26209)potential-shaped global diversity3.113.728.3
unlikeliness (He et al., 2025)accuracy-only rank penalty4.216.332.8
novelty bonus (paper recipe)α 0.1, top-100 tokens4.018.036.3

Table T2 shows that not every exploration approach transfers effectively: the entropy approaches and DIVER land at or below the vanilla level. Among existing alternate approaches, unlikeliness is the closest form of rewarding uncommon successful trajectories and it does transfer, but it differs from our reward in what the reference is (unlikeliness uses the current policy rather than the frozen SFT model our approach uses), the statistic (the rank of whole-sequence likelihood rather than top-k token NLL) and the way it enters the reward (a multiplicative rank discount rather than an additive z-score) at once, so its result speaks to the general effect you name and cannot separate these differences. The ablation runs in Table T1 answer which components of our reward matter. The novelty bonus is above it at pass@8 and pass@32 and level with it at pass@1.

Revision: In the revised version we will add an appendix with Tables T1 and T2, and Section 5 will state the four design choices and point to the ablation results.

AuthorsOpenReview ↗

Response to Reviewer TKsW (2/3): source of the transfer: distillation, elicitation, what transfers (W2, Q2)

W2 and Q2. What is the source of puzzle-to-math transfer?

The transfer is carried by two domain-general reasoning operations that puzzle RL changes and that we measure on the mathematics traces themselves: the length of uninterrupted compute chains, which RL increases, and recovery after an error, which vanilla RL suppresses and the bonus restores. No mathematics enters post-training, so these changes elicit mathematics the base model already holds. Two inference-time interventions show that changing the restored recovery changes accuracy on mathematics: supplying a one-sentence backtrack cue at an error raises the vanilla checkpoint's accuracy by 5.6 points, and removing the recovery spans from the novelty checkpoint's own traces lowers its pass@8 by 5.2 points and brings it to the vanilla level (Tables T3 and T4). The three sources you name are each present in our pipeline at a different stage, and we take them in turn.

Is the gain reasoning-trace distillation? Partly yes. The SFT stage is distillation from puzzle reasoning traces, and Section 4.1 shows that it installs the teacher's reasoning primitive profile in the model, on puzzles and on mathematics alike. The RL stages add no traces: vanilla GSPO and the bonus train on puzzle rewards only, and they move OlymMATH-Hard pass@32 from 23 (SFT) to 29 and 36. The gain beyond SFT is therefore not distillation. We do not claim that puzzle traces are the only traces that transfer, and the paper's claims are about what RL then does and how.

Is the gain elicitation of existing knowledge? Yes, and we can measure how much by looking at newly unlocked problems. Table T6 tracks which OlymMATH-Hard problems each checkpoint solves at least once in 32 samples, and how the set changes at each stage of the pipeline.

Table T6. OlymMATH-Hard, 100 problems, 32 samples per checkpoint. Problems gained and lost at each stage, relative to the previous checkpoint. At 32 samples, pass@32 equals the share of problems solved at least once.

transitionpass@32problems gainedproblems lostnet
base → puzzle SFT15 → 23+13−5+8
puzzle SFT → vanilla GSPO23 → 29+16−10+6
puzzle SFT → novelty GSPO23 → 36+16−3+13
vanilla GSPO → novelty GSPO29 → 36+16−9+7

Every stage reaches problems the previous one did not, without any mathematics in training, so the gain is elicitation of what the base model holds. The bonus extends the reach of the model while preserving what SFT could already do.

What transfers? A change in domain-general reasoning operations, not puzzle-specific ones. The paper measures the puzzle and mathematics traces at every stage with the same primitive classifier, so the transferred change is described at the level of reasoning operations. SFT installs the teacher's primitive profile in both puzzles and mathematics. Vanilla RL lengthens the uninterrupted compute chains (chain fraction 0.93 → 0.98 on OlymMATH-Hard) and suppresses backtracking (0.41 → 0.08 episodes per trace). The novelty bonus restores backtracking to the SFT level (0.41) and the chain fraction with it (0.93), and accuracy improves with these changes.

Interventions. To test whether recovery drives the novelty checkpoint's gain, we ran two inference-time interventions that hold the model fixed and change only recovery. If recovery is linked to the gain, supplying it should raise accuracy and removing it should lower it.

Supplying recovery. We cut failed OlymMATH-Hard traces of both checkpoints at the start of the sentence that states the final, wrong answer, so the error is in the prefix, and let both checkpoints continue every prefix with the same 40k-token budget. At the cut we inserted either a one-sentence backtrack cue ("Wait — this contradicts an earlier assumption…") or a length-matched control sentence about notation, unrelated to recovery.

Table T3. Continuations from error prefixes: unprompted backtracking and recovery under the control sentence, and recovery under the cue.

prefixescontinuerncontrol: backtrackedcontrol: solvedcue: solved
vanilla checkpoint's own errorsvanilla1,2001%0.3%5.9%
vanilla checkpoint's own errorsnovelty1,20011%2.0%10.1%
novelty checkpoint's errorsvanilla1,1682%0.9%7.1%
novelty checkpoint's errorsnovelty1,16810%1.4%9.3%
AuthorsOpenReview ↗

Response to Reviewer TKsW (3/3): interventions on recovery (W2, Q2); definition of reasoning diversity (Q3); summary of changes

In Table T3, backtracked is the share of continuations in which the classifier detects a backtrack, and solved the share whose final answer is correct. The cue raises the vanilla checkpoint's accuracy by 5.6 points over the control. After vanilla RL, backtracking is suppressed in its initiation (1% of continuations against 11% for the novelty checkpoint on the same prefixes) but, once initiated through a recovery cue, the vanilla checkpoint recovers, though not as effectively as the novelty checkpoint on the same prefixes (5.9% vs 10.1%).

Removing recovery. We removed recovery from whole traces of the novelty checkpoint (100 OlymMATH-Hard problems, 32 traces per condition), so the checkpoint is the same in both conditions and only recovery differs. An LLM judge flags recovery spans as they close. In the ban condition a flagged span is deleted and regenerated, and the control condition is the same generation loop with the censor off. A manipulation check confirms that the censor removes almost all recovery spans (Table T4).

Table T4. OlymMATH-Hard accuracy of the novelty checkpoint with recovery spans banned, against its loop control and the vanilla checkpoint.

conditionpass@1pass@8pass@32
novelty, unmodified4.018.036.3
novelty, loop control3.816.331.0
novelty, recovery banned2.411.124.0
vanilla, unmodified3.213.827.9

Together, the two interventions support the reading that the restored recovery mediates the bonus's gain on mathematics, and that this recovery is what the puzzle training transfers.

Revision: We will add an appendix with the per-problem gained and lost sets of Table T6. Section 5 will address the three sources you name, distillation, elicitation and the transferred primitives, in turn.

Q3. How should "reasoning diversity" be interpreted?

By reasoning diversity we mean how differently the model reasons across traces sampled for the same problem. Each trace is a sequence of the primitives of Section 4 (plan, hypothesise, check, backtrack and the others), and diversity is how much these sequences differ from one sample to the next. It is not variation within a single trace, and it is not variation in wording. We will state this definition in Section 1, where the term is first used. We measure differences using edit distance on the sequence of primitives, which we detail below.

Measurement. On math rollouts of the SFT, vanilla and novelty checkpoints (OlymMATH-Hard, 100 problems, 32 traces per problem), we reduce each trace to its primitive sequence, merging adjacent repeats, and compute the mean pairwise edit distance among the 32 sequences of a problem, normalised by sequence length (0 means all 32 traces follow the same sequence).

Table T5. Across-sample reasoning diversity on OlymMATH-Hard: mean pairwise normalised edit distance between the primitive sequences of the 32 traces of a problem, averaged over 100 problems.

checkpointacross-sample distance
SFT0.630
vanilla GSPO0.608
novelty GSPO0.632

Reasoning diversity goes down under vanilla RL and comes back under the bonus: the vanilla checkpoint's traces follow more similar primitive sequences for the same problem than the SFT checkpoint's, and the novelty checkpoint's traces return to the SFT level. This is the across-sample counterpart of the per-trace statistics of Table 6 of the paper, which are averaged over all traces of a dataset and so describe what a typical trace does, while the across-sample distance measures how much the traces differ from each other. It shows in accuracy as a novelty − vanilla pass@k gap that grows with k (W2 and Q2 above).

Revision: We will state the definition in Section 1 and add the across-sample measurement (Table T5) to the appendix.

Summary of changes

  1. We will add the single-change ablations of the reward and the matched exploration baselines (Tables T1, T2) in a new appendix.
  2. We will add the per-problem gained and lost sets across the pipeline (W2, Table T6) in a new appendix.
  3. We will state the definition of reasoning diversity in Section 1 and add its across-sample measurement to the appendix.

We are happy to clarify any remaining point.

Reviewer_Pp35OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the problem that whether reinforcement learning with verifiable rewards (RLVR) can transfer across domains. Based on the primary base model OLMo3-7B-Instruct-SFT, the authors use puzzle-only SFT and GSPO, then analyze CoT traces with a 9-class span classifier. On the OlymMATH-Hard, pass@32 improves from 16.0% to 23.0% (SFT), 29.0% (vanilla GSPO), and 36.0% with the proposed frozen-reference novelty bonus. The experiments suggest that vanilla RL increases compute–verify depth but suppresses the backtracking that puzzle SFT introduced. The proposed novelty bonus partially restores part of the suppressed backtracking in both model families.

Critical strengths
  1. The SFT and RL stages use only constraint-satisfaction puzzles, while evaluation is conducted on three hard out-of-domain mathematics benchmarks. This provides a relatively clear analysis on whether post-training can transfer across domains and why it does so.

  2. The primitive-level analysis motivates the proposed frozen-reference novelty bonus, and the matched vanilla-GSPO comparison demonstrates certain performance gains. This paper also introduces a targeted intervention designed to preserve the backtracking what vanilla RL suppresses, and shows that the restored backtracking accompanies the held-out math gain in both families

Critical weaknesses
  1. Generalization beyond the current suite. This paper mainly conducts experiments based on the 7B base model, one RL algorithm family (GSPO), and one puzzle domain. Thus, please consider performing experiments based on larger base models and extending to broader reasoning tasks to substantiate cross-domain robustness.
Questions for authors

See Weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pp35: transfer to competitive programming, RL algorithm choice, puzzle domain

Thank you for the review and for the suggestion to look beyond mathematics: we thus evaluated the puzzle-trained checkpoints on competitive programming and found a positive transfer result. We provide further details and address each weakness and concern below.

W1. Generalisation beyond the current suite: other RL algorithms, broader tasks

Other RL algorithms. We used GSPO due to its stability on long generated sequences. The target responses are 25–28k tokens, and in our pilot experiments at that length GSPO trained reliably whereas token-level GRPO did not (Section 3). The bonus enters as a reward term, added to the verifier reward before group normalisation, and due to compute constraints we have not run it under another algorithm.

One puzzle domain, broader tasks. The training distribution is only the puzzle domain by design, so that no mathematics or code enters training and any gain on mathematics is due to transfer alone. The puzzle domain contains four puzzle types (Bridges, Pattern, Undead, Galaxies) and each puzzle type has different constraint structures and rules. So even though the domain is puzzles, the four are different puzzle types and solving each requires a different approach. In the paper, we evaluated held-out puzzles on larger grids in-domain and one out-of-domain task, mathematics. We further add a new competitive coding transfer experiment to address generalisation to another domain different from mathematics.

Competitive Coding Transfer Experiment. Based on your suggestion, we evaluated transfer performance on competitive coding and find that a second held-out domain, competitive programming, has similar results to the mathematics evaluation. We evaluated the same puzzle-trained checkpoints, which saw no code in training, on LiveCodeBench, restricted to the 182 problems released after the base model's data cutoff (contest date ≥ 2025-01-01; 45 easy / 55 medium / 82 hard), with the same sampling as the mathematics evaluations. A sample counts as correct only if it passes every public and private test (Table P1).

Table P1. LiveCodeBench, 182 problems released after the base model's data cutoff.

checkpointpass@1pass@8pass@32
base (OLMo-3-7B-Instruct-SFT)17.428.334.6
puzzle SFT21.340.852.2
vanilla GSPO20.838.550.5
novelty bonus25.847.858.8

The novelty checkpoint is above vanilla, SFT and base at every k.

Revision: In the revised version we will add an appendix with the code evaluation.

Summary of changes

  1. We will add the competitive-programming evaluation of the puzzle-trained checkpoints (Table P1) in a new appendix.

We are happy to clarify any remaining point.

AuthorsOpenReview ↗

Follow-up to Reviewer Pp35: larger base model (OLMo-3.1-32B)

To test whether the recipe holds for larger base models we ran the same pipeline on OLMo-3.1-32B: puzzle-only SFT, then vanilla GSPO and the novelty bonus from the same SFT checkpoint, with the bonus hyperparameters unchanged from the 7B runs and a 32k-token training budget. Table P2 gives OlymMATH-Hard pass@k.

Table P2. OLMo-3.1-32B, OlymMATH-Hard, 100 problems, 32 samples per problem.

checkpointpass@1pass@8pass@32
puzzle SFT8.131.551.0
vanilla GSPO8.634.657.0
novelty bonus8.834.562.0

Both RL arms improve on the SFT at 32B, and the novelty bonus is ahead of vanilla at pass@32. We will add the 32B results to the appendix.

We thank Reviewer Pp35 again for the constructive review, which prompted the larger-model experiment and competitive coding evaluations. Our response, the follow-up and the revised PDF address the point raised, and we would welcome any further questions or clarifications before the discussion period closes today.

Reviewer_nZpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether post-training solely on constraint-satisfaction puzzles improves language models' performance on difficult mathematics. Across two model families, the reported SFT-to-GSPO experiments and reasoning-primitive analysis associate gains on some mathematics evaluations with reduced backtracking, motivating a proposed tradeoff between sustained execution and error recovery. The novelty bonus favors correct trajectories that are relatively uncommon under a frozen SFT reference model. Matched comparisons report partial recovery of backtracking-related statistics and pass@32 gains over vanilla GSPO.

Critical strengths

● The paper asks whether training on other verifiable tasks can improve mathematical problem solving without adding target-domain mathematics problems, and which reasoning behaviors may transfer alongside that improvement. This setting focuses attention on how existing knowledge is organized and used, narrowing the explanation based on direct exposure to additional mathematics data. It is a meaningful direction for distinguishing task adaptation from transferable problem-solving behavior. ● Stage-specific checkpoints and matched reward branches allow gains associated with distillation, vanilla RL, and the reward modification to be examined separately. Two model families with different starting points and evaluations across tasks extend the observations beyond a single checkpoint or test set. Although these controls do not establish a specific mechanism, the paper provides a useful empirical framework for studying how post-training changes cross-domain performance. ● Primitive-sequence analysis makes the question of what training changes observable, while the novelty bonus turns the diagnosis into an implementable intervention. This brings performance, behavioral organization, and the training objective into a common framework rather than stopping at score comparisons. Even though the current causal explanation remains incomplete, this research approach generates more specific, testable hypotheses and provides a useful starting point for subsequent mechanistic studies.

Critical weaknesses

● The explanation of why transfer occurs remains an appealing but insufficiently identified mechanistic hypothesis. The results connect improved performance with changes in backtracking-related behavior, but do not distinguish changes in recovery ability from more accurate initial approaches, altered computational allocation, or changes in expression style. The reward intervention also affects several behaviors at once and does not isolate effective backtracking as the cause of the gains. Because restoring reasoning behavior suppressed by RL is the paper's central explanation, this gap directly limits its main scientific contribution: the evidence supports a phenomenon worth investigating more strongly than an established mechanism of transfer. ● The independent methodological advance and new understanding remain insufficiently clear in the context of prior work. Puzzle transfer has precedents such as Logic-RL^1, behavioral analysis has been studied in ThinkARM^2, and rewards for rare successful trajectories have been explored in Rewarding the Unlikely^3 and Rewarding the Rare^4. The paper combines these directions in a new experimental setting, but comparison with vanilla GSPO does not establish that the benefits arise from its distinctive design or rule out similar results from existing exploration-preserving methods. With the mechanistic explanation still unestablished, the remaining contribution is primarily an empirical extension of a training recipe, leaving its status as an independent methodological advance insufficiently convincing. ● The paper's results show that recovered backtracking-related statistics do not always accompany higher accuracy, and some gains do not correspond to substantially more backtracking. This suggests different roles for effective correction, unproductive repetition, and general exploration, which the training objective does not adequately distinguish. Together with limited analysis of actual inference cost and robustness, this makes it difficult to determine whether the method is a broadly useful post-training improvement or a recipe effective for particular starting models, difficulty levels, and sampling configurations. This boundary matters both for adoption and for the broader interpretation that preserving recovery behavior benefits transfer.

Questions for authors

● Does vanilla RL actually reduce recovery ability after an error occurs? Are there recovery rates conditioned on prefixes containing actual errors, or continuation comparisons using the same erroneous prefixes and remaining token budgets? I would like to distinguish reduced need for backtracking from reduced ability to recover when backtracking is needed; counts in naturally generated traces do not directly answer this question. ● Is backtracking a causal mediator of the novelty bonus's gains? Since Table 2 shows changes in both backtracking statistics and accuracy, have you compared continuations that retain, replace, or remove specific recovery actions, using semantically unrelated but length-matched edits as controls? Such results would help distinguish gains caused by backtracking itself from gains caused by other behaviors changed by the reward. ● Where do the Top-100 tokens actually occur? Is novelty computed only over the reasoning body, or does it include the answer, tags, and other formatting? Could you label a sample of the selected tokens as strategy decisions, effective checks, ordinary computation, or irrelevant tail content, and test the relationship between the score and useful exploration after controlling length within each problem? ● What role do all-correct groups play in optimization? When base rewards are identical and stabilization terms and differing KL terms are ignored, subsequent GSPO normalization cancels α. Does the implementation retain these originally zero-advantage groups, and how much do these groups contribute to updates or gradients? If such groups are excluded as in Rewarding the Unlikely, does the novelty bonus retain its main gains?

6 comments
AuthorsOpenReview ↗

Response to Reviewer nZpT (1/5): summary of the response; Intervention 1, supplying recovery (W1, Q1)

Thank you for the careful review and for the two experiments you proposed. We provide a summary of our responses below:

  • To address the concern that the evidence for the mechanistic hypothesis was correlational only, we ran two intervention experiments:
    • (W1, Q1) In Intervention 1, we supply a single backtrack cue at an error point of the reasoning trace. We find that this intervention raises recovery on both checkpoints, and the vanilla checkpoint rarely initiates recovery on its own (1% of continuations against 11% for the novelty checkpoint).
    • (Q2) In Intervention 2, we remove recovery spans from the novelty checkpoint's own traces. We find that this intervention lowers its accuracy to the vanilla level.
    • Recovery therefore moves accuracy in both directions, providing intervention-based support for the mechanistic hypothesis.
  • (W2) To address the concern about comparison with previous work, we compare the proposed bonus with five other existing exploration methods on the same recipe. We find that the proposed approach is above every other method at pass@8 and pass@32.
  • (W3) To address whether the method is broadly useful, we also report a transfer evaluation on competitive programming, where the novelty checkpoint is above vanilla, SFT and base at every k.

Detailed responses to the specific questions are provided below.

W1, Q1 and Q2. Is restored recovery the cause of the gain?

Yes. Using the inference-time interventions you proposed, we show that changing recovery changes accuracy in both directions. We agree that the submission's evidence for the relationship between hard-math pass@32 and backtracking was correlational: the paper shows that a training-time intervention, the novelty bonus, reverses the suppression of backtracking seen under vanilla GSPO (Table 6 of the paper), but the link between the restored backtracking and the pass@32 gain rested on the two occurring together. The new experiments hold the model fixed and change only recovery. We expect that if the recovery is mechanistically linked to the gain, supplying it should raise accuracy and removing it should lower it.

Intervention 1 separates two parts of recovery ability. Vanilla RL suppresses the initiation of a recovery: at vanilla rollout prefixes containing an error, the vanilla checkpoint initiates backtracking in 1% of continuations, while the novelty checkpoint does in 11%. The ability to recover once a backtrack is under way is only partly lost in the vanilla checkpoint: given a one-sentence cue, the vanilla checkpoint recovers 5.9% of the errors against 0.3% on its own, but less often than the novelty checkpoint on the same prefixes (10.1%). Intervention 2 removes recovery spans from the novelty checkpoint's own traces and this costs it about a third of its per-trace accuracy, bringing pass@32 to or below the vanilla level.

Intervention 1, supplying recovery (Q1). On the 30 OlymMATH-Hard problems that both checkpoints solve 1–20 times in 96, we took 2,368 failed traces (1,200 from the vanilla checkpoint, 1,168 from the novelty checkpoint) and cut each at the start of the sentence that states its final, wrong answer, so the error is in the prefix and the answer has not yet been written. Both checkpoints continue every prefix with the same 40k-token budget, so the initial approach and the compute are the same for both and a need for recovery is present in every case. At the cut we inserted either a one-sentence backtrack cue ("Wait — this contradicts an earlier assumption…") or a length-matched control sentence about notation, unrelated to recovery.

Table R1. Continuations from error prefixes: unprompted backtracking and recovery under the control sentence, and recovery under the cue.

prefixescontinuerncontrol: backtrackedcontrol: solvedcue: solved
vanilla checkpoint's own errorsvanilla1,2001%0.3%5.9%
vanilla checkpoint's own errorsnovelty1,20011%2.0%10.1%
novelty checkpoint's errorsvanilla1,1682%0.9%7.1%
novelty checkpoint's errorsnovelty1,16810%1.4%9.3%

In Table R1, backtracked is the share of continuations in which the classifier detects a backtrack, and solved the share whose final answer is correct, under the control sentence and under the cue.

The cue raises the vanilla checkpoint's accuracy by 5.6 points over the control. After vanilla RL, backtracking is suppressed in its initiation but, once initiated through a recovery cue, the vanilla checkpoint recovers, though not as effectively as the novelty checkpoint on the same prefixes (5.9% vs 10.1%). The pattern is the same on the novelty checkpoint's error prefixes (rows 3 and 4).

AuthorsOpenReview ↗

Response to Reviewer nZpT (2/5): Intervention 2, removing recovery (Q2); matched exploration baselines and design ablations (W2)

Intervention 2, removing recovery (Q2). We removed recovery from whole traces of the novelty checkpoint (100 OlymMATH-Hard problems, 32 traces per condition, the evaluation's own sampling), so the checkpoint and its writing style are the same in both conditions and only recovery differs. An LLM judge (GPT-5-mini) reads each reasoning span as it closes and flags recovery spans (reactions to the current line being wrong, stuck or unpromising). In the ban condition a flagged span is deleted and regenerated from the same prefix until the continuation is not a recovery span. The control condition is the identical chunked generation loop with the censor off. In both conditions the model receives no extra instruction or marker. A manipulation check confirms that the censor removes almost all recovery spans (Table R2).

Table R2. OlymMATH-Hard accuracy of the novelty checkpoint with recovery spans banned, against its loop control and the vanilla checkpoint.

conditionpass@1pass@8pass@32
novelty, unmodified4.018.036.3
novelty, loop control3.816.331.0
novelty, recovery banned2.411.124.0
vanilla, unmodified3.213.827.9

We find that banning recovery spans brings the novelty checkpoint to or below the vanilla level at every k (24.0 vs 27.9 at pass@32), which provides support for the hypothesis that recovery spans act as a mediator of the bonus's gain.

Revision: In the revised version we will add an appendix, "Intervening on recovery", with both interventions, and a summary in Section 5.

W2. Is the gain specific to the design, and what is new relative to prior work?

The gain is not shared by other exploration methods. We trained five alternatives on the same recipe, initialisation, data and software stack, one exploration approach each, and evaluated the best checkpoint of each run on OlymMATH-Hard (100 problems, 96 samples per problem).

Table R3. Matched OLMo3-7B runs with one exploration approach each, best checkpoint, OlymMATH-Hard.

runexploration approachpass@1pass@8pass@32
vanilla GSPOnone3.213.827.9
entropy bonusentropy coefficient 0.0012.611.827.0
entropy-advantage (Cheng et al., 2025)α 0.4, κ 22.912.825.7
DIVER (arXiv 2509.26209)potential-shaped global diversity3.113.728.3
unlikeliness (He et al., 2025)accuracy-only rank penalty4.216.332.8
novelty bonus (paper recipe)α 0.1, top-100 tokens4.018.036.3

Table R3 shows that not every exploration approach transfers: the entropy approaches and DIVER land at or below the vanilla level. Unlikeliness does transfer but the novelty bonus is above it at pass@8 and pass@32 and level with it at pass@1. The gain also depends on the design itself. We changed one design choice at a time, keeping everything else fixed, and evaluated the best checkpoint of each run (Table R4).

Table R4. Single-change ablations of the novelty bonus, OlymMATH-Hard, best checkpoint of each run.

rundesign choice changedpass@1pass@8pass@32
novelty bonus (paper recipe)–4.018.036.3
mean NLL over all tokens instead of the top-100statistic3.616.634.1
current policy instead of the frozen SFT model as referencereference3.013.427.3
bonus on all rollouts instead of correct rollouts onlynormalisation set3.114.230.3
bonus withheld from all-correct groupsgroups kept2.713.831.0
vanilla GSPO (no bonus)–3.213.827.9

Each of the four choices contributes to the gain over vanilla.

AuthorsOpenReview ↗

Response to Reviewer nZpT (3/5): relation to prior work (W2); accuracy against recovery, repetition check, inference cost (W3)

What is new relative to the four works is the object of study: a named reasoning primitive, backtracking, that vanilla RL suppresses and the novelty bonus restores, with the interventions of W1. Logic-RL, Rewarding the Unlikely and Rewarding the Rare are cited and positioned in the paper (Sections 1, 2 and 5). We will add ThinkARM to Section 2 in the revision. Logic-RL reports pass@1 on AIME/AMC after puzzle RL and does not ask what changed in the traces or whether the capability ceiling moved. In contrast, we measure pass@k up to 32 and trace the change to backtracking primitives and longer compute chains. ThinkARM segments released traces of existing models into Schoenfeld-style episodes and relates episode patterns to correctness and to efficiency methods. It trains no models and reports no comparison of RL with SFT. We train each stage ourselves and intervene, with the novelty bonus at training time and on the primitive at inference time. Rewarding the Unlikely is run as a matched baseline above. Of the arms we ran, unlikeliness is the closest in mechanism to rewarding rare successful trajectories, and DIVER the closest in goal, trajectory diversity.

Revision: We will add DIVER to the citations in Section 2 and a new appendix table with the matched exploration runs and the single-change ablations.

W3. Recovered backtracking does not always accompany higher accuracy; is the method broadly useful?

Yes, with some nuances. Restored recovery is necessary for the gain on the hard sets (W1), not sufficient for a gain on every set, which is what showed up as an inconsistent correlation between recovery and performance. The weakness additionally raises four points: (1) the rows where accuracy and recovery move apart, (2) whether the restored backtracking is repetition or idle exploration, (3) inference cost, (4) robustness and whether this is a broad post-training improvement.

(1) Where accuracy and recovery move apart. In the submission, we find that accuracy on these problems is correlated with two things, longer uninterrupted computation and effective backtracking. Accuracy can rise without a rise in recovery: vanilla RL's gain over SFT on OlymMATH-Hard (23 → 29 pass@32) comes with longer uninterrupted chains while backtracking is suppressed. RL with the bonus restores backtracking to the SFT level and gives a larger gain over SFT (23 → 36), and we find that it does not give back the accuracy that came with vanilla RL: at pass@1 it is at or above vanilla, and its lead over vanilla grows with k. Recovery can also be restored without a pass@32 gain: on OlymMATH-Easy and AIME25 we find that backtracking recovers under the bonus while pass@32 stays within two points of vanilla. We hypothesize this is because pass@32 is already saturated on these sets, at 71–77, so the ceiling has little room to move. On the unsaturated sets, OlymMATH-Hard and HMMT, the bonus improves pass@32.

(2) Repetition and idle exploration. The bonus is computed among correct rollouts only, so unproductive behaviour is not rewarded unless it co-occurs with a correct answer. We checked the restored backtracks directly: a GPT-5-mini judge read 300 backtrack episodes per checkpoint on OlymMATH-Hard in context and labelled each as unproductive repetition or not: SFT 10.0%, vanilla GSPO 2.7%, novelty bonus 3.0%. RL removes the SFT's repetition loops in both runs, and the bonus does not bring them back. We also show in Q2 that the restored backtracking is not idle exploration either: banning recovery spans brings accuracy to or below the vanilla level.

(3) Inference cost. Mean tokens per response for the paper checkpoints, identical sampling and a 25k-token budget on every set (Table R5). On every set the novelty checkpoint's mean length is within 13% of the SFT's and 22–32% below vanilla's, so its gains are not bought with longer reasoning at test time.

Table R5. Mean response length in tokens.

checkpointOlymMATH-HardOlymMATH-EasyAIME25HMMT
puzzle SFT8,5968,4449,36713,032
vanilla GSPO14,23812,80413,19517,445
novelty bonus9,6449,4439,16213,608

(4) Robustness and broad post-training improvement. We summarize the robustness results along starting models, difficulty levels and sampling configurations. Table R6 covers the first two and adds the training seed: the paper's appendix has two runs each for vanilla and novelty, and we add one vanilla and two novelty runs trained since. The paper uses one sampling configuration throughout (T 0.6, top-p 0.95, 25k-token budget).

Comments on the paper

2 comments
Area_Chair_VMiuOpenReview ↗

AC Note: Rebuttal Received, Please Engage in Discussion

Dear authors and reviewers,

Thank you to the authors for their detailed rebuttal and for the time and care they put into addressing the reviewers' comments. I appreciate the additional clarifications and results.

Reviewers: please read the authors' responses carefully and reply to them before the end of the discussion period. Even a short acknowledgment is valuable. It is especially helpful to say whether your concerns have been addressed and, if not, what remains unresolved. If the rebuttal changes your assessment, please update your score and review accordingly, whether up or down. If your score stays the same, a brief explanation of why helps me and the senior area chairs.

Authors: I will follow the discussion closely and take your rebuttal fully into account when forming my recommendation.

Thank you all for your time and effort.

Best regards, Your Area Chair

AuthorsOpenReview ↗

General response: summary of the reviews and the revised PDF

We thank the reviewers for their careful reading and constructive critique that have helped us improve the quality of the paper.

We appreciate the positive views that the puzzle-to-mathematics setting gives "a relatively clear analysis on whether post-training can transfer across domains and why it does so" (Pp35, nZpT), that the matched vanilla and novelty branches isolate the reward modification (nZpT, TKsW), that the behavioural analysis gives more than accuracy and response length alone (nZpT, TKsW), and that the novelty bonus "turns the diagnosis into an implementable intervention" (nZpT, Pp35).

The individual responses below address each concern directly, with the full tables. We have uploaded a revised PDF. The main text stays within ten pages and the submitted results and figures are unchanged. The main changes:

  • A causal test of restored recovery: two inference-time interventions on fixed checkpoints, a backtrack cue that raises accuracy and a recovery ban that brings the novelty checkpoint to or below vanilla (nZpT W1, TKsW W2). New Section 5.1, Table 4, Appendix V.
  • Single-change ablations of the four design choices and five matched exploration baselines, none of which reaches the recipe (nZpT W2, TKsW W1). Section 5.1, Table 3, DIVER added to Section 2.
  • The pipeline on OLMo-3.1-32B, a single training run showing transfer to mathematics, and transfer of the OLMo3-7B checkpoints to LiveCodeBench problems released after the base model's data cutoff (Pp35 W1, nZpT W3). Appendix P, limitations in Section 6.
  • A code-trace SFT control, a working code model that is found to have poor math transfer performance in our evaluations, and the problems gained and lost across the pipeline (TKsW W2). Appendix Q.
  • Extended seeds and replicates, response lengths and a repetition check on the restored backtracks (nZpT W3). Appendices D and R.
  • Where the top-100 tokens fall and how all-correct groups are handled (nZpT Q3, Q4). Appendices S and T.
  • A definition of reasoning diversity in Section 1 and its measurement in Appendix U (TKsW Q3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.