- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submission studies puzzle-only SFT and RL, evaluates transfer to mathematics, and analyzes generated reasoning through a nine-class primitive classifier. It reports that vanilla GSPO reduces visible backtracking, while a frozen-reference novelty reward restores some backtracking and improves sampled mathematical success.
Critical strengths
- Useful controlled comparison. Vanilla and novelty GSPO start from the same SFT checkpoint with matched training settings. This meaningfully isolates the reward modification. Substantive behavioral analysis. The taxonomy, blind human annotation, per-problem bootstrap analysis, and worked examples provide more information than accuracy and response length alone.
Critical weaknesses
- The distinctive methodological contribution is not sufficiently established. The proposed reward combines several plausible design choices, but the comparison against vanilla GSPO does not clarify which choices are essential. Consequently, it remains unclear whether the gains arise from the proposed frozen-reference formulation or from the more general effect of rewarding uncommon successful trajectories.
- The source of cross-domain transfer remains insufficiently disentangled. Although excluding mathematics examples from post-training is a useful experimental choice, it does not distinguish puzzle-specific skill transfer from more general effects of reasoning-trace distillation or improved elicitation of existing knowledge. The experiments therefore establish the effectiveness of the recipe more clearly than they explain why puzzle training transfers to mathematics.
Questions for authors
-
What is the essential ingredient of the proposed novelty reward? How should we distinguish the benefit of using a frozen reference and top-k NLL from the more general benefit of encouraging exploration among successful trajectories? Which existing evidence best supports these particular design choices?
-
What does the study establish about the source of puzzle-to-math transfer? How do the authors distinguish the transfer of puzzle-specific problem-solving operations from broader effects of reasoning-trace distillation or improved elicitation of the base model’s existing knowledge?
-
How should “reasoning diversity” be interpreted in this paper? Does it refer primarily to behavioral variation within individual traces, or to different solution strategies across sampled traces? What evidence connects the reported primitive statistics to the intended notion of diversity?
6 comments
Response to Reviewer TKsW (2/3): source of the transfer: distillation, elicitation, what transfers (W2, Q2)
W2 and Q2. What is the source of puzzle-to-math transfer?
The transfer is carried by two domain-general reasoning operations that puzzle RL changes and that we measure on the mathematics traces themselves: the length of uninterrupted compute chains, which RL increases, and recovery after an error, which vanilla RL suppresses and the bonus restores. No mathematics enters post-training, so these changes elicit mathematics the base model already holds. Two inference-time interventions show that changing the restored recovery changes accuracy on mathematics: supplying a one-sentence backtrack cue at an error raises the vanilla checkpoint's accuracy by 5.6 points, and removing the recovery spans from the novelty checkpoint's own traces lowers its pass@8 by 5.2 points and brings it to the vanilla level (Tables T3 and T4). The three sources you name are each present in our pipeline at a different stage, and we take them in turn.
Is the gain reasoning-trace distillation? Partly yes. The SFT stage is distillation from puzzle reasoning traces, and Section 4.1 shows that it installs the teacher's reasoning primitive profile in the model, on puzzles and on mathematics alike. The RL stages add no traces: vanilla GSPO and the bonus train on puzzle rewards only, and they move OlymMATH-Hard pass@32 from 23 (SFT) to 29 and 36. The gain beyond SFT is therefore not distillation. We do not claim that puzzle traces are the only traces that transfer, and the paper's claims are about what RL then does and how.
Is the gain elicitation of existing knowledge? Yes, and we can measure how much by looking at newly unlocked problems. Table T6 tracks which OlymMATH-Hard problems each checkpoint solves at least once in 32 samples, and how the set changes at each stage of the pipeline.
Table T6. OlymMATH-Hard, 100 problems, 32 samples per checkpoint. Problems gained and lost at each stage, relative to the previous checkpoint. At 32 samples, pass@32 equals the share of problems solved at least once.
| transition | pass@32 | problems gained | problems lost | net |
|---|---|---|---|---|
| base → puzzle SFT | 15 → 23 | +13 | −5 | +8 |
| puzzle SFT → vanilla GSPO | 23 → 29 | +16 | −10 | +6 |
| puzzle SFT → novelty GSPO | 23 → 36 | +16 | −3 | +13 |
| vanilla GSPO → novelty GSPO | 29 → 36 | +16 | −9 | +7 |
Every stage reaches problems the previous one did not, without any mathematics in training, so the gain is elicitation of what the base model holds. The bonus extends the reach of the model while preserving what SFT could already do.
What transfers? A change in domain-general reasoning operations, not puzzle-specific ones. The paper measures the puzzle and mathematics traces at every stage with the same primitive classifier, so the transferred change is described at the level of reasoning operations. SFT installs the teacher's primitive profile in both puzzles and mathematics. Vanilla RL lengthens the uninterrupted compute chains (chain fraction 0.93 → 0.98 on OlymMATH-Hard) and suppresses backtracking (0.41 → 0.08 episodes per trace). The novelty bonus restores backtracking to the SFT level (0.41) and the chain fraction with it (0.93), and accuracy improves with these changes.
Interventions. To test whether recovery drives the novelty checkpoint's gain, we ran two inference-time interventions that hold the model fixed and change only recovery. If recovery is linked to the gain, supplying it should raise accuracy and removing it should lower it.
Supplying recovery. We cut failed OlymMATH-Hard traces of both checkpoints at the start of the sentence that states the final, wrong answer, so the error is in the prefix, and let both checkpoints continue every prefix with the same 40k-token budget. At the cut we inserted either a one-sentence backtrack cue ("Wait — this contradicts an earlier assumption…") or a length-matched control sentence about notation, unrelated to recovery.
Table T3. Continuations from error prefixes: unprompted backtracking and recovery under the control sentence, and recovery under the cue.
| prefixes | continuer | n | control: backtracked | control: solved | cue: solved |
|---|---|---|---|---|---|
| vanilla checkpoint's own errors | vanilla | 1,200 | 1% | 0.3% | 5.9% |
| vanilla checkpoint's own errors | novelty | 1,200 | 11% | 2.0% | 10.1% |
| novelty checkpoint's errors | vanilla | 1,168 | 2% | 0.9% | 7.1% |
| novelty checkpoint's errors | novelty | 1,168 | 10% | 1.4% | 9.3% |
Response to Reviewer TKsW (3/3): interventions on recovery (W2, Q2); definition of reasoning diversity (Q3); summary of changes
In Table T3, backtracked is the share of continuations in which the classifier detects a backtrack, and solved the share whose final answer is correct. The cue raises the vanilla checkpoint's accuracy by 5.6 points over the control. After vanilla RL, backtracking is suppressed in its initiation (1% of continuations against 11% for the novelty checkpoint on the same prefixes) but, once initiated through a recovery cue, the vanilla checkpoint recovers, though not as effectively as the novelty checkpoint on the same prefixes (5.9% vs 10.1%).
Removing recovery. We removed recovery from whole traces of the novelty checkpoint (100 OlymMATH-Hard problems, 32 traces per condition), so the checkpoint is the same in both conditions and only recovery differs. An LLM judge flags recovery spans as they close. In the ban condition a flagged span is deleted and regenerated, and the control condition is the same generation loop with the censor off. A manipulation check confirms that the censor removes almost all recovery spans (Table T4).
Table T4. OlymMATH-Hard accuracy of the novelty checkpoint with recovery spans banned, against its loop control and the vanilla checkpoint.
| condition | pass@1 | pass@8 | pass@32 |
|---|---|---|---|
| novelty, unmodified | 4.0 | 18.0 | 36.3 |
| novelty, loop control | 3.8 | 16.3 | 31.0 |
| novelty, recovery banned | 2.4 | 11.1 | 24.0 |
| vanilla, unmodified | 3.2 | 13.8 | 27.9 |
Together, the two interventions support the reading that the restored recovery mediates the bonus's gain on mathematics, and that this recovery is what the puzzle training transfers.
Revision: We will add an appendix with the per-problem gained and lost sets of Table T6. Section 5 will address the three sources you name, distillation, elicitation and the transferred primitives, in turn.
Q3. How should "reasoning diversity" be interpreted?
By reasoning diversity we mean how differently the model reasons across traces sampled for the same problem. Each trace is a sequence of the primitives of Section 4 (plan, hypothesise, check, backtrack and the others), and diversity is how much these sequences differ from one sample to the next. It is not variation within a single trace, and it is not variation in wording. We will state this definition in Section 1, where the term is first used. We measure differences using edit distance on the sequence of primitives, which we detail below.
Measurement. On math rollouts of the SFT, vanilla and novelty checkpoints (OlymMATH-Hard, 100 problems, 32 traces per problem), we reduce each trace to its primitive sequence, merging adjacent repeats, and compute the mean pairwise edit distance among the 32 sequences of a problem, normalised by sequence length (0 means all 32 traces follow the same sequence).
Table T5. Across-sample reasoning diversity on OlymMATH-Hard: mean pairwise normalised edit distance between the primitive sequences of the 32 traces of a problem, averaged over 100 problems.
| checkpoint | across-sample distance |
|---|---|
| SFT | 0.630 |
| vanilla GSPO | 0.608 |
| novelty GSPO | 0.632 |
Reasoning diversity goes down under vanilla RL and comes back under the bonus: the vanilla checkpoint's traces follow more similar primitive sequences for the same problem than the SFT checkpoint's, and the novelty checkpoint's traces return to the SFT level. This is the across-sample counterpart of the per-trace statistics of Table 6 of the paper, which are averaged over all traces of a dataset and so describe what a typical trace does, while the across-sample distance measures how much the traces differ from each other. It shows in accuracy as a novelty − vanilla pass@k gap that grows with k (W2 and Q2 above).
Revision: We will state the definition in Section 1 and add the across-sample measurement (Table T5) to the appendix.
Summary of changes
- We will add the single-change ablations of the reward and the matched exploration baselines (Tables T1, T2) in a new appendix.
- We will add the per-problem gained and lost sets across the pipeline (W2, Table T6) in a new appendix.
- We will state the definition of reasoning diversity in Section 1 and add its across-sample measurement to the appendix.
We are happy to clarify any remaining point.
Response to Reviewer TKsW (1/3): design ablations and matched exploration baselines (W1, Q1)
Thank you for the constructive review. In our response, we answer W1 together with Q1, W2 together with Q2, and Q3 on its own. To address W1 and Q1, we provide additional ablation experiments and show that each component of our proposed design is essential. To address W2 and Q2, we identify the source of puzzle-to-math transfer as elicitation of mathematical knowledge the base model already holds, through changes in reasoning behaviour at the primitive level. For Q3, we clarify that by reasoning diversity, we mean how differently the model reasons across traces sampled for the same problem, at the primitive vocabulary level.
Additional details for each raised weakness/question are addressed below.
W1 and Q1. Which design choices are essential, and is the gain specific to the frozen-reference top-k formulation?
Yes, in large part: the gain depends on the frozen reference, the normalisation over correct rollouts and the bonus in all-correct groups, and rewarding uncommon successful trajectories in general produces a smaller gain. Each of the four design choices contributes, and no other exploration method we ran matches the gain. The bonus has four design choices: the statistic (mean reference NLL of the 100 highest-NLL tokens of a response), the reference the NLL is computed under (the frozen SFT model), the set the score is normalised over (the correct rollouts of a group), and the treatment of all-correct groups (they keep the bonus, which is then their only source of advantage). We ran one matched run per choice (same recipe, initialisation, data, α = 0.1 and software stack), and evaluated the best checkpoint of each on OlymMATH-Hard (Table T1).
Table T1. OlymMATH-Hard, best checkpoint of each run.
Each choice matters:
Design or general effect. The gain is specific to the design. We trained five other exploration approaches on the same recipe, one per run, and evaluated the best checkpoint of each on OlymMATH-Hard (Table T2).
Table T2. Matched OLMo3-7B runs with one exploration approach each, best checkpoint, OlymMATH-Hard.
Table T2 shows that not every exploration approach transfers effectively: the entropy approaches and DIVER land at or below the vanilla level. Among existing alternate approaches, unlikeliness is the closest form of rewarding uncommon successful trajectories and it does transfer, but it differs from our reward in what the reference is (unlikeliness uses the current policy rather than the frozen SFT model our approach uses), the statistic (the rank of whole-sequence likelihood rather than top-k token NLL) and the way it enters the reward (a multiplicative rank discount rather than an additive z-score) at once, so its result speaks to the general effect you name and cannot separate these differences. The ablation runs in Table T1 answer which components of our reward matter. The novelty bonus is above it at pass@8 and pass@32 and level with it at pass@1.
Revision: In the revised version we will add an appendix with Tables T1 and T2, and Section 5 will state the four design choices and point to the ablation results.