- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces mathematical primitives as compact representations of essential solution ideas, constructs PRIM to evaluate four reasoning capabilities, and proposes ABSORB to transfer primitive-conditioned teacher guidance through a capped reverse-KL objective.
Critical strengths
- PRIM provides an informative perspective beyond answer accuracy by separating independent primitive discovery, solution generation, retrospective digestion, and primitive-guided execution. The same-teacher comparison between primitives and procedural plans is particularly useful: it supports the value of structural guidance beyond simply supplying an external solution outline.
- ABSORB turns primitive annotations into useful training supervision without requiring primitives at inference time. Improvements across three model scales, together with controlled comparisons of teacher context, transfer objective, and rollout target, provide substantive support for this training design.
Critical weaknesses
- How does the failure decomposition establish Discovery as the dominant bottleneck rather than a frequently co-occurring deficit? The reported 83.6% of failures with unsuccessful Discovery includes 49.1% classified as D−E−, which remain unsolved even when the correct primitive is supplied; only 34.6% are D−E+. Moreover, Table 9 shows that 75% of GPT-5.4’s Generation failures already have successful Discovery. These differences raise questions about the scope and causal interpretation of the headline conclusion.
- Does the capped objective preserve positive teacher guidance after differentiation? For the stated loss under ordinary softmax differentiation, consider student probabilities P=(0.5,0.5), teacher probabilities Q=(0.1,0.9), and τ=0.06. The first contribution is capped, and the gradient with respect to the second student logit is approximately +0.103. Gradient descent therefore decreases the probability of the teacher-preferred second token. How is this behavior reconciled with the claim that positive teacher guidance is retained in full? Preserving negative loss contributions does not establish the claimed direction of the probability update.
Questions for authors
- What does unsuccessful standalone Discovery diagnose when the same model identifies the relevant primitive inside a correct Generation trajectory, as observed for Qwen? How should primitive articulation before exploration be distinguished from structural discovery during problem solving?
- Is a primitive one dominant organizing insight or the complete set of essential conceptual insights? Figures 7 and 10 require every essential idea, whereas Figure 9 excludes secondary conceptual steps from scoring. Where does a missing second insight become an Execution failure rather than a Discovery failure?
- Does the implementation differentiate the capped objective exactly as written, or use a modified gradient convention? What optimization-level behavior is intended by preserving positive guidance?
- Does the OPSD baseline in Table 4 use the standard uncapped objective or the capped ABSORB objective? Is it the same solution-conditioned variant reported in Table 13, where the ABSORB objective is held fixed?
- What distinguishes primitive-specific internalization from broader preservation or improved utilization of existing reasoning? Discovery changes little after training, while Table 11 also shows substantial repairs in D+E+ cases where the primitive was already discoverable.
6 comments
R1: Thanks for the helpful feedback! (part 2/3)
[Q1] What does unsuccessful standalone Discovery diagnose when the same model identifies the relevant primitive inside a correct Generation trajectory, as observed for Qwen? How should primitive articulation before exploration be distinguished from structural discovery during problem solving?
Thank you for the question. Discovery asks the model to identify and explain the primitive from the problem alone (Section 2.2). A model may fail this task yet identify the same structure while working through a solution. In Qwen3.6-27B's cases (as reported in Finding 1), the standalone Discovery response often misses the decisive idea, which can emerge after exploration or appear at the outset of Generation. The model may also revise or abandon a correct idea during rethinking, so its appearance alone does not show that it consistently guided the solution.
Furthermore, the low Discovery accuracy cannot be explained solely by a general inability to articulate primitives. Given a reference solution, Qwen models achieve 70.88%–93.41% on Digestion, compared with 6.59%–28.57% on Discovery, despite using the same primitive output format and scoring protocol (Table 1).
We have clarified the distinction between standalone Discovery and discovery during Generation in Section 2.2 and Finding 1 of Section 3.2.
[Q2] Is a primitive one dominant organizing insight or the complete set of essential conceptual insights? Figures 7 and 10 require every essential idea, whereas Figure 9 excludes secondary conceptual steps from scoring. Where does a missing second insight become an Execution failure rather than a Discovery failure?
Thank you for the question. Figures 7 and 10 request complete conceptual guidance for annotation and teacher generation. Figure 9 evaluates whether a prediction captures the core structure and its mechanism (Definition 1), accepting equivalent formulations without requiring every insight in the reference annotation. Gold annotations are also human-reviewed against the problem and reference solution (Section 2.3 and Appendix A.1). Discovery evaluates the core structure and its explanation, without requiring every secondary insight needed to complete the solution. Execution separately evaluates final-answer correctness with the human-verified gold primitive, rather than the model's Discovery output.
We have clarified the distinction between annotation completeness and scoring the core primitive in Appendix A.3.
[Q4] Does the OPSD baseline in Table 4 use the standard uncapped objective or the capped ABSORB objective? Is it the same solution-conditioned variant reported in Table 13, where the ABSORB objective is held fixed?
Thank you for the question. For a fair baseline comparison, we also apply the capped reverse-KL objective to OPSD in Table 4, while retaining its reference-solution-conditioned on-policy distillation. The "Solution privilege" row in Table 13 reports the same Qwen3.5-9B OPSD run. Table 14 separately examines the transfer mechanism with primitive conditioning held fixed.
R1: Thanks for the helpful feedback! (part 3/3)
[Q5] What distinguishes primitive-specific internalization from broader preservation or improved utilization of existing reasoning? Discovery changes little after training, while Table 11 also shows substantial repairs in D+E+ cases where the primitive was already discoverable.
Thank you for the valuable question. Internalization refers to transferring primitive-guided reasoning into the student's solution generation (Section 4.4). We deployed Absorb to train the student to generate solutions with supervision from a primitive-conditioned teacher, rather than to identify and explain primitives as a standalone task. Generation can therefore improve without comparable gains in standalone Discovery.
Under the same capped reverse-KL objective, Qwen3.5-9B achieves 43.96% Generation accuracy when the teacher receives primitives, compared with 33.52% when it receives reference solutions (Table 13). This comparison supports the benefit of primitive supervision.
To examine the changes in repaired Prim cases, three experts with graduate-level mathematical training compared each base-model solution with its Absorb counterpart across Qwen3.5-4B, 9B, and 27B. The observed corrections fall into five categories:
| Failure of the base model | Description | Share of repaired cases |
|---|---|---|
| Abandoned correct result | The base model derives the correct result (or the decisive intermediate result) but then overrides it and commits to a different answer; Absorb retains it. | 35.9% |
| Incomplete derivation | The base model sets up the correct approach but does not carry out the decisive step, and falls back on a guess, a numerical extrapolation, or a trivial bound; Absorb completes the step. | 26.6% |
| Unjustified assumption | The base model relies on a theorem, bound, or structural assumption that does not hold for the problem; Absorb uses the correct structure. | 25.0% |
| Calculation error | The base model makes an error in a specific evaluation step of an otherwise viable solution; Absorb performs this step correctly. | 10.9% |
| Misreading of the problem | The base model misreads a condition given in the problem. | 1.6% |
The repairs show that Absorb corrects unjustified assumptions (25.0%) and helps models retain correct results or complete viable derivations (62.5%). The gains therefore involve both correcting structural mistakes and making better use of reasoning already present in the base model. For the cases in Table 11, the model already identifies the primitive in Discovery and solves the problem when given the primitive in Execution, but fails in Generation. Repairs in this group are consistent with more effective use of existing reasoning during Generation. We have clarified this interpretation in Appendix B.3.6.
R1: Thanks for the helpful feedback! (part 1/3)
Dear Reviewer,
Thank you for your insightful reviews. We greatly appreciate your insightful comments and suggestions. Below is our point-by-point response, with further experimental details. We hope these clarifications address all your concerns.
Thank you for your comment. Finding 3 identifies Discovery as the dominant bottleneck in the pooled evaluation based on both the prevalence of Discovery failures and the substantial gains from primitive guidance. Discovery is unsuccessful in 83.6% of Generation failures, and supplying ground truth primitives improves final-answer accuracy by 17.58–29.67 percentage points across all 12 models (Table 1). The D−E+ cases provide evidence beyond co-occurrence: for 34.6% of all Generation failures, the same model solves the problem once it receives the primitive it failed to discover independently. These discovery-limited failures are 4.3 times as common as execution-limited failures (D+E−, 8.1%; Table 9).
The D−E− cases (49.1% of all Generation failures) show that primitive guidance alone is insufficient for a substantial group of failures. Table 9 explicitly separates these capability-limited cases from discovery-limited failures, and our aggregate conclusion does not assume that correcting Discovery is sufficient to resolve every failure.
For GPT-5.4, its Discovery accuracy (82.42%) exceeds Generation accuracy (67.03%), indicating that it can often identify the primitive but still fails to solve the problem. We would therefore expect its failures to arise mainly during the process of problem solving, consistent with the observation that 75% of its Generation failures occur despite successful Discovery. Our analysis in Finding 1 supports this interpretation, identifying technical knowledge gaps and procedural errors in these D+G− cases.
Thank you for the helpful comment. Absorb is designed to selectively transfer primitive-guided reasoning, motivated by the post-training regressions discussed in Section 3.3. The teacher has access to privileged information while the student does not, so strong local disagreement can reflect differences in available context as well as errors in the student's reasoning. We therefore use capped reverse KL to learn from the privileged teacher while limiting strong corrections to student-preferred choices.
Our implementation utilized the capped objective in Section 4.3 exactly as written, with no modified gradient convention. The mentioned example in the review is consistent with our selective-transfer motivation, and we aim to learn useful primitive-guided reasoning while limiting strong teacher overrides without requiring every teacher-favored token to increase in probability at each update.
To examine how selective transfer operates in practice, we analyze capping activity using 570k positions from stored on-policy rollouts of Qwen3.5-9B:
Most positions continue to contribute standard reverse-KL gradients. Where capping applies, it also limits penalties on the teacher's own top token; capping other student-favored tokens occurs at 6.0–11.7% of positions. We have included the above analysis of capping activity during post-training in Appendix G.4.
Furthermore, Table 14 (as summarized below) shows the benefit of capping on Qwen3.5-9B with primitive conditioning and the solution-generation rollout target held fixed. Absorb outperforms the base model, uncapped reverse KL, and divergence control on Generation.
By “transferred in full,” we meant retaining the reverse-KL loss contributions associated with positive teacher guidance without capping them, while limiting large penalties on student-preferred choices. We have further clarified this point in Section 4.2 to make the intended meaning clearer.