- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies why reasoning models underperform on knowledge-intensive tasks. A factorized diagnostic separates isolated knowledge accessibility from in-chain knowledge use, and finds that knowledge points answered correctly in isolation are frequently omitted from CoT trajectories, with omission dominating over incorrect invocation; explicitly prompting the omitted knowledge improves accuracy. The authors term this the "knowledge dispatch bottleneck" and propose Knowledge Retrospect Learning (KRL): on-policy self-distillation in which the student generates from a problem-only prompt while the same model, conditioned on extracted knowledge points, retrospectively scores the student's prefixes and provides token-level distributional supervision. Experiments on Qwen3-1.7B/4B/8B over seven benchmarks report modest average gains over a GRPO baseline. Overall, I find the diagnostic contribution solid, but the method is a straightforward instance of on-policy privileged-context distillation, and the evidence that it works as claimed is thin.
Critical strengths
S1. The factorized diagnostic in Sec. 2 is a genuinely useful contribution. Separating isolated knowledge accessibility from in-chain knowledge use, and decomposing failures into omission vs. incorrect invocation, gives a much sharper picture of knowledge-intensive failures than final-answer accuracy. This matters for my assessment because the omission-dominant finding is likely to be of interest to the community regardless of whether the proposed method is convincing; it is the main reason I would not score this paper lower.
S2. The diagnostic is supported by sensible controls. The matched irrelevant/shuffled-knowledge prompting controls (Sec. 2.3) and the relevance-probing test make the "explicitization failure" conclusion reasonably credible rather than a prompt-length artifact. This is consequential because it establishes the phenomenon the whole paper builds on; without these controls the motivation for KRL would collapse.
Critical weaknesses
W1. The experiments do not isolate what the proposed method actually contributes. The main table compares only against Vanilla SFT and GRPO. There is no comparison to stronger RL recipes (e.g., DAPO-style variants), to on-policy distillation from a strong external teacher, or to simpler privileged-context variants (gold-solution or self-generated knowledge as teacher context). This is not a minor omission: Table 2 shows that plain off-policy distillation on R1-0528 trajectories already reaches 58.4 avg vs. KRL's 60.2, so a large share of the improvement over GRPO may come from distillation itself rather than the knowledge-retrospect design. Moreover, KRL uses extra annotated knowledge points plus strong-model CoT data while GRPO receives only outcome reward, so the two are not data- or compute-matched; and the Vanilla SFT row, which degrades far below Base (42.5 vs. 57.4 avg at 1.7B), inflates the headline "+17.7 points over SFT" claim. No variance or significance testing is reported anywhere. The consequence is that the paper's central empirical claim -- that knowledge-conditioned retrospective supervision is what helps -- is not established by the evidence presented.
W2. The measured benefit is small, shrinks with scale, and is never validated against the paper's own diagnostic. KRL beats GRPO by only +2.3 / +1.0 / +0.7 avg points at 1.7B / 4B / 8B; at 8B it is within noise of the untrained Base model (73.9 vs. 73.8) and regresses on AIME24 (75.2 vs. 76.7). Most importantly, the paper never re-measures the knowledge utilization gap Delta(k) after KRL training: the single most direct test of whether the method fixes the bottleneck it diagnoses is missing. Given that the method is essentially on-policy distillation (Agarwal et al., 2024; Gu et al., 2024) with a privileged-context teacher, the combination of limited demonstrated benefit and missing mechanistic validation leaves the contribution incremental.
Questions for authors
Q1. How does KRL compare to on-policy distillation from a strong external teacher (e.g., R1-0528) under matched compute and data? This seems essential for attributing the gains to the retrospect design rather than to distillation in general.
Q2. Can you provide a GRPO baseline with access to the same knowledge-augmented data (e.g., knowledge-conditioned prompts or an auxiliary reward), to separate the effect of the supervision content from the supervision source?
Q3. After KRL training, does Delta(k) from Sec. 2 shrink on held-out data, and does the omission rate of previously omitted knowledge points decrease? A positive answer here would substantially raise my score.
Q4. How sensitive is KRL to the quality and coverage of K(x)? What happens with noisy or partial knowledge annotations, or with knowledge points produced by the student model itself rather than a strong external LLM?
Q5 (minor clarification). Is the teacher in Eq. 6 the frozen initial model or an EMA of the current model? The notation (theta-bar) is never defined.