- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EMBER, a knowledge distillation framework that combines step-level credit assignment with online reinforcement learning. It estimates the benefit of individual teacher steps from student continuation outcomes, uses these estimates to weight teacher supervision, and reuses the rollouts for GRPO training. The authors evaluate the method on mathematical reasoning and instruction-following tasks, and provide theoretical analyses of credit estimation and teacher handoff.
Critical strengths
- Reasonable and practically motivated method.
The method evaluates teacher steps based on the student's current capabilities rather than imitating all steps equally. This is a reasonable approach that can adapt supervision as the student evolves.
- Theoretical analysis and experimental evaluation.
The paper analyzes credit estimation errors, policy drift, and teacher handoff. It also includes experiments across different models and tasks to examine the proposed framework.
Critical weaknesses
- Lack of fair baseline comparisons makes the performance gains difficult to attribute.
The main benchmark results rely largely on baselines from previous papers without matching training data or compute budgets. For example, EMBER uses the full OpenR1-Math-220k dataset, while the reported Prefix-RFT baseline uses approximately 46k examples. Their average performance differs by only 0.5 percentage points.
The main training study lacks a compute-matched SFT + GRPO baseline, making it difficult to determine whether the gains come from the proposed credit assignment or existing hybrid training strategies.
These issues weaken the evidence that EMBER outperforms existing methods.
- The necessity and effectiveness of step-level credit assignment are not sufficiently demonstrated.
In Table 4, step-aligned credit improves over shuffled, question-level, and uniform weighting by only 0.4–0.5 percentage points, without multiple training seeds or statistical significance analysis. This provides limited evidence that precise step-level assignment is important.
The accuracy of the online value model's credit estimates is not adequately validated, particularly its ability to distinguish positive and negative credit.
Adaptive prefix sampling and credit weighting are not fully disentangled, leaving their individual contributions unclear.
Given the additional rollout and value estimation costs, the practical benefit of this complexity remains uncertain.
Questions for authors
-
Can the authors provide fair comparisons with strong baselines under matched training data and compute budgets, particularly SFT + GRPO, to establish a meaningful advantage of EMBER?
-
Can the authors provide convincing evidence that step-level credit assignment offers a substantial and reliable improvement over simpler weighting strategies, given the small differences reported in Table 4?