- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PACE, a segment-level calibration method for on-policy distillation (OPD). The authors argue that token-level teacher–student discrepancies are often affected by stylistic differences and other capability-irrelevant factors. To address this issue, PACE adaptively segments student-generated trajectories based on changes in teacher negative log-likelihood (NLL) and uses the initial segments as a trajectory-specific anchor to calibrate subsequent discrepancies. Experiments on tool-use and coding benchmarks demonstrate consistent improvements over existing OPD baselines.
Critical strengths
The paper addresses an interesting and important problem: distinguishing meaningful capability gaps from noisy teacher–student discrepancies in OPD.
PACE introduces an intuitive approach that combines adaptive segmentation with trajectory-specific calibration, without requiring additional reward models or annotations during training.
Experiments across multiple teacher–student configurations and six benchmarks demonstrate consistent improvements over standard OPD.
Critical weaknesses
The assumption that the initial segments primarily capture capability-irrelevant discrepancies is not sufficiently justified. Different tasks may enter substantive reasoning at different positions, and early segments may already contain important capability-related information.
Using a fixed number of three segments as the prefix anchor seems somewhat heuristic. Although the authors provide an ablation study, it remains unclear whether this choice generalizes across different tasks, response lengths, and reasoning patterns. An adaptive anchor length may be more appropriate.
Changes in teacher NLL do not necessarily correspond to meaningful reasoning or semantic boundaries. While the diagnostic analyses provide supporting evidence, it remains unclear whether the resulting segments consistently capture capability-relevant transitions rather than lexical or formatting variations.
The interpretation of the calibration objective could be clearer. The squared log-ratio penalizes deviations in both directions, including cases where the relative mismatch decreases. Moreover, the sigmoid gate assigns non-negligible weights even near the prefix baseline. The authors should clarify how these design choices specifically target capability degradation.
Questions for authors
1.Why is the prefix anchor constructed from a fixed number of three segments? Would adaptively determining the anchor length based on trajectory characteristics provide more reliable calibration?
2.How can the authors distinguish capability-related changes from stylistic or lexical variations when identifying segment boundaries using teacher NLL?
3.Since the calibration loss penalizes both increases and decreases in the relative mismatch ratio, could the authors clarify why symmetric regularization is preferred over a one-sided penalty targeting increased discrepancy?
4 comments
Q1. Fixed vs. Adaptive Prefix Length
We thank the reviewer for this suggestion.
1. The anchor fixes the number of segments, not the number of tokens.
We clarify that PACE dynamically partitions each trajectory based on teacher-NLL changes, structural boundaries, and length constraints. Although the anchor always uses the first $k=3$ segments, the token span it covers varies across trajectories, providing implicit adaptation to each trajectory's characteristics.
2. Adaptive anchor selection is promising but sensitive to the annotator model.
Adaptive anchor selection is promising but requires per-trajectory annotation. When identifying the onset of substantive reasoning, the trained agent selects a mean anchor length of 1.538 segments, versus 3.938 for GPT-5.4, showing substantial judge dependence. Fixed falls between the two judge estimates and avoids per-trajectory model inference. It also balances baseline stability against content contamination: is more sensitive to local fluctuations, whereas risks including substantive reasoning in the baseline. In our experiments, outperforms both fixed alternatives (Table 2 in paper) and the agent-judged adaptive variant. Reliable, low-cost adaptive selection remains a direction for future work.
| Teacher → Student | Method | ACEBench | BFCL | BCB | MBPP |
|---|---|---|---|---|---|
| 9B → 2B | PACE () | 66.80 | 71.30 | 0.220 | 43.60 |
| 9B → 2B | Adaptive anchor | 65.63 | 68.40 | 0.205 | 41.60 |
Q2. Distinguishing Capability Changes from Surface Variations
1. The objective is to reduce the influence of irrelevant variation. Segment-level averaging dampens isolated token-level fluctuations, while prefix calibration discounts persistent teacher–student discrepancies arising from stylistic preferences and other capability-irrelevant factors. These steps do not fully separate capability-related changes from irrelevant variation, and residual noise may remain as noted in the limitations section. Our analysis and experiments support the effectiveness of prefix calibration.
2. The boundary rule supports stable local aggregation. Teacher-NLL boundaries do not distinguish capability changes from stylistic or lexical variation. They define local units for aggregation: sharp NLL changes separate regions with different NLL levels, while a minimum-length constraint limits over-segmentation (Appendix D). A boundary may therefore reflect a stylistic change or a capability change. PACE addresses this uncertainty downstream through segment-level averaging and prefix-relative calibration, which reduce the influence of token-level fluctuations and persistent mismatch without claiming to classify their causes.
3. Ablation confirms the independent value of prefix calibration. Replacing with uncalibrated as the auxiliary loss input, with all other hyperparameters unchanged, consistently degrades performance:
| Teacher → Student | Method | ACEBench | BFCL | BCB | MBPP |
|---|---|---|---|---|---|
| 9B → 2B | PACE () | 66.80 | 71.30 | 0.220 | 43.60 |
| 9B → 2B | Uncalibrated () | 62.62 | 66.40 | 0.190 | 41.80 |
This confirms that the calibration step provides value independent of segmentation, even without explicit separation of capability-related and stylistic variation.
Q3. Symmetric Regularization vs. One-Sided Penalties
1. The calibration term penalizes deviation from the prefix baseline.
penalizes deviations of each later segment’s student-to-teacher NLL ratio from its trajectory’s prefix baseline (Eqs. 12 and 14). Since is minimized at , deviations in either direction are penalized; neither necessarily indicates an increase in absolute mismatch.
2. Downward deviations () are not reliably indicative of improvement.
means that falls below the prefix baseline; it does not prove student improvement, since also reflects teacher NLL, style, and confidence. Exempting such segments would wrongly treat every decrease as beneficial.
3. Statistical evidence shows that both directions co-occur with failure.
We sample 300 training trajectories and find that 74.85% of high-deviation segments, defined by , are identified as erroneous by a strong LLM judge (GPT-6). Among these high-deviation segments, 41.91% have and 58.09% have . Thus, errors occur with deviations in both directions, providing empirical support for symmetric penalization.
4. The one-sided ablation underperforms PACE.
| Teacher → Student | Method | ACEBench | BFCL | BCB | MBPP |
|---|---|---|---|---|---|
| 9B → 2B | PACE (symmetric) | 66.80 | 71.30 | 0.220 | 43.60 |
| 9B → 2B | One-sided ( only) | 64.80 | 67.84 | 0.205 | 42.80 |
The consistent gap confirms that retaining the downward calibration signal is empirically beneficial.
I thank the authors for their response. I will keep my score unchanged. Good luck.
Thank you for the careful review, encouraging comments, and constructive suggestions. These insights help us improve the paper.
We thank the reviewer for highlighting the significance of the problem, the intuitive combination of adaptive segmentation and trajectory-specific calibration, and the consistent empirical improvements without additional reward models or training annotations. Below, we clarify our design choices and address your concerns.
Appendix C.5 shows that these openings contain substantial non-substantive content, motivating the prefix as a trajectory-specific mismatch reference without assuming it is entirely capability-irrelevant.
2. The reasoning entry point is difficult to identify precisely, and $k=3$ segments approximates it well.
As noted in our response to Q1, the onset of substantive reasoning is difficult to identify and varies across judges. In most trajectories, the first three segments end at or before the onset identified by a stronger judge. PACE fixes the number of segments, not tokens, so the prefix length adapts to each trajectory.
3. The segmentation mechanism itself detects the reasoning entry point.
We analysed 100 randomly sampled trajectories. In 87, a sharp change in teacher NLL triggered a segment boundary near the onset of reasoning, at an average position of 4.26 segments. This supports using the first three segments as an opening reference, while acknowledging that some may contain substantive reasoning.
Together, the content analysis, anchor-length diagnostics, and observed boundary alignment jointly support the assumption that the prefix anchor predominantly reflects capability-irrelevant intrinsic mismatch.
Please see our response to Q1, which discusses the choice of three segments and compares it with adaptive anchor selection.
We thank the reviewer for this helpful suggestion. We have revised the manuscript to describe the segments as “coherent local aggregation units,” clarifying that the criterion aims to avoid averaging across distinct teacher predictive regimes. This change does not affect the method or conclusions.
1. Segment boundaries are defined as aggregation units.
PACE uses sharp teacher-NLL changes as candidate boundaries for averaging token-level fluctuations, not as markers of semantic or capability transitions. As Appendix D.2 shows, placing a boundary near a sharp NLL jump reduces the segment-mean approximation error.
2. Empirical evidence supports segment coherence.
We evaluate 3,127 segments from 50 trajectories using an LLM judge, which assesses 83% as complete, coherent linguistic or code units. We acknowledge that segment-level coherence does not imply that every boundary marks a capability-relevant transition, but this is consistent with PACE's intent: to provide interpretable local aggregation units.
3. Teacher-NLL reuses existing distillation quantities at negligible cost.
The boundary rule reuses teacher NLLs already computed for the base OPD objective, requiring no additional model calls or semantic classifiers during training. Minimum-length constraints limit over-segmentation caused by isolated fluctuations, while structural boundaries and maximum-length constraints further stabilize the partition.
Together, the coherence analysis and the diagnostics in Appendix C support the practical value of segmentation combined with prefix-anchored calibration, without requiring every boundary to carry independent semantic meaning.
1. The squared log-ratio penalizes deviation from the prefix baseline in both directions. PACE compares each later segment’s mismatch ratio qi with the trajectory’s prefix baseline bτ through ρi=qi/bτ. The term (logρi)2 penalizes departures from that baseline in either direction. This is by design: both an unexpected increase and an unexpected decrease relative to the prefix baseline represent deviations from the trajectory's established mismatch pattern that the calibration term regularizes.
2. Insensitivity near the baseline arises primarily from the squared log-ratio, not the gate. The full per-segment contribution is gi(logρi)2. When ρi≈1, the gate gi remains non-negligible, but logρi≈0, so the squared term rapidly suppresses the direct penalty. For example, at ρi=1.02, (logρi)2≈0.00039. Thus, the insensitivity to small fluctuations near the baseline comes from the squared log-ratio's measure of deviation magnitude, not from the gate approaching zero.