acceptodds
Under review as a conference paper at ICLR 2027

PACE: Prefix-Anchored Calibration with Adaptive Segmentation for On-Policy Distillation

Abstract

On-policy distillation (OPD) offers a practical way to align smaller student models with stronger teachers on student-generated trajectories. However, token-level teacher-student discrepancies are often noisy and poorly calibrated, as they are susceptible to capability-irrelevant noise, such as lexical variation, formatting artifacts and teacher preferences rather than reflecting differences in reasoning ability. Consequently, standard OPD struggles to distinguish meaningful teacher-student divergence from stable response-level mismatch. To address this problem, we propose PACE (Prefix-Anchored Calibration with adaptive sEgmentation), a segment-level calibration objective for on-policy distillation. PACE first adaptively partitions each student generated response into locally coherent segments using change points in teacher's negative log-likelihood (NLL), together with structural boundaries. PACE computes a student-to-teacher NLL ratio from segment-averaged losses, yielding a more stable local measure of mismatch than token-level discrepancies. The mean ratio over the initial segments serves as a trajectory-specific anchor against which subsequent segment ratios are calibrated. This relative formulation helps reduce the influence of persistent teacher–student discrepancies unrelated to reasoning ability, allowing the learning objective to focus more directly on segment-specific capability gaps. It complements standard OPD without requiring additional reward models or human annotations, providing a lightweight segment-level calibration signal for LLM agents. Experiments across tool-use and coding benchmarks demonstrate consistent improvements over OPD baselines across different teacher-student model scales, supporting the effectiveness and generality of PACE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1mcnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACE, a segment-level calibration method for on-policy distillation (OPD). The authors argue that token-level teacher–student discrepancies are often affected by stylistic differences and other capability-irrelevant factors. To address this issue, PACE adaptively segments student-generated trajectories based on changes in teacher negative log-likelihood (NLL) and uses the initial segments as a trajectory-specific anchor to calibrate subsequent discrepancies. Experiments on tool-use and coding benchmarks demonstrate consistent improvements over existing OPD baselines.

Critical strengths

The paper addresses an interesting and important problem: distinguishing meaningful capability gaps from noisy teacher–student discrepancies in OPD.

PACE introduces an intuitive approach that combines adaptive segmentation with trajectory-specific calibration, without requiring additional reward models or annotations during training.

Experiments across multiple teacher–student configurations and six benchmarks demonstrate consistent improvements over standard OPD.

Critical weaknesses

The assumption that the initial segments primarily capture capability-irrelevant discrepancies is not sufficiently justified. Different tasks may enter substantive reasoning at different positions, and early segments may already contain important capability-related information.

Using a fixed number of three segments as the prefix anchor seems somewhat heuristic. Although the authors provide an ablation study, it remains unclear whether this choice generalizes across different tasks, response lengths, and reasoning patterns. An adaptive anchor length may be more appropriate.

Changes in teacher NLL do not necessarily correspond to meaningful reasoning or semantic boundaries. While the diagnostic analyses provide supporting evidence, it remains unclear whether the resulting segments consistently capture capability-relevant transitions rather than lexical or formatting variations.

The interpretation of the calibration objective could be clearer. The squared log-ratio penalizes deviations in both directions, including cases where the relative mismatch decreases. Moreover, the sigmoid gate assigns non-negligible weights even near the prefix baseline. The authors should clarify how these design choices specifically target capability degradation.

Questions for authors

1.Why is the prefix anchor constructed from a fixed number of three segments? Would adaptively determining the anchor length based on trajectory characteristics provide more reliable calibration?

2.How can the authors distinguish capability-related changes from stylistic or lexical variations when identifying segment boundaries using teacher NLL?

3.Since the calibration loss penalizes both increases and decreases in the relative mismatch ratio, could the authors clarify why symmetric regularization is preferred over a one-sided penalty targeting increased discrepancy?

4 comments

We thank the reviewer for highlighting the significance of the problem, the intuitive combination of adaptive segmentation and trajectory-specific calibration, and the consistent empirical improvements without additional reward models or training annotations. Below, we clarify our design choices and address your concerns.

W1. Capability Relevance of Early Segments

  1. In both tool-use and coding tasks, response openings often contain problem restatements, intent statements, formatting preambles, and generic planning before substantive reasoning begins.

Appendix C.5 shows that these openings contain substantial non-substantive content, motivating the prefix as a trajectory-specific mismatch reference without assuming it is entirely capability-irrelevant.

2. The reasoning entry point is difficult to identify precisely, and $k=3$ segments approximates it well.

As noted in our response to Q1, the onset of substantive reasoning is difficult to identify and varies across judges. In most trajectories, the first three segments end at or before the onset identified by a stronger judge. PACE fixes the number of segments, not tokens, so the prefix length adapts to each trajectory.

3. The segmentation mechanism itself detects the reasoning entry point.

We analysed 100 randomly sampled trajectories. In 87, a sharp change in teacher NLL triggered a segment boundary near the onset of reasoning, at an average position of 4.26 segments. This supports using the first three segments as an opening reference, while acknowledging that some may contain substantive reasoning.

Together, the content analysis, anchor-length diagnostics, and observed boundary alignment jointly support the assumption that the prefix anchor predominantly reflects capability-irrelevant intrinsic mismatch.

W2. Choice and Generalizability of the Prefix Length

Please see our response to Q1, which discusses the choice of three segments and compares it with adaptive anchor selection.

W3. Semantic Validity of Teacher-NLL Boundaries

We thank the reviewer for this helpful suggestion. We have revised the manuscript to describe the segments as “coherent local aggregation units,” clarifying that the criterion aims to avoid averaging across distinct teacher predictive regimes. This change does not affect the method or conclusions.

1. Segment boundaries are defined as aggregation units.

PACE uses sharp teacher-NLL changes as candidate boundaries for averaging token-level fluctuations, not as markers of semantic or capability transitions. As Appendix D.2 shows, placing a boundary near a sharp NLL jump reduces the segment-mean approximation error.

2. Empirical evidence supports segment coherence.

We evaluate 3,127 segments from 50 trajectories using an LLM judge, which assesses ​83% as complete, coherent linguistic or code units​. We acknowledge that segment-level coherence does not imply that every boundary marks a capability-relevant transition, but this is consistent with PACE's intent: to provide interpretable local aggregation units.

3. Teacher-NLL reuses existing distillation quantities at negligible cost.

The boundary rule reuses teacher NLLs already computed for the base OPD objective, requiring no additional model calls or semantic classifiers during training. Minimum-length constraints limit over-segmentation caused by isolated fluctuations, while structural boundaries and maximum-length constraints further stabilize the partition.

Together, the coherence analysis and the diagnostics in Appendix C support the practical value of segmentation combined with prefix-anchored calibration, without requiring every boundary to carry independent semantic meaning.

W4. Interpretation of Symmetric Calibration and Soft Gating

1. The squared log-ratio penalizes deviation from the prefix baseline in both directions. PACE compares each later segment’s mismatch ratio qiq_i with the trajectory’s prefix baseline bτb_\tau through ρi=qi/bτ\rho_i=q_i/b_\tau. The term (log⁡ρi)2(\log\rho_i)^2 penalizes departures from that baseline in either direction. This is by design: both an unexpected increase and an unexpected decrease relative to the prefix baseline represent deviations from the trajectory's established mismatch pattern that the calibration term regularizes.

2. Insensitivity near the baseline arises primarily from the squared log-ratio, not the gate. The full per-segment contribution is gi(log⁡ρi)2g_i(\log \rho_i)^2. When ρi≈1\rho_i \approx 1, the gate gig_i remains non-negligible, but log⁡ρi≈0\log \rho_i \approx 0, so the squared term rapidly suppresses the direct penalty. For example, at ρi=1.02\rho_i = 1.02, (log⁡ρi)2≈0.00039(\log \rho_i)^2 \approx 0.00039. Thus, the insensitivity to small fluctuations near the baseline comes from the squared log-ratio's measure of deviation magnitude, not from the gate approaching zero.

  1. Please refer to more analysis and experiments in response to Q3.

Q1. Fixed vs. Adaptive Prefix Length

We thank the reviewer for this suggestion.

1. The anchor fixes the number of segments, not the number of tokens.

We clarify that PACE dynamically partitions each trajectory based on teacher-NLL changes, structural boundaries, and length constraints. Although the anchor always uses the first $k=3$ segments, the token span it covers varies across trajectories, providing implicit adaptation to each trajectory's characteristics.

2. Adaptive anchor selection is promising but sensitive to the annotator model.

Adaptive anchor selection is promising but requires per-trajectory annotation. When identifying the onset of substantive reasoning, the trained agent selects a mean anchor length of 1.538 segments, versus 3.938 for GPT-5.4, showing substantial judge dependence. Fixed k=3k=3 falls between the two judge estimates and avoids per-trajectory model inference. It also balances baseline stability against content contamination: k=1k=1 is more sensitive to local fluctuations, whereas k=5k=5 risks including substantive reasoning in the baseline. In our experiments, k=3k=3 outperforms both fixed alternatives (Table 2 in paper) and the agent-judged adaptive variant. Reliable, low-cost adaptive selection remains a direction for future work.

Teacher → StudentMethodACEBenchBFCLBCBMBPP
9B → 2BPACE (k=3k=3)66.8071.300.22043.60
9B → 2BAdaptive anchor65.6368.400.20541.60

Q2. Distinguishing Capability Changes from Surface Variations

1. The objective is to reduce the influence of irrelevant variation. Segment-level averaging dampens isolated token-level fluctuations, while prefix calibration discounts persistent teacher–student discrepancies arising from stylistic preferences and other capability-irrelevant factors. These steps do not fully separate capability-related changes from irrelevant variation, and residual noise may remain as noted in the limitations section. Our analysis and experiments support the effectiveness of prefix calibration.

2. The boundary rule supports stable local aggregation. Teacher-NLL boundaries do not distinguish capability changes from stylistic or lexical variation. They define local units for aggregation: sharp NLL changes separate regions with different NLL levels, while a minimum-length constraint limits over-segmentation (Appendix D). A boundary may therefore reflect a stylistic change or a capability change. PACE addresses this uncertainty downstream through segment-level averaging and prefix-relative calibration, which reduce the influence of token-level fluctuations and persistent mismatch without claiming to classify their causes.

3. Ablation confirms the independent value of prefix calibration. Replacing ρ\rho with uncalibrated qq as the auxiliary loss input, with all other hyperparameters unchanged, consistently degrades performance:

Teacher → StudentMethodACEBenchBFCLBCBMBPP
9B → 2BPACE (ρ\rho)66.8071.300.22043.60
9B → 2BUncalibrated (qq)62.6266.400.19041.80

This confirms that the calibration step provides value independent of segmentation, even without explicit separation of capability-related and stylistic variation.

Q3. Symmetric Regularization vs. One-Sided Penalties

1. The calibration term penalizes deviation from the prefix baseline.

LPACE\mathcal{L}_{\text{PACE}} penalizes deviations of each later segment’s student-to-teacher NLL ratio qτ,iq_{\tau,i} from its trajectory’s prefix baseline bτb_\tau (Eqs. 12 and 14). Since (log⁡ρτ,i)2(\log\rho_{\tau,i})^2 is minimized at ρτ,i=qτ,i/bτ=1\rho_{\tau,i}=q_{\tau,i}/b_\tau=1, deviations in either direction are penalized; neither necessarily indicates an increase in absolute mismatch.

2. Downward deviations (ρ<1\rho<1) are not reliably indicative of improvement.

ρi<1\rho_i<1 means that qiq_i falls below the prefix baseline; it does not prove student improvement, since qiq_i also reflects teacher NLL, style, and confidence. Exempting such segments would wrongly treat every decrease as beneficial.

3. Statistical evidence shows that both directions co-occur with failure.

We sample 300 training trajectories and find that 74.85% of high-deviation segments, defined by (log⁡ρ)2>1.2(\log\rho)^2>1.2, are identified as erroneous by a strong LLM judge (GPT-6). Among these high-deviation segments, 41.91% have ρ>1\rho>1 and 58.09% have ρ<1\rho<1. Thus, errors occur with deviations in both directions, providing empirical support for symmetric penalization.

4. The one-sided ablation underperforms PACE.

Teacher → StudentMethodACEBenchBFCLBCBMBPP
9B → 2BPACE (symmetric)66.8071.300.22043.60
9B → 2BOne-sided (ρ>1\rho > 1 only)64.8067.840.20542.80

The consistent gap confirms that retaining the downward calibration signal is empirically beneficial.

I thank the authors for their response. I will keep my score unchanged. Good luck.

Thank you for the careful review, encouraging comments, and constructive suggestions. These insights help us improve the paper.

Reviewer_8VH7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACE adds a segment-level calibration loss to on-policy distillation. It partitions student-generated responses using changes in teacher negative log-likelihood (NLL), computes a student-to-teacher NLL ratio for each segment, and uses the opening segments as a detached reference. The auxiliary loss encourages later ratios to remain close to this reference, alongside the standard OPD objective. The motivation is to separate persistent stylistic differences from reasoning degradation. Experiments on tool-use and coding benchmarks show consistent improvements over standard OPD.

Critical strengths

PACE consistently improves over standard OPD across the evaluated teacher–student pairs and generally outperforms the stronger distillation baselines, particularly on tool-use tasks. The segmentation ablations also favor teacher-NLL-based boundaries over fixed windows and purely structural boundaries. These results provide evidence that the proposed regularization is useful beyond a single task or model configuration.

Critical weaknesses
  1. The prefix ratio is not sufficiently justified as a target for later segments.

    The method computes the segment ratio and its prefix-relative value as

    qτ,i=Sτ,i+ϵTτ,i+ϵ,ρτ,i=qτ,i+ϵbτ+ϵ.q_{\tau,i}=\frac{S_{\tau,i}+\epsilon}{T_{\tau,i}+\epsilon},\qquad \rho_{\tau,i}=\frac{q_{\tau,i}+\epsilon}{b_\tau+\epsilon}.

    Here, Sτ,iS_{\tau,i} and Tτ,iT_{\tau,i} are the student and teacher segment NLLs, and bτb_\tau is the detached average ratio over the prefix. Penalizing (log⁡ρτ,i)2(\log\rho_{\tau,i})^2 therefore encourages later segments to reproduce the opening’s relative likelihood.

    However, an opening dominated by formatting or problem restatement need not have the same normal student–teacher likelihood relationship as later reasoning, code, or tool calls. The prefix-content and model-size diagnostics do not establish that this ratio captures a nuisance factor that remains valid throughout the response. Appendix D shows cancellation of an assumed common multiplicative factor, but does not establish that this factor represents style rather than capability.

    The existing ablations vary segmentation and prefix length without isolating the value of the trajectory-specific anchor. Consequently, the central claim that prefix calibration separates nuisance mismatch from reasoning degradation remains insufficiently supported.

  2. The auxiliary update is not justified by the error-localization evidence.

    Appendix C.3 associates high-ρ\rho segments with erroneous content. However, the proposed loss can increase the likelihood of precisely these segments.

    Consider a fixed sampled trajectory with one post-prefix segment. The normalized gate in Eq. (14) cancels. Omitting numerical stabilizers and differentiating directly through the student NLL gives

    LPACE(τ)=(log⁡ρτ,i)2,∂LPACE(τ)∂Sτ,i=2log⁡ρτ,iSτ,i.L_{\mathrm{PACE}}(\tau)=(\log\rho_{\tau,i})^2,\qquad \frac{\partial L_{\mathrm{PACE}}(\tau)}{\partial S_{\tau,i}}=\frac{2\log\rho_{\tau,i}}{S_{\tau,i}}.

    When ρτ,i>1\rho_{\tau,i}>1, the auxiliary gradient pushes the student NLL downward, increasing the likelihood of the generated segment. Increasing likelihood can be appropriate for useful teacher-preferred content, but the paper specifically interprets high ρ\rho as evidence of reasoning degradation. It does not explain why reinforcing such content is the appropriate correction.

    This example does not imply that the combined objective necessarily harms performance, since the base OPD loss also contributes. It does expose a gap between identifying error-prone segments and justifying the update applied to them. The error-localization diagnostics do not resolve that gap.

Questions for authors
  1. Is Eq. (14) differentiated directly through both Sτ,iS_{\tau,i} and gτ,ig_{\tau,i}? How does the resulting update address the reinforcement of high-ρ\rho segments described in Weakness 2?

  2. With segmentation and base OPD unchanged, how does the method compare with bτ=1b_\tau=1, a shared anchor, or a shuffled-prefix anchor, allowing comparable tuning of the auxiliary weight?

  3. On the same segments and at a matched flagging rate, does the calibrated score ρτ,i\rho_{\tau,i} identify errors better than the uncalibrated ratio qτ,iq_{\tau,i}?

3 comments

We appreciate the reviewer’s recognition of PACE’s consistent improvements across teacher–student configurations, particularly on tool-use tasks, and the ablation evidence supporting teacher-NLL-guided segmentation. Below, we clarify the calibration objective and provide additional evidence addressing your concerns.

W1. Justification of the Prefix Anchor

Thank you for this important point. We revised our claim from separating nuisance mismatch to mitigating mismatch, as also acknowledged in our original limitations.

1. The prefix and later segments are generated by the same student and evaluated by the same teacher. Thus, even when the content shifts to reasoning, they can share a student–teacher mismatch in linguistic form, such as lexical preferences and surface conventions. Prefix calibration aims to mitigate this persistent, capability-irrelevant mismatch from being penalized as reasoning degradation.

OPD and PACE are complementary when the student–teacher gap widens substantially relative to the prefix baseline. While LOPDL_{OPD} provides teacher-guided next-token alignment, LPACEL_{PACE} reinforces it at the segment level. PACE helps mitigate persistent, capability-irrelevant mismatch.

2. Small deviations contribute little auxiliary noise. The soft gate reweights segments without a hard cutoff, and the squared log-ratio makes PACE’s penalty small near ρ=1\rho=1. Thus, PACE emphasizes substantial deviations without forcing later segments to reproduce the prefix exactly and uses a soft gating mechanism to reduce noise from small prefix-relative deviations.

3. Additional ablations examine the anchor’s contribution.

Teacher → StudentMethodACEBenchBFCLBCBMBPP
9B → 2BPACE66.8071.300.22043.60
9B → 2BUncalibrated (absolute qq)62.6266.400.19041.80
9B → 2BUncalibrated (code/tool only)64.1567.700.20042.20

Removing the anchor and replacing it with absolute qq degrades performance across all benchmarks. Even restricting the uncalibrated objective to code and tool-call segments underperforms PACE, demonstrating that the anchor's value is not limited to reasoning segments and that the absence of a relative reference degrades training regardless of content type.

4. Segment-level statistics confirm the anchor is content-type-agnostic.

Content type# SegmentsMean ρ\rhoMean raw S/TS/T
Code / Tool-call11,2792.141.80
Reasoning / NL7,3062.231.91

Both content types show similar raw $S/T$ ratios and anchor-relative $\rho$ distributions, providing evidence that the anchor captures a cross-content-type constant rather than a formatting-specific quantity.

5. Appendix C.5 provides the evidence that the anchor reflects nuisance.

We thank the reviewer for this helpful observation. If the anchor were driven primarily by student capability, one might expect its distribution to vary across students of different sizes (2B, 4B, and 9B). However, their initial anchor ratios nearly overlap (Fig. 3). We agree that Appendix D establishes the presence of a shared factor, rather than showing that this factor is entirely noise. Nevertheless, Fig. 3 shows consistency across models suggests that it depends only weakly on model capability. Together, our theoretical analysis and empirical results support the presence of an intrinsic noise component within the prefix. To state this more precisely, we have revised the relevant wording in the manuscript to clarify that our method mitigates the influence of this intrinsic noise rather than separating it.

W2. Error Localization and Auxiliary Update Direction

We thank the reviewer for this precise analysis.

1. Reinforcing teacher-supported erroneous content is a shared limitation of teacher-anchored distillation, not specific to PACE.

When ρi>1\rho_i>1 for an erroneous segment, the teacher may still assign higher probability to the erroneous tokens than the student does, even after prefix calibration. PACE cannot identify or correct teacher errors on its own—a limitation shared by teacher-referenced distillation methods [1, 2]. Correcting such errors requires external feedback, such as execution results, unit tests, or reward models, which lies outside the scope of PACE.

2. PACE targets capability-irrelevant noise, not systematic teacher errors.

We clarify that PACE cannot reliably identify or correct errors when the teacher favors incorrect content. Instead, it reduces noisy gradient updates arising from lexical, formatting, and punctuation variations through segment-level aggregation and prefix-relative calibration.

PACE's scope is reducing the influence of isolated token-level variance on the auxiliary update signal, not end-to-end error detection and correction.

[1] Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [2] When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation

3. The reinforcement risk is sparse and empirically bounded.

We also clarify the terminology in Appendix C.3: the “high-ρ\rho” segment is more accurately described as the “high-ρ\rho-deviation” segment, defined by (log⁡ρ)2>1.2(\log \rho)^2 > 1.2. This criterion captures substantial deviations of ρ\rho from 1 in either direction, rather than only cases where ρ>1\rho > 1. We will revise the wording accordingly.

To better analyze the deviation pattern, we sample 300 training trajectories and report the following:

  • Segments with (log⁡ρ)2>1.2(\log \rho)^2 > 1.2 account for only 14.35% of post-prefix segments, indicating that the auxiliary signal fires sparsely.
  • Of these segments, 74.85% belong to failed trajectories, consistent with reasoning degradation.
  • Among high-deviation segments, 41.91% have ρ>1\rho > 1, while 58.09% have ρ<1\rho < 1.

The worst-case risk of reinforcing erroneous content affects approximately 6% of post-prefix segments; its influence is further limited by λ=0.1\lambda=0.1. Improvements over OPD on all six benchmarks (Table 1) suggest that the benefits of noise reduction outweigh this risk.

Q1. Gradient Flow Through Student NLL and the Gate

In Eq. (14), gradients flow only through the student NLL Sτ,iS_{\tau,i}; the teacher NLL Tτ,iT_{\tau,i} is fixed, and the prefix anchor bτb_\tau is stop-gradiented. As discussed in our response to Weakness 2, PACE aims to reduce token-level noise and persistent teacher–student mismatch. Its improvements over OPD on all six benchmarks suggest that these benefits outweigh the risk of reinforcing teacher errors in our experiments.

Q2. Comparisons with Fixed, Shared, and Shuffled Anchors

We thank the reviewer for this helpful suggestion. We add three controlled ablations:

We test three anchor variants: b=1b=1 removes the trajectory-specific reference and penalizes qq directly; shared anchor uses the microbatch mean prefix ratio for every trajectory; and shuffled anchor replaces the first k=3k=3 consecutive segments with three segments sampled uniformly from the first half of each trajectory.

MethodACEBenchBFCLBCBMBPP
PACE66.8071.300.22043.60
b=1b = 162.6266.400.19041.80
Shared anchor61.2463.250.21040.70
Shuffled anchor61.0658.370.20039.60

All three variants underperform PACE. Each exposes a distinct failure mode:

Setting b=1b=1 removes trajectory-specific calibration, causing the auxiliary term to penalize intrinsic teacher–student mismatch. A shared microbatch anchor can be worse than no reference because it ignores differences across trajectories. A shuffled anchor may include capability-relevant content from later in the response, contaminating the reference and weakening its ability to isolate nuisance mismatch.

Together, these results confirm that PACE's gains require the anchor to be simultaneously trajectory-specific and structurally restricted to the response opening that reflects the intrinsic teacher–student mismatch rather than capability differences.

Q3. Error Detection with Calibrated vs. Uncalibrated Ratios

We thank the reviewer for this question. PACE does not aim to localize errors; it reduces token-level noise through segment aggregation and discounts capability-irrelevant mismatch through prefix anchoring. High deviation can correlate with errors for both calibrated and uncalibrated scores, so we attribute no unique error-detection ability to PACE. To assess calibration, we compare both scores on the same segments using (log⁡(⋅))2>1.2(\log(\cdot))^2>1.2 as the high-deviation threshold; the diagnostic and ablations above support its value.

Criterion% High-deviation segments% from failed trajectories
(log⁡ρ)2>1.2(\log \rho)^2 > 1.2 (PACE, calibrated)14.35%74.85%
(log⁡q)2>1.2(\log q)^2 > 1.2 (ablation, uncalibrated)16.88%65.75%

The calibrated score ρ\rho fires on fewer segments (14.35% vs. 16.88%) while aligning more tightly with failed trajectories (74.85% vs. 65.75%). This confirms that prefix calibration filters out capability-irrelevant persistent mismatch, leaving high-deviation segments more concentrated on genuine local degradation. The performance gap corroborates this:

MethodACEBenchBFCLBCBMBPP
PACE (ρ\rho)66.8071.300.22043.60
Uncalibrated (qq)62.6266.400.19041.80

High deviation correlates with errors in both calibrated and uncalibrated teacher–student NLL scores; the diagnostic shows only that prefix calibration makes this signal more focused, not that PACE can localize teacher errors. PACE instead aims to reduce token-level noise through segment aggregation and filter out capability-irrelevant mismatch through prefix anchoring.

Thank you for your time and valuable feedback. We would be grateful if you could share any further suggestions or follow-up comments.

Reviewer_aWn4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PACE, an auxiliary loss for on-policy distillation of LLM agents. Each student trajectory is split into segments at sharp changes in teacher NLL, with tool-observation and newline boundaries and length limits. For each segment, PACE computes the ratio of student NLL to teacher NLL. The average ratio over the first k segments serves as a detached, trajectory-specific anchor. Later segments whose ratio rises above this anchor are penalized through a soft-gated squared log-ratio loss, added to standard token-level OPD. With Qwen3.5 teacher→student pairs on tool-use and coding benchmarks, PACE outperforms OPD and other variants.

Critical strengths
  • The paper looks at a real problem in on-policy distillation: token-level KL/NLL differences between teacher and student are noisy and often get dominated by superficial stylistic or formatting differences early in the trajectory rather than actual capability gaps.
  • The proposed method is lightweight: it segments rollouts using change-points in teacher NLL, uses the first few segments as a detached baseline anchor, and adds a soft-gated segment loss. Because it reuses the forward passes already computed in OPD, it adds very little overhead and doesn't need an external reward model.
  • Experiments cover two teacher-student pairs across 6 coding and tool-use benchmarks, showing steady improvements over standard OPD and three recent variants, with particularly nice gains on BFCL and CRMArena.
Critical weaknesses
  • Missing standard baselines and unclear baseline tuning. All compared baselines are OPD variants. Standard alternatives like SFT on teacher trajectories, RL using the ground-truth ToolRL environment rewards, or simple token-level entropy/uncertainty-weighted OPD are missing.
  • The mathematical behavior of the auxiliary loss LPACEL_{\text{PACE}} is questionable and not properly analyzed. Since the teacher NLL is detached/fixed, the gradient of LPACEL_{\text{PACE}} only flows through the student's NLL on its own sampled tokens. That means minimizing this loss simply pushes the student to increase (or decrease) the probability of its own generated rollout segments—which is basically a form of weighted self-training or entropy sharpening/flattening, not pulling the student toward the teacher's distribution. The paper never writes out or discusses this gradient, nor does it compare against simple baselines that explicitly do confidence-weighted OPD or segment-level self-training.
  • All results are single-run (1 epoch, no random seeds or confidence intervals), and several benchmark gains are quite small
Questions for authors
  1. Were hyperparameters tuned on a held-out validation set, and can you report mean ± std over 3 seeds?
  2. Why does random segmentation degrade ACEBench 7 points below the untrained base model (59.64 vs. 66.95)?
  3. How are tool observations generated during on-policy rollouts on ToolRL, and how do SFT and GRPO perform on the same data?
4 comments

We sincerely thank the reviewer for recognizing the importance of addressing noisy teacher–student discrepancies, the lightweight design of PACE, and its consistent gains across coding and tool-use benchmarks. Below, we address your concerns and questions in detail.

W1. Baseline Coverage and Tuning

Thank you for this suggestion. ​Our method aims to ​address the limitations of OPD, so we conduct extensive comparisons against four OPD-based baselines.​ We have added three additional baselines: ​SFT on teacher trajectories​, ​RL with ToolRL environment rewards (GRPO)​, and ​token-entropy-weighted OPD​. Results are summarized below.

Teacher → StudentMethodMBPPBCBACEBenchBFCL
9B → 2BSFT0.3860.17557.3059.00
9B → 2BGRPO0.4180.20059.6663.75
9B → 2BToken-entropy OPD0.4180.19059.6664.25
9B → 2BPACE (Ours)0.4360.22066.8071.30
27B → 4BSFT0.6720.31065.0865.55
27B → 4BGRPO0.7120.34067.2872.05
27B → 4BToken-entropy OPD0.7260.34567.4473.80
27B → 4BPACE (Ours)0.7560.37569.8479.30

These baselines provide complementary comparisons:

  • SFT trains on teacher-generated trajectories offline. In multi-step agentic settings, the resulting train–inference distribution mismatch can limit its effectiveness, as the student encounters its own generation states at inference time.
  • GRPO optimizes directly on task-level rewards and represents a different training paradigm that requires environment reward supervision not needed by PACE. We include it as an informative reference point rather than a direct competitor.
  • Token-entropy-weighted OPD adjusts per-token supervision strength but lacks segment-level aggregation and within-trajectory calibration, leaving it exposed to the same token-level noise that motivates PACE. PACE consistently outperforms this variant, confirming that the gains stem from segment-level structure and prefix anchoring rather than token reweighting alone.

W2. Auxiliary Loss Gradients and Relation to Self-Training

We thank the reviewer for this careful analysis. We clarify the following points.

  1. We agree the teacher scores are not backpropagated through the auxiliary term and the prefix anchor is stop-gradiented, the gradient of $\mathcal{L}_\text{PACE}$ flows through the student's likelihood on its own sampled tokens. This is accurately described as a ​signed, teacher-score-conditioned segment-level self-likelihood update​.
  2. The teacher defines a calibrated reference in PACE. Crucially, the PACE loss serves as a calibration signal, not an objective to fit the teacher's distribution. Although teacher scores do not carry gradients through the auxiliary term, they serve two structural roles: (i) driving the adaptive segmentation boundaries, and (ii) forming the segment-level mismatch ratio $q_{\tau,i} = S_{\tau,i}/T_{\tau,i}$ and the prefix anchor $b_\tau$. The auxiliary term $\mathcal{L}​*\text{PACE}$ penalizes segments whose ratio deviates from the trajectory's own initial level, not unconditional self-reinforcement. Full teacher distribution matching remains the role of $\mathcal{L}*​\text{OPD}$.
  3. To directly test whether the gains reduce to confidence-weighted OPD or segment-level self-training, we add: (i) entropy-weighted OPD (segment-level confidence reweighting without prefix anchoring) and (ii) self-segment training (segment-level self-likelihood update without teacher-conditioned mismatch calibration). Results are below. Segment-level self-likelihood updates alone do not recover PACE’s performance, supporting the value of teacher-conditioned mismatch calibration. The gains over entropy-weighted OPD further suggest that segment-level reweighting alone does not account for the improvements achieved by combining segment-level aggregation with ​within-trajectory calibration​.
StudentTeacherMethodMBPPBCBACEBenchBFCL
Qwen3.5-2B9BSelf-segment0.4160.18559.2062.50
Qwen3.5-2B9BEntropy-weighted OPD0.4260.20561.3465.30
Qwen3.5-2B9BPACE0.4360.22066.8071.30
Qwen3.5-4B27BSelf-segment0.7020.34066.9872.05
Qwen3.5-4B27BEntropy-weighted OPD0.7180.35569.1674.50
Qwen3.5-4B27BPACE0.7560.37569.8479.30

4.The auxiliary term complements, rather than replaces, the base OPD objective.

In our setting, LOPDL_{{OPD}} aligns the student with the teacher’s output distribution, while LPACE{L}_{\text{PACE}} uses the same rollouts and teacher evaluations to provide a low-cost segment-level signal that discounts persistent teacher–student mismatch.

W3. Statistical Reliability of Reported Gains

Thank you for highlighting the need to quantify variability across runs. PACE improves over standard OPD in all 12 comparisons reported in the paper, with substantial gains on several benchmarks. For the 4B student, BFCL improves from 71.44% to 79.73% (+8.29 percentage points), and CRMArena from 16.80% to 22.28% (+5.48 points). For the 2B student, ACEBench improves from 59.04% to 66.36% (+7.32 points), and BFCL from 64.10% to 71.02% (+6.92 points). These results show consistent improvements across the evaluated settings. To assess their statistical reliability, particularly where gains are smaller, we report mean ±\pm standard deviation across three random seeds for all main results in the paper.

Q1. Hyperparameter Selection and Results Across Seeds

Hyperparameters were selected from a small candidate set using a held-out validation split of the training data only. The test sets of all benchmarks were not used for hyperparameter selection. Once fixed, the same configuration was applied across all reported experiments. Mean ± std over 3 seeds is reported in paper.

Q2. Performance Drop with Random Segmentation

We first clarify the random segmentation setup. Boundaries are sampled uniformly at random ​without minimum or maximum segment-length constraints​, while the number of segments is matched to PACE. The resulting segment lengths range from 1 to 172 tokens. This differs from the fixed-window ablation, which enforces a constant length. Comparing the two isolates the effect of boundary placement from that of segment length distribution. We attribute the degradation to two potential sources of instability:

  • Degraded prefix anchor. PACE constructs the trajectory-specific anchor $b_\tau$ from the first $k=3$ segments. Without length constraints, the first three random segments may be very short (dominated by a few tokens) or may accidentally cover substantive reasoning content, making $b_\tau$ an unreliable estimate of the trajectory's initial capability-irrelevant mismatch.
  • Noisy post-prefix statistics. Subsequent random segments may be too short for stable NLL averaging, or too long such that they conflate semantically distinct generation phases. Both cases corrupt $\rho_i = q_i / b_\tau$, causing the auxiliary loss to respond to aggregation artifacts rather than genuine local capability gaps.

When both the calibration reference and the post-prefix segment statistics are simultaneously destabilized, the auxiliary term can reinforce or suppress the wrong segments, producing updates that are worse than no auxiliary signal.

Q3. ToolRL Observation Generation and SFT/GRPO Comparisons

In our ToolRL setup, tool calls are evaluated without executing real tools. Given a user query and tool descriptions, the model generates a sequence of tool-call decisions. Evaluation checks whether the appropriate tools are selected, whether their arguments satisfy the specified constraints, and whether the calling order is valid.

For coding tasks, student-generated code is executed in an actual interpreter. The execution results are then included as observations in the context for subsequent generation.

The SFT and GRPO comparisons using the same training data are reported in our response to W1.

Thanks for the author's response. I have a better understanding of the paper now, and I think it's above average. I'll increase my score.

Thank you for your thoughtful review and for your positive recognition of our work. Your feedback is valuable and constructive for improving our paper.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

We sincerely thank all reviewers for their careful reading and constructive feedback. Their comments have helped us clarify important points and improve the manuscript.

PACE adaptively partitions student trajectories into coherent segments and aggregates discrepancies within each segment to reduce token-level noise. It then uses a trajectory-specific prefix anchor to calibrate later segments, helping to mitigate intrinsic teacher–student mismatch arising in part from capability-irrelevant factors, such as differences in linguistic style.

We appreciate the reviewers’ recognition of the importance of addressing capacity-irrelevant noise in teacher–student discrepancies during OPD, including noise arising from differences in style and formatting (aWn4, 1mcn), their positive assessment of PACE’s lightweight and intuitive design (aWn4, 1mcn), and their acknowledgment of its consistent improvements across teacher–student configurations and benchmarks (aWn4, 8VH7, 1mcn). In response to their comments, we have conducted additional experiments and analyses and revised the manuscript to clarify the following points:

  • Additional baselines (Appendix B.2): Comparisons with SFT, GRPO, entropy-weighted OPD, and segment-level self-training show that PACE’s gains are not explained by token reweighting or self-likelihood updates alone.
  • Prefix-anchor ablations (Appendix B.3; Section 5.3): Comparisons with uncalibrated, shared, and shuffled anchors support the value of a trajectory-specific opening reference. The fixed three-segment anchor outperforms the evaluated adaptive variant while avoiding additional per-trajectory model inference for anchor selection. We also compare symmetric calibration with a one-sided penalty.
  • Segmentation and calibration analyses (Section 5.4; Appendices C.3–C.7; Tables 6–9 and Figure 3): Additional analyses and quantitative evaluations of segment coherence, prefix content, and calibrated versus uncalibrated discrepancy scores provide further evidence for the value of segment-level aggregation and prefix calibration.
  • Reliability and methodological clarity (Section 5; Appendix B.1; Appendix B.3): We report results across three seeds and clarify hyperparameter selection, the random-segmentation setup, ToolRL evaluation, and the gradient behavior of the auxiliary loss.
  • Clarified claims and segmentation rationale (Sections 4 and 6; Appendix A; Appendix D.2): We clarify that PACE mitigates noisy teacher–student discrepancies without claiming to fully disentangle capability-related and capability-irrelevant variation. We also clarify that segmentation provides local aggregation units and aims to avoid averaging across markedly different teacher-NLL levels, rather than explicitly identifying capability-related transitions.

We are grateful to reviewers aWn4 and 1mcn for their thoughtful questions. We are pleased that our responses during the rebuttal addressed these questions, further clarified our work, and contributed to their positive assessments. Through additional experiments and analyses, we addressed reviewer 8VH7’s constructive questions about the role of the prefix anchor and clarified a potential misunderstanding regarding error localization, which we do not claim as a contribution.

We thank the reviewers for their constructive suggestions, which helped us clarify the method, evidence, and limitations. We also thank the ACs, SACs, and PCs for coordinating the review process and appreciate the time and care given to our submission. We hope that the clarifications and additional evidence provided during the rebuttal will be considered in the final decision. Thank you for your thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.