acceptodds
Under review as a conference paper at ICLR 2027

TrajPD: Constructing Trajectory-Aware Targets for On-Policy Distillation in Tool-Integrated Reasoning

Abstract

Transferring tool-integrated reasoning (TIR) capabilities to small language models (SLMs) benefits from supervision adapted to their multi-turn interactions. On-policy distillation (OPD) provides dense teacher guidance on trajectories generated by the student. In multi-turn TIR, student responses and tool feedback shape subsequent prediction contexts, and the tokens within each complete response jointly form an action or answer. This motivates constructing targets that remain close to the student’s current predictions while retaining a specified teacher contribution across each response. We propose Trajectory-Aware Policy Distillation (TrajPD), which constructs targets jointly within each complete response. TrajPD selects targets that minimize the KL divergence from the targets to the student’s predictions, averaged over positions within the response, while satisfying a response-level constraint on the retained teacher contribution. At the sampled contexts and the start of an update, the weighted distillation gradient preserves the prescribed component along the direct teacher gradient in logit space, aggregated across the response. The targets are obtained by adjusting the student’s log probabilities along directions derived from differences between teacher and student probabilities, using a single scale for each response. A weight determined by the average divergence between teacher and student predictions within the response determines this scale and also weights the distillation loss. Evaluation on four benchmarks spanning mathematical reasoning, scientific reasoning, and code generation shows improved aggregate performance at both student scales, with gains across multiple tasks and evaluation metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Tool-integrated reasoning, Small language models, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_coHiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrajPD for distilling tool-integrated reasoning into small language models. It constructs targets jointly within each response, minimizing their KL divergence from the student while retaining a specified component of the teacher gradient. The targets are used with response weights and task rewards to update the student. Experiments with Qwen3-0.6B and Qwen3-1.7B report higher average performance across four benchmarks, with the clearest gains over SOD on GPQA. The authors also analyze response completion and the geometry of the constructed targets.

Critical strengths
  1. Jointly constructing targets for a complete response is a reasonable design for tool use. The optimization problem is clearly defined and can be solved through a single scalar search per response.

  2. The analysis of completion rates is helpful for understanding the GPQA gains. The appendix also provides uncertainty estimates and a training–evaluation overlap check.

  3. The derivation is careful about the scope of the gradient identity. In particular, it acknowledges the difference between logit-space properties and updates to shared model parameters.

Critical weaknesses
  1. The improvement over SOD is uneven. Mean Avg@8 increases by 2.67 points at 0.6B and 0.89 at 1.7B, while both AIME scores decrease at both sizes. Appendix G.2 gives a 95% interval of [-1.77, 3.34] for the 1.7B average gain; the LiveCodeBench intervals also include zero. On GPQA, much of the gain appears to come from completing more responses. This is useful, but I would like to see independent training runs and controls for stopping behavior before drawing a broad conclusion about improved reasoning.

  2. Does the gain specifically require joint target construction across a response? Table 2 ablates the main components, but the token-wise alternative is compared mainly through target KL on recorded predictions. Lower KL is what TrajPD is designed to optimize. Training the token-wise alternative under the same settings would provide a more direct test of whether response-level construction improves the policy. A comparison with direct teacher matching at matched gradient scale would also be informative.

  3. The Pass@8 definition needs to be stated in the main text. Appendix C.2 explains that it uses 1,000 resamples with replacement from eight recorded answers. For a question with one correct answer, this gives an expectation of 1-(7/8)^8, or about 0.656, although the observed set of eight already contains a correct answer. Please report the observed eight-sample success rate and label the resampling result separately, so that readers can compare it with other work.

  4. I am still unsure why minimum target KL is the right criterion for the actual update. Appendix A.2.7 shows that the constructed logit gradient can have a larger norm than direct matching, and its projection property need not survive the model Jacobian and optimizer. The paper acknowledges this limitation. Measuring parameter-update alignment and teacher loss after the update would help establish the connection between the target construction and training behavior.

Questions for authors

see Critical Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer coHi - part one

Thank you for taking the time to read our manuscript closely and for sharing your valuable and constructive comments. We particularly appreciate your suggestions to examine completion and stopping behavior and the parameter updates produced by the proposed target construction. We will address each of your comments below.

W1. Completion, stopping behavior, and independent training

Thank you for your valuable suggestions on training variability and controls for stopping behavior. We report results from repeated training and compare the number of responses that reach the per-turn limit and mean Avg@8 under both settings.

W1(a). Independent training.

We ran TrajPD three times at each size. At 1.7B, each run scored above the original SOD result of 37.40% mean Avg@8.

TrajPD modelMean Avg@8 (%)Mean standard Pass@8 (%)
0.6B23.83 ± 0.2957.02 ± 0.40
1.7B38.69 ± 0.3564.88 ± 1.19

We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

W1(b). Stopping behavior. Thank you for this suggestion. We increased the generation limit per turn from 8,192 to 12,288 tokens for both TrajPD models while keeping the total generation budget fixed. Table 8 of the revised manuscript reports the number of responses that reach the per-turn limit and mean Avg@8 under each setting.

TrajPD modelResponses reaching the turn limitMean Avg@8 (%) at 8,192 tokensMean Avg@8 (%) at 12,288 tokensChange (pp)
0.6B1,926 → 1,14523.8122.93−0.88
1.7B1,715 → 98838.2937.66−0.63

The higher limit reduced the number of responses reaching the turn limit. The original setting retained higher mean Avg@8 at both model sizes, supporting our choice of the original generation limit.

W2(a). Training with per-token

Both 0.6B methods use 100 updates and 128 rollouts per update. We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

0.6B methodMean Avg@8 (%)Mean standard Pass@8 (%)
TrajPD23.83 ± 0.2957.02 ± 0.40
per-token22.07 ± 1.0152.84 ± 2.46

W2(b). Mixture and the gradient reference. Thank you for suggesting a comparison with direct teacher targets. For current probabilities qq, teacher probabilities pp, and response weight wkw_k, we use the Mixture target in Eq. (20)

r=q+p−qwk.r=q+\frac{p-q}{w_k}.

At the logits used to construct rr, holding the target and weight fixed gives the identity underlying Eq. (21)

wk(q−r)=q−p.w_k(q-r)=q-p.

The weighted Mixture gradient therefore matches the gradient from the unweighted teacher target at that point. The benchmark scores below are from seed 42.

0.6B targetAIME 2024AIME 2025GPQA DiamondLiveCodeBench v6Mean Avg@8
TrajPD, Tables 1 and 217.0819.1727.2131.7923.81
Mixture18.3317.5026.6425.6422.03

We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

0.6B methodMean Avg@8 (%)Mean standard Pass@8 (%)
TrajPD23.83 ± 0.2957.02 ± 0.40
Mixture21.97 ± 0.1750.00 ± 0.23

TrajPD achieves higher mean Avg@8 and standard Pass@8 across the three seeds. On LiveCodeBench v6, its seed 42 run scores 31.79% Avg@8, compared with 25.64% for Mixture.

AuthorsOpenReview ↗

Response to Reviewer coHi - part two

W3. Standard Pass@8. Thank you for asking us to report standard Pass@8 explicitly. Standard Pass@8 counts a question as solved if at least one of its eight recorded answers is correct. Equation (8) defines this metric, and Table 1 reports it by benchmark for the original runs.

Method0.6B Avg@80.6B Pass@81.7B Avg@81.7B Pass@8
Vanilla (old)12.2025.4813.5325.82
Vanilla (standard)12.2028.7813.5329.70
SFT (old)11.9329.709.3528.13
SFT (standard)11.9335.359.3534.46
GRPO (old)12.9228.0817.7734.53
GRPO (standard)12.9232.5917.7738.97
OPD (old)16.7438.3330.5451.04
OPD (standard)16.7444.3930.5455.44
EOPD (old)20.4141.9131.9550.97
EOPD (standard)20.4147.7031.9555.90
SOD (old)21.1441.9937.4059.20
SOD (standard)21.1448.0137.4064.48
TrajPD (old)23.8149.9938.2959.53
TrajPD (standard)23.8156.5738.2964.14

W4. Actual parameter updates. Thank you for suggesting measurements of the actual parameter update. We minimize target KL while retaining the prescribed teacher contribution. This keeps the targets as close as possible to the current predictions. We then measure the actual parameter update and teacher KL after one AdamW step.

We compare TrajPD, Mixture, and per-token using the 0.6B model (Table 6 and Appendix C.6). Update cosine measures the alignment of the parameter update with the negative gradient of the teacher loss. Teacher KL is evaluated before and after the update, with an initial value of 0.13280.

TargetMean target KLActual update normUpdate cosineTeacher KL after updateTeacher KL reduction (%)
TrajPD0.090.062670.303480.0626252.85
Mixture0.120.062770.315650.0557158.05
per-token0.100.062730.313170.0571456.97

TrajPD has the lowest target KL and an update norm similar to the alternatives. Its actual parameter update has a positive cosine of 0.30348 with the teacher descent direction, and teacher KL decreases from 0.13280 to 0.06262. These measurements show positive alignment and a 52.85% reduction in teacher KL after the actual optimizer step on these saved responses.

Dear Reviewer coHi

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

TrajPD authors

Reviewer_EahNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation for tool-integrated reasoning. It treats each complete model response as a decision and jointly constructs distillation targets for the tokens within that response. On a shared partition consisting of the teacher’s top-16 tokens and a single aggregate for the remaining vocabulary, the method minimizes the mean KL divergence from the targets to the current student predictions, subject to a constraint on the projection of the weighted distillation gradient onto the direct teacher gradient. This convex problem yields exponentially tilted targets, obtained by solving for a scalar shared across the response using bisection. A response weight is determined by the mean JSD. Training also uses verifier scores for completed trajectories. Experiments use a Qwen3-4B teacher and 0.6B and 1.7B students on AIME24/25, GPQA-Diamond, and LiveCodeBench. Relative to SOD, the four-benchmark mean Avg@8 improves by 2.67 and 0.89 percentage points, respectively.

Critical strengths
  1. The central target construction has an explicit and testable theoretical foundation. Equation (6) goes beyond heuristic interpolation by finding minimum-KL targets subject to preserving a specified teacher contribution. Appendix A derives the exponential form with a shared scalar, establishes existence and uniqueness, and accounts for the gradient metric induced by grouping the vocabulary tail. The authors also explicitly show that lower target KL does not imply a smaller weighted gradient norm, and that the logit-space result does not automatically survive transformation by the model Jacobian. These qualifications strengthen the technical argument.

  2. The experiments and diagnostic checks provide a reasonably complete body of evidence, with limitations reported transparently. Table 1 achieves the highest mean Avg@8 at both student sizes. GPQA improves over SOD by approximately 9.85 and 4.67 percentage points, with paired question-level 95% intervals of [6.63, 13.13] and [1.64, 7.70] in Appendix G.2. These intervals directly support this part of the improvement. In Table 2, the full method also outperforms a variant that retains response weights but uses direct teacher targets. The paper additionally provides constraint-residual checks, failure cases, and a benchmark-overlap audit of the actual training splits, while disclosing overlaps found in the upstream data files.

Critical weaknesses
  1. The robustness and scope of the overall gains remain limited. The 0.6B four-benchmark mean improvement has a positive question-level interval, but the 1.7B gain of 0.89 points has an interval of [-1.77, 3.34], which includes zero. The LiveCodeBench difference intervals also include zero at both sizes (Appendix G.2). Avg@8 on both AIME24 and AIME25 is lower than SOD at both sizes. The strongest evidence therefore concerns recovery from SOD’s degradation on GPQA, rather than consistently improved performance across domains. Relative to EOPD, the GPQA gains are only 0.63 and 0.57 points, and the 0.6B result remains below Vanilla, showing that the apparent advantage depends on the comparator. The reported intervals describe evaluation-question variability for fixed trained policies; as the authors acknowledge, they do not capture variation across independent training runs. The central target ablations are also restricted to 0.6B, further limiting conclusions about robustness across scales.

  2. The geometric advantage of joint construction within a response lacks a corresponding downstream policy comparison. Figure 3 and Appendix A.2.8 show that joint optimization yields lower target KL than independently satisfying the same requirement at each token. However, these comparisons reuse the same predictions rather than compare separately trained policies. The direct-teacher-target ablation in Table 2 supports the full target reformulation but does not isolate joint allocation across a complete response from token-wise exponential targets. Appendix A.2.7 correctly notes that lower target KL does not guarantee smaller parameter updates or better task performance. The current evidence therefore supports the optimization property, but does not sufficiently establish the practical necessity of using complete responses as the grouping unit. Since this is an important distinction from more general target reformulation, I would like to see a training comparison that holds weights, task rewards, and training budgets fixed.

Questions for authors
  1. Are there existing training results for token-wise targets with response weights, rewards, and training budgets matched to the full method? If target construction is instead grouped into fixed-length blocks or complete interaction trajectories, does performance still support using natural response boundaries? This would affect my assessment of the independent value of response-level construction.

  2. How much do the central comparisons and target ablations vary across independent training seeds? In particular, do the 1.7B overall gain and the 0.6B improvement of 1.65 points over direct teacher targets persist across repeated training runs?

  3. Could you report the standard empirical Pass@8, namely whether at least one of the eight recorded responses is correct, and clarify whether its rankings agree with those of the resampling-with-replacement metric in the main table? Appendix C.2 already discloses the distinction; making it explicit in the main table would affect the interpretation of benefits from multiple samples.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EahN - part one

Thank you for your careful reading of our paper and your constructive comments. We appreciate your questions about the choice to construct targets jointly within each response, the consistency of the results across independently trained models, and the definition of Pass@8. Your suggestions help us clarify the motivation for this design and the interpretation of the comparisons. We provide our responses to each point below.

W1. Results by task and comparator

Thank you for asking us to clarify the performance gains across different tasks and comparators. TrajPD improves mean Avg@8 over SOD by 2.67 and 0.89 percentage points at 0.6B and 1.7B, respectively. The corresponding paired 95% confidence intervals across evaluation questions are [0.59, 4.60] and [−1.77, 3.34] for the original trained models (Appendix G.2).

On 0.6B GPQA, TrajPD achieves 27.21%, compared with 17.36% for SOD and 28.35% for Vanilla. The reported 9.85-point gain is therefore relative to SOD. Improvements also occur where SOD already improves over Vanilla. At 1.7B on GPQA, SOD increases accuracy from 27.27% to 29.36%, and TrajPD further improves it to 34.03%. The GPQA gains over EOPD are 0.63 and 0.57 points at the two model sizes.

On LiveCodeBench, TrajPD improves over SOD by 2.93 and 2.64 points at 0.6B and 1.7B, with paired 95% intervals of [−0.50, 6.43] and [−0.86, 6.29], respectively. SOD already improves over Vanilla on this benchmark at both model sizes, and TrajPD achieves further gains.

The four-benchmark mean includes both AIME benchmarks. TrajPD achieves 17.08% and 19.17% on AIME 2024 and AIME 2025 at 0.6B, and 41.67% and 33.33% at 1.7B. The corresponding SOD scores are 18.33% and 20.00% at 0.6B, and 44.58% and 34.17% at 1.7B. The gains on GPQA and LiveCodeBench produce the reported overall mean improvements.

We further evaluate the contribution of target construction through the 0.6B comparisons in Tables 2 and 3. These comparisons retain the same task evaluation and examine alternative distillation targets directly.

W2 and Q1(a). Training with per-token

Thank you for suggesting a direct training comparison to examine the contribution of constructing targets jointly within a complete response.

TrajPD imposes the required teacher contribution over a complete response, allowing different positions to contribute different amounts. The per-token variant imposes the requirement independently at each position while retaining response weights and task scoring rules. This comparison directly examines the contribution of joint target construction to the trained model.

Both 0.6B methods use 100 updates and 128 rollouts per update. We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

0.6B methodMean Avg@8 (%)Mean standard Pass@8 (%)
TrajPD23.83 ± 0.2957.02 ± 0.40
per-token22.07 ± 1.0152.84 ± 2.46

Across these runs, allocating the teacher contribution over a complete response improves mean Avg@8 by 1.76 percentage points and mean standard Pass@8 by 4.18 percentage points.

Q1(b). Groups of fixed length and complete interactions

For fixed-length blocks and complete interactions, Table 7 reports target KL on saved predictions. The task scores in Q1(a) come from training TrajPD and per-token.

To examine this choice, we compared four grouping strategies using the same predictions from 56 saved interactions, containing 151 responses and 557,959 generated positions. All comparisons use unit weights. Target KL is averaged over positions within each record, followed by equal weighting across records.

Grouping on saved predictionsMean target KL
per-token0.242556
Contiguous blocks of 256 generated positions0.189626
Complete responses0.188783
All recorded generated positions in each interaction0.188696

Complete-response grouping reduces mean target KL from 0.242556 to 0.188783, approaching the 0.188696 obtained by grouping all recorded positions within each interaction.

Fixed-length blocks divide positions according to token count, whereas grouping an entire interaction combines positions generated across successive observations. Complete responses preserve the interaction-defined boundaries while allowing positions within each response to jointly satisfy the teacher contribution requirement.

AuthorsOpenReview ↗

Response to Reviewer EahN - part two

Q2(a). Independent training at both model sizes

The three 1.7B TrajPD runs all scored above 37.40% mean Avg@8, the score of the original SOD run. For both model sizes, the table reports the mean ± sample standard deviation across seeds 42, 2028, and 1997.

TrajPD modelMean Avg@8 (%)Mean standard Pass@8 (%)
0.6B23.83 ± 0.2957.02 ± 0.40
1.7B38.69 ± 0.3564.88 ± 1.19

Q2(b). The 0.6B comparison with direct teacher targets

The 0.6B comparison in Table 2 uses the original trained pair and retains response weights. Q2(a) reports variation across repeated TrajPD runs.

TrajPD achieves 23.81% mean Avg@8, compared with 22.16% for direct teacher targets, an improvement of 1.65 points with a paired 95% confidence interval of [−0.06, 3.37] across evaluation questions. On LiveCodeBench, TrajPD achieves 31.79%, compared with 26.57% for direct teacher targets. The reported gain is 5.21 points, with a paired 95% interval of [2.79, 7.79] (Appendix G.2).

Q3. Standard Pass@8 and rankings

Standard Pass@8 counts a question as solved if at least one of its eight recorded answers is correct. Equation (8) defines this metric, and Table 1 reports it by benchmark for the original runs.

TrajPD sizeAIME 2024AIME 2025GPQALiveCodeBenchMean
0.6B40.0043.3373.2369.7156.57
1.7B70.0046.6766.1673.7164.14

TrajPD achieves the highest mean standard Pass@8 at 0.6B, at 56.57%. At 1.7B, SOD, TrajPD, EOPD, and OPD rank in that order, with mean standard Pass@8 of 64.48%, 64.14%, 55.90%, and 55.44%, respectively.

Dear Reviewer EahN

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

TrajPD authors

Reviewer_uyp6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies on-policy distillation for small language models in multi-turn tool-integrated reasoning (TIR). The authors argue that direct teacher matching and token- or step-wise supervision may introduce unnecessary changes to student predictions. To address this issue, they propose TrajPD, which jointly constructs distillation targets within each complete model response. Subject to a constraint that preserves a specified component of the teacher-matching gradient, the method minimizes the average KL divergence between the constructed targets and the current student predictions. The targets are obtained using a shared scaling parameter within each response. A response weight based on the Jensen–Shannon divergence between teacher and student predictions is incorporated into both target selection and the distillation loss, which is combined with a policy-gradient term based on final task scores.

The theoretical analysis establishes the existence and uniqueness of the constrained solution and a logit-space gradient projection identity at fixed contexts and the beginning of an update. Experiments are conducted with Qwen3-0.6B and Qwen3-1.7B students, a Qwen3-4B teacher, and four benchmarks: AIME 2024, AIME 2025, GPQA-Diamond, and LiveCodeBench. Compared with SOD, TrajPD improves the four-benchmark mean Avg@8 by 2.67 and 0.89 percentage points, respectively. Improvements are concentrated in scientific reasoning and code generation, while Avg@8 is lower on both AIME benchmarks. The paper also includes component ablations, target-KL analysis, and response-completion analysis.

Critical strengths

S1. A mathematically well-defined approach to response-level distillation target construction.

The paper proposes a method for jointly constructing distillation targets within a complete response, allowing different token positions to receive different amounts of teacher correction. The authors formulate this process as a KL minimization problem subject to a constraint on the teacher-gradient contribution. They derive the existence, uniqueness, and exponential form of the optimal solution, and establish a logit-space gradient projection property at the beginning of an update. Overall, the mathematical derivations are reasonably thorough and provide a useful perspective on allocating distillation supervision across token positions.

S2. Useful experimental analyses of method components and model behavior.

Beyond comparing TrajPD with existing distillation and reinforcement learning methods, the paper includes component ablations, target-KL analysis, and response-completion analysis. The experiments verify that the constructed targets achieve the intended minimum-KL property and show performance gains on some tasks. In particular, the improvements over SOD on GPQA are supported by positive evaluation bootstrap intervals at both student sizes. The analysis of completion rates and conditional correctness also provides useful observations for understanding changes in final accuracy in multi-turn tool-integrated reasoning.

Critical weaknesses

W1. Insufficient empirical evidence connecting the proposed distributional geometry problem to actual performance limitations.

The central motivation of the paper is that existing token-wise or step-wise distillation may introduce unnecessary changes to the target distributions, which response-level joint target construction aims to reduce. The authors mathematically establish that their constructed targets minimize KL divergence under a prescribed teacher-gradient contribution constraint, and empirically verify this property. However, a smaller target-to-student KL divergence does not necessarily imply better parameter updates or downstream task performance. It therefore remains unclear whether the additional distributional changes identified by the authors are an important factor limiting the performance of existing distillation methods.

Although the paper includes an ablation that removes target construction, it does not report a direct training comparison between response-level and token-wise target construction under matched teacher-contribution constraints and training conditions. As a result, the current evidence does not fully establish the connection between the geometric advantage and the observed learning gains. Such a comparison would help directly validate the paper's central algorithmic design and clarify its practical value relative to existing approaches.

W2. Modest and task-dependent performance gains with limited evidence of training stability.

Although TrajPD achieves higher four-benchmark mean Avg@8 than SOD at both student sizes, the improvements are 2.67 and 0.89 percentage points, respectively. For the 1.7B student, the 95% bootstrap confidence interval for the mean improvement includes zero. TrajPD also obtains lower Avg@8 than SOD on both AIME 2024 and AIME 2025 at both student sizes, indicating substantial variation across tasks. In addition, the paper does not report results from independent training seeds, and the existing uncertainty analysis only captures variation across evaluation questions for the fixed trained policies.

Given that the main contribution is a new distillation target construction method, consistent and attributable performance improvements are important for establishing its practical value. Additional independent training runs and more direct target-construction comparisons would help determine whether the observed gains can be reliably attributed to the proposed design rather than particular tasks, training configurations, or randomness.

Questions for authors

Q1. Motivation and the practical benefit of response-level target construction

The paper shows that response-level joint target construction achieves lower KL divergence under a prescribed teacher-contribution constraint. However, it remains unclear whether the larger target KL associated with existing methods actually harms downstream performance. Could the authors clarify this connection? In particular, could they directly compare response-level and token-wise target construction under matched teacher-contribution constraints, training budgets, and loss settings to demonstrate that joint construction improves learning outcomes beyond simply reducing target KL?

Q2. Stability and statistical reliability of the performance gains

TrajPD improves the mean accuracy over SOD by 2.67 and 0.89 percentage points at the two student sizes, respectively. However, the improvement at 1.7B is relatively small, and its 95% confidence interval includes zero. In addition, the reported uncertainty analysis only considers variation across evaluation questions, not independent training runs. Could the authors provide results across multiple independent training seeds, particularly at 1.7B, to assess the stability of the reported improvements?

Q3. Necessity of the JSD-based response weight

The ablation removing response weights shows that this component contributes to the average performance. However, it does not establish why the proposed JSD-based dynamic weighting scheme is necessary. Could the authors compare it with fixed weights or other simple weighting alternatives? Since the response weight affects both target construction and the distillation loss, could the authors also clarify the contribution of each role to the observed performance gains?

Q4. Relationship between performance gains and response completion

Figure 4 shows that TrajPD improves response completion rates across all task groups, while accuracy conditional on completion is lower than that of SOD. Could the authors clarify how much of the overall accuracy improvement comes from higher completion rates versus changes in reasoning correctness? In particular, why do the substantial completion improvements on AIME not translate into higher final accuracy?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer uyp6 - part one

We thank you for your thoughtful review and helpful suggestions. We appreciate your questions about the benefits of joint target construction, the role of response weighting, and the relationship between response completion and accuracy. Your feedback helps us explain the motivation for our design and the interpretation of our findings more clearly. Below, we address these points together with your questions about training stability.

W1 and Q1: Joint construction in training.

Thank you for suggesting a direct training comparison to examine the contribution of joint construction. At 0.6B, TrajPD and per-token both use 100 updates and 128 rollouts per update, with the same response weighting and task scoring rules.

The teacher contribution in Eq. (6) specifies the guidance that the targets must retain. Minimizing KL then selects the targets that depart least from the current predictions while providing that contribution. This makes KL a criterion for choosing among targets that meet the same requirement. The practical question is whether the freedom to distribute the contribution over a complete response improves the resulting trained model.

The difference between joint construction and per-token concerns where this requirement is imposed. TrajPD uses

wk∣Ik∣∑t∈Ik(ut−qt)⊤dt≥Gk.\frac{w_k}{|I_k|}\sum_{t\in I_k}(u_t-q_t)^\top d_t \ge G_k.

Here, IkI_k contains the positions in response kk, wkw_k is the response weight, and utu_t and qtq_t are the target and current probabilities. Equation (4) defines the teacher direction dtd_t and reference contribution GkG_k. The requirement applies to the sum over the response, allowing its positions to contribute different amounts.

Positions can differ in the probability change needed to supply a contribution in the teacher direction. Joint construction can use these differences while meeting the response's requirement. The per-token variant requires each position to meet its own requirement. Both variants construct targets through KL minimization, so their comparison examines the choice of grouping within the target formulation.

The table reports Avg@8 percentages for individual runs.

0.6B targetAIME 2024AIME 2025GPQA DiamondLiveCodeBenchMean Avg@8
TrajPD, Tables 1 and 217.0819.1727.2131.7923.81
Direct teacher, Table 218.7518.3325.0026.5722.16
Mixture18.3317.5026.6425.6422.03

The direct teacher comparison additionally examines target construction as a whole. With response weights retained, TrajPD improves mean Avg@8 from 22.16% to 23.81%. On LiveCodeBench, the reported gain is 5.21 points, with a paired 95% interval of [2.79, 7.79] across evaluation questions (Appendix G.2).

We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

0.6B methodMean Avg@8 (%)Mean standard Pass@8 (%)
TrajPD23.83 ± 0.2957.02 ± 0.40
per-token22.07 ± 1.0152.84 ± 2.46
Mixture21.97 ± 0.1750.00 ± 0.23

Figure 3 characterizes joint construction on fixed predictions, reporting a median reduction in target KL of 17.35% across 91 responses while retaining the prescribed teacher contribution.

W2 and Q2: Independent training.

To measure variation across training runs, we trained TrajPD independently at both sizes. All three 1.7B runs scored above the original SOD result of 37.40% mean Avg@8. We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

TrajPD modelMean Avg@8 (%)Mean standard Pass@8 (%)
0.6B23.83 ± 0.2957.02 ± 0.40
1.7B38.69 ± 0.3564.88 ± 1.19

Q3: Response weights

Q3(a): Comparison with unit weights. At 0.6B, the variant with wk=1w_k=1 in both roles achieves 20.41% mean Avg@8, and the main run with JSD weights achieves 23.81%. Both use 100 updates, with 16 and 128 rollouts per update, respectively. JSD measures disagreement symmetrically, and the response weights are normalized to [1,2][1,2].

Q3(b): The two roles of the weight. The weight in target construction sets the required projection before weighting to Gk/wkG_k/w_k. The weight applied during the update scales the contribution of the selected targets. We varied these choices independently on the same 0.6B checkpoint and three saved responses covering 206 generated positions. We report the projection onto the direct teacher gradient at the starting logits, divided by the squared norm of that gradient (Table 10 and Appendix C.5).

Weight in target constructionWeight in updateMean target KLRelative teacher projection
JSDJSD0.088431.000
JSD10.088430.948 to 0.965
1JSD0.099741.036 to 1.054
110.099741.000

This analysis separates the effects on target KL and gradient projection at one checkpoint. Q3(a) reports Avg@8 from the two trained models at their respective rollout budgets.

AuthorsOpenReview ↗

Responses to Reviewer uyp6 - part two

Q4. Completion and correctness

We analyzed 13,856 saved responses from TrajPD and SOD. With completion rate CC and accuracies aca_c and aia_i among completed and unfinished responses, overall accuracy is Cac+(1−C)aiCa_c+(1-C)a_i. In the decomposition (Table 11 and Appendix C.7), the contributions are ΔC(aˉc−aˉi)\Delta C(\bar a_c-\bar a_i), CˉΔac\bar C\Delta a_c, and (1−Cˉ)Δai(1-\bar C)\Delta a_i, respectively. The bars denote averages of the two models. All values are percentage points for TrajPD minus SOD.

StudentBenchmarkCompletionAccuracy among completedAccuracy among unfinishedChange in Avg@8
0.6BAIME 2024+9.60−10.850.00−1.25
0.6BAIME 2025+9.62−10.450.00−0.83
0.6BGPQA+11.32−1.29−0.18+9.85
0.6BLiveCodeBench+7.33−11.03+6.63+2.93
1.7BAIME 2024+3.67−6.580.00−2.92
1.7BAIME 2025+4.43−5.270.00−0.83
1.7BGPQA+4.89−0.220.00+4.67
1.7BLiveCodeBench+1.45−4.39+5.58+2.64

Completion accounts for the largest positive contribution on GPQA. On 0.6B AIME 2024, completion rises from 45.83% to 76.67%, and accuracy among completed responses changes from 40.00% to 22.28%. The change in accuracy among completed responses offsets the increase in completion in all four AIME comparisons. All correct AIME responses in these records were completed.

Reviewer_uyp6OpenReview ↗

response

First, I apologize for my delayed response, and thank you for your patience and for providing the additional experimental results.

I particularly appreciate the matched-budget, three-seed comparison between response-level and per-token target construction. This experiment addresses my primary concern by providing direct evidence that the proposed joint construction can improve downstream performance, rather than merely reducing target KL.

However, my second and third concerns are not fully resolved. The additional independent runs demonstrate the stability of TrajPD itself, but the lack of corresponding multi-seed comparisons with SOD leaves uncertainty about its relative performance advantage. Moreover, the JSD weighting ablation uses different rollout budgets, making it difficult to isolate the actual contribution of dynamic weighting.

Overall, I acknowledge that the additional experiments have substantially strengthened the paper, particularly regarding its core methodological contribution. I have therefore provisionally increased my rating from 2 to 3 (Weak Accept). However, this is a marginal and tentative increase rather than a firm acceptance. Depending on the assessments of other reviewers and the subsequent discussion, I may further reconsider my rating, including potentially returning to 2.

Reviewer_E4hAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For multi-turn tool-integrated reasoning, TrajPD builds per-response distillation targets that minimize mean KL to the student, subject to retaining the teacher-gradient component of direct teacher matching. The solution is softmax(log q + λ_k d_t), with one λ_k per response. These targets are combined with a JS-based response weight and a task loss. With a Qwen3-4B teacher and 0.6B/1.7B students, TrajPD achieves the best four-benchmark mean Avg@8.

Critical strengths

1.A well-founded closed-form target with 1-D bisection, and careful statements about what the theory implies. 2.Gains over SOD (+2.67 / +0.89 mean Avg@8) and retrained component ablations. 3.A transparent appendix: contamination audit, paired confidence intervals, failure analysis.

Critical weaknesses

1.Pass@8 is non-standard. It is computed by resampling the 8 responses with replacement: with 1 correct out of 8 it gives about 65.6% instead of 100%. The main text should report the standard metric. 2.Some gains come from a baseline degrading, and statistical support is uneven. On 0.6B GPQA, SOD (17.36) falls below Vanilla (28.35), and TrajPD (27.21) is also below Vanilla. SOD wins on all AIME Avg@8 entries. The 1.7B mean gain has a CI of [−1.77, 3.34], and the intervals cover question-level variation only, not training runs. 3.The core construction's contribution is not isolated. Lower target KL holds by construction (a standard I-projection). Equal-budget comparisons are needed against per-token constraints, a gradient-matched q + β(p − q) mixture, skew-KL, and TAID. 4.The protocol and scope need disclosure. The GRPO group size is not given (TrajPD itself uses 1 rollout per prompt). Shared initialization, tuning budget, and solver overhead are not reported. "Trajectory-aware" overstates a construction that is response-level. SFT < Vanilla, the unusual evaluation sampling, and the w_k / top-16 choices are unexplained.

Questions for authors

1.Can you report standard Pass@8 and explain how method rankings change? 2.Under equal budgets and multiple seeds, does TrajPD beat per-token, gradient-matched mixture, skew-KL, and TAID targets? 3.What are the GRPO group size, the shared baseline settings, and the total compute? How are the LCB problems selected and graded?

6 comments
AuthorsOpenReview ↗

Response to Reviewer E4hA - part one

W1 and Q1. Standard Pass@8 and rankings

Thank you for asking us to report standard Pass@8 and clarify the resulting rankings. We have recomputed this metric for the original runs from the eight recorded answers to each question. For a benchmark with NN questions, let cic_i denote the number of correct answers among the eight answers to question ii. Standard Pass@8, defined in Eq. (8) of the revised manuscript, is

Pass@8⁡=100N∑i=1N1[ci>0].\operatorname{Pass@8}=\frac{100}{N}\sum_{i=1}^{N}\mathbf{1}[c_i>0].

Table 1 of the revised manuscript reports standard Pass@8 by benchmark.

Method0.6B Avg@80.6B Pass@81.7B Avg@81.7B Pass@8
Vanilla (original)12.2025.4813.5325.82
Vanilla (standard)12.2028.7813.5329.70
SFT (original)11.9329.709.3528.13
SFT (standard)11.9335.359.3534.46
GRPO (original)12.9228.0817.7734.53
GRPO (standard)12.9232.5917.7738.97
OPD (original)16.7438.3330.5451.04
OPD (standard)16.7444.3930.5455.44
EOPD (original)20.4141.9131.9550.97
EOPD (standard)20.4147.7031.9555.90
SOD (original)21.1441.9937.4059.20
SOD (standard)21.1448.0137.4064.48
TrajPD (original)23.8149.9938.2959.53
TrajPD (standard)23.8156.5738.2964.14

TrajPD has the highest mean Avg@8 at both model sizes and the highest standard Pass@8 at 0.6B. At 1.7B, SOD, TrajPD, EOPD, and OPD rank in that order on mean standard Pass@8, with 64.48%, 64.14%, 55.90%, and 55.44%, respectively.

W2. Task performance and uncertainty

Thank you for your suggestion. We will clarify the source of the gains and the variation captured by the reported intervals. On 0.6B GPQA, the gain of 9.85 points is relative to SOD. Improvements also occur on 1.7B GPQA and LiveCodeBench at both model sizes, where SOD already improves over Vanilla.

StudentBenchmarkVanillaSODTrajPD (Δ)
0.6BGPQA28.3517.3627.21 (+9.85)
0.6BAIME 20241.2518.3317.08 (−1.25)
0.6BAIME 20250.4220.0019.17 (−0.83)
0.6BLiveCodeBench18.7928.8631.79 (+2.93)
0.6BMean12.2021.1423.81 (+2.67)
1.7BGPQA27.2729.3634.03 (+4.67)
1.7BAIME 20242.9244.5841.67 (−2.92)
1.7BAIME 20252.0834.1733.33 (−0.83)
1.7BLiveCodeBench21.8641.5044.14 (+2.64)
1.7BMean13.5337.4038.29 (+0.89)

Values in parentheses are differences from SOD in percentage points. The gains on GPQA and LiveCodeBench more than offset the AIME differences, yielding mean Avg@8 improvements of 2.67 and 0.89 points at 0.6B and 1.7B, respectively.

The direct teacher comparison provides additional evidence independent of SOD's performance. With response weights retained, TrajPD achieves 23.81% mean Avg@8 at 0.6B, compared with 22.16% for direct teacher targets. The corresponding LiveCodeBench scores are 31.79% and 26.57%.

We computed the confidence intervals in Appendix G.2 by resampling evaluation questions while keeping the trained models fixed.

With the time and compute available for the rebuttal, we repeated TrajPD training at both sizes (Table 4 in the revised manuscript). Each 1.7B run exceeded the original SOD score of 37.40% mean Avg@8. The table reports the mean ± sample standard deviation across seeds 42, 2028, and 1997.

TrajPD modelMean Avg@8 (%)Mean standard Pass@8 (%)
0.6B23.83 ± 0.2957.02 ± 0.40
1.7B38.69 ± 0.3564.88 ± 1.19
AuthorsOpenReview ↗

Response to Reviewer E4hA - part two

W3 and Q2. Target construction and alternative objectives

Thank you for suggesting these comparisons to clarify the contribution of joint target construction. Different positions may require different probability changes to provide the same teacher contribution. TrajPD allows positions within a response to contribute different amounts while jointly satisfying Eq. (6). KL minimization then selects targets that stay closest to the current predictions under this requirement.

The per-token variant retains target optimization, response weights, and task scoring and applies the requirement at each position. This comparison therefore tests the benefit of constructing targets jointly. The direct teacher ablation retains response weights and replaces the constructed targets with teacher distributions.

For current probabilities qq, teacher probabilities pp, and response weight wkw_k, Mixture uses

r=q+p−qwk.r=q+\frac{p-q}{w_k}.

At the logits used to construct rr, holding the target and weight fixed gives

wk(q−r)=q−p.w_k(q-r)=q-p.

Thus, its weighted logit gradient equals the gradient from unweighted teacher targets at that point.

The four supplementary methods (per-token, Mixture, Skew KL, and TAID) use the same frozen teacher as the original 0.6B TrajPD run and match its 100 updates and 128 rollouts per update. Table 5 of the revised manuscript reports the individual runs, including the direct teacher ablation from Table 2.

0.6B targetAIME 2024AIME 2025GPQA DiamondLiveCodeBenchMean Avg@8
TrajPD, Tables 1 and 217.0819.1727.2131.7923.81
Direct teacher, Table 218.7518.3325.0026.5722.16
Mixture18.3317.5026.6425.6422.03
Skew KL [1]17.5012.9225.4424.7120.14
TAID [2]19.1718.3326.3927.9322.95

The Skew KL run uses DKL(p∥ρp+(1−ρ)q)D_{\mathrm{KL}}(p\|\rho p+(1-\rho)q) with ρ=0.1\rho=0.1 on the shared vocabulary partition. TAID adjusts the interpolation between the current model and the teacher over training.

The table reports Avg@8 percentages for individual runs.

TrajPD has the highest mean Avg@8 and LiveCodeBench score among the individual runs above.

We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.

0.6B methodMean Avg@8 (%)Mean standard Pass@8 (%)
TrajPD23.83 ± 0.2957.02 ± 0.40
per-token22.07 ± 1.0152.84 ± 2.46
Mixture21.97 ± 0.1750.00 ± 0.23

For the 0.6B ablation, we focused the three-seed comparison on TrajPD, per-token, and Mixture. As shown in the table, TrajPD achieves higher mean Avg@8 and standard Pass@8 than both alternatives.

The analysis on fixed predictions describes the effect on target distributions. With identical predictions and weights, Fig. 3 reports a median reduction in target KL of 17.35% across 91 responses relative to per-token. In the grouping comparison, mean target KL is lowest for entire interactions, followed by complete responses, fixed-length blocks, and per-token (Table 7). The training comparisons above assess the practical value of this construction through task performance.

[1] DistiLLM: Towards Streamlined Distillation for Large Language Models, ICML 2024.

[2] TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models, ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer E4hA - part three

W4 and Q3. Scope, protocol, and compute

Scope. TrajPD constructs targets jointly within each complete response (Sec. 3.2 and Fig. 2). Its positions are generated before the next external observation enters the context and share one requirement on the teacher contribution. The full interaction supplies the contexts and final task score.

Training settings. GRPO uses eight rollouts per prompt and 16 prompts per update and defaults to an SFT checkpoint. The completion records for both evaluated GRPO checkpoints report 120 training updates. The main TrajPD runs use 100 updates, with one rollout per prompt and 128 prompts per update (Appendix B.1).

Coefficient comparison and compute. Table 2 compares the main 0.6B run at α=1.0\alpha=1.0 with additional runs at α=0.1\alpha=0.1 and 0.50.5. Each additional run uses 100 updates and 128 rollouts per update, accounting for 48.61 GPU hours combined. The main 0.6B and 1.7B runs take 66.29 and 75.83 GPU hours, respectively, totaling 142.12 GPU hours. The 283.09 GPU hours cover the two main runs and five component and coefficient comparisons.

Evaluation sampling. All evaluations use eight samples per question, temperature 1.0, and a generation limit of 16,384 tokens per sample. Evaluations with tool interaction use top-p=0.6p=0.6. The archived OPD and EOPD evaluations use top-p=0.8p=0.8 and generate each of their eight samples in one turn without tools.

SFT and completion. The saved 1.7B GPQA records contain 1,584 responses per model across 198 questions. A response counts as completed when its recorded termination status is completed. SFT completes 724 responses (45.71%), of which 236 are correct (32.60%). Vanilla completes 1,535 responses (96.91%), of which 432 are correct (28.14%). All correct responses are marked as completed, giving overall Avg@8 values of 14.90% (236/1,584) for SFT and 27.27% (432/1,584) for Vanilla. The completion rates explain why the ordering among completed answers differs from the ordering over all answers.

Response weights and vocabulary. We average JSD over each response to measure prediction disagreement symmetrically and normalize the weight to [1, 2] (Sec. 4.3). We retain the 16 most probable teacher tokens and group the remaining probability mass into one entry (Sec. 3.3). Across 56 saved sets, the selected tokens carry at least 94.52% of teacher probability mass on average within each set. The cosine similarity between the logit gradients computed with 8 and 16 tokens is at least 0.9975 on the shared partition.

LiveCodeBench. We used all 175 problems in the official v6 test split of LiveCodeBench's "code_generation_lite" dataset and verified that the evaluation file contains the same problem IDs in the same order. The grader extracts the last Python code block from the final answer and marks the response correct if it passes the public input and output test retained for that problem within 30 seconds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.