- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TrajPD for distilling tool-integrated reasoning into small language models. It constructs targets jointly within each response, minimizing their KL divergence from the student while retaining a specified component of the teacher gradient. The targets are used with response weights and task rewards to update the student. Experiments with Qwen3-0.6B and Qwen3-1.7B report higher average performance across four benchmarks, with the clearest gains over SOD on GPQA. The authors also analyze response completion and the geometry of the constructed targets.
Critical strengths
-
Jointly constructing targets for a complete response is a reasonable design for tool use. The optimization problem is clearly defined and can be solved through a single scalar search per response.
-
The analysis of completion rates is helpful for understanding the GPQA gains. The appendix also provides uncertainty estimates and a training–evaluation overlap check.
-
The derivation is careful about the scope of the gradient identity. In particular, it acknowledges the difference between logit-space properties and updates to shared model parameters.
Critical weaknesses
-
The improvement over SOD is uneven. Mean Avg@8 increases by 2.67 points at 0.6B and 0.89 at 1.7B, while both AIME scores decrease at both sizes. Appendix G.2 gives a 95% interval of [-1.77, 3.34] for the 1.7B average gain; the LiveCodeBench intervals also include zero. On GPQA, much of the gain appears to come from completing more responses. This is useful, but I would like to see independent training runs and controls for stopping behavior before drawing a broad conclusion about improved reasoning.
-
Does the gain specifically require joint target construction across a response? Table 2 ablates the main components, but the token-wise alternative is compared mainly through target KL on recorded predictions. Lower KL is what TrajPD is designed to optimize. Training the token-wise alternative under the same settings would provide a more direct test of whether response-level construction improves the policy. A comparison with direct teacher matching at matched gradient scale would also be informative.
-
The Pass@8 definition needs to be stated in the main text. Appendix C.2 explains that it uses 1,000 resamples with replacement from eight recorded answers. For a question with one correct answer, this gives an expectation of 1-(7/8)^8, or about 0.656, although the observed set of eight already contains a correct answer. Please report the observed eight-sample success rate and label the resampling result separately, so that readers can compare it with other work.
-
I am still unsure why minimum target KL is the right criterion for the actual update. Appendix A.2.7 shows that the constructed logit gradient can have a larger norm than direct matching, and its projection property need not survive the model Jacobian and optimizer. The paper acknowledges this limitation. Measuring parameter-update alignment and teacher loss after the update would help establish the connection between the target construction and training behavior.
Questions for authors
see Critical Weaknesses
3 comments
Response to Reviewer coHi - part two
W3. Standard Pass@8. Thank you for asking us to report standard Pass@8 explicitly. Standard Pass@8 counts a question as solved if at least one of its eight recorded answers is correct. Equation (8) defines this metric, and Table 1 reports it by benchmark for the original runs.
| Method | 0.6B Avg@8 | 0.6B Pass@8 | 1.7B Avg@8 | 1.7B Pass@8 |
|---|---|---|---|---|
| Vanilla (old) | 12.20 | 25.48 | 13.53 | 25.82 |
| Vanilla (standard) | 12.20 | 28.78 | 13.53 | 29.70 |
| SFT (old) | 11.93 | 29.70 | 9.35 | 28.13 |
| SFT (standard) | 11.93 | 35.35 | 9.35 | 34.46 |
| GRPO (old) | 12.92 | 28.08 | 17.77 | 34.53 |
| GRPO (standard) | 12.92 | 32.59 | 17.77 | 38.97 |
| OPD (old) | 16.74 | 38.33 | 30.54 | 51.04 |
| OPD (standard) | 16.74 | 44.39 | 30.54 | 55.44 |
| EOPD (old) | 20.41 | 41.91 | 31.95 | 50.97 |
| EOPD (standard) | 20.41 | 47.70 | 31.95 | 55.90 |
| SOD (old) | 21.14 | 41.99 | 37.40 | 59.20 |
| SOD (standard) | 21.14 | 48.01 | 37.40 | 64.48 |
| TrajPD (old) | 23.81 | 49.99 | 38.29 | 59.53 |
| TrajPD (standard) | 23.81 | 56.57 | 38.29 | 64.14 |
W4. Actual parameter updates. Thank you for suggesting measurements of the actual parameter update. We minimize target KL while retaining the prescribed teacher contribution. This keeps the targets as close as possible to the current predictions. We then measure the actual parameter update and teacher KL after one AdamW step.
We compare TrajPD, Mixture, and per-token using the 0.6B model (Table 6 and Appendix C.6). Update cosine measures the alignment of the parameter update with the negative gradient of the teacher loss. Teacher KL is evaluated before and after the update, with an initial value of 0.13280.
| Target | Mean target KL | Actual update norm | Update cosine | Teacher KL after update | Teacher KL reduction (%) |
|---|---|---|---|---|---|
| TrajPD | 0.09 | 0.06267 | 0.30348 | 0.06262 | 52.85 |
| Mixture | 0.12 | 0.06277 | 0.31565 | 0.05571 | 58.05 |
| per-token | 0.10 | 0.06273 | 0.31317 | 0.05714 | 56.97 |
TrajPD has the lowest target KL and an update norm similar to the alternatives. Its actual parameter update has a positive cosine of 0.30348 with the teacher descent direction, and teacher KL decreases from 0.13280 to 0.06262. These measurements show positive alignment and a 52.85% reduction in teacher KL after the actual optimizer step on these saved responses.
Dear Reviewer coHi
Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.
If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.
Sincerely,
TrajPD authors
Response to Reviewer coHi - part one
Thank you for taking the time to read our manuscript closely and for sharing your valuable and constructive comments. We particularly appreciate your suggestions to examine completion and stopping behavior and the parameter updates produced by the proposed target construction. We will address each of your comments below.
W1. Completion, stopping behavior, and independent training
Thank you for your valuable suggestions on training variability and controls for stopping behavior. We report results from repeated training and compare the number of responses that reach the per-turn limit and mean Avg@8 under both settings.
W1(a). Independent training.
We ran TrajPD three times at each size. At 1.7B, each run scored above the original SOD result of 37.40% mean Avg@8.
We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.
W1(b). Stopping behavior. Thank you for this suggestion. We increased the generation limit per turn from 8,192 to 12,288 tokens for both TrajPD models while keeping the total generation budget fixed. Table 8 of the revised manuscript reports the number of responses that reach the per-turn limit and mean Avg@8 under each setting.
The higher limit reduced the number of responses reaching the turn limit. The original setting retained higher mean Avg@8 at both model sizes, supporting our choice of the original generation limit.
W2(a). Training with per-token
Both 0.6B methods use 100 updates and 128 rollouts per update. We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.
W2(b). Mixture and the gradient reference. Thank you for suggesting a comparison with direct teacher targets. For current probabilities q, teacher probabilities p, and response weight wk, we use the Mixture target in Eq. (20)
r=q+wkp−q.
At the logits used to construct r, holding the target and weight fixed gives the identity underlying Eq. (21)
wk(q−r)=q−p.
The weighted Mixture gradient therefore matches the gradient from the unweighted teacher target at that point. The benchmark scores below are from seed 42.
We report the mean ± sample standard deviation across seeds 42, 2028, and 1997.
TrajPD achieves higher mean Avg@8 and standard Pass@8 across the three seeds. On LiveCodeBench v6, its seed 42 run scores 31.79% Avg@8, compared with 25.64% for Mixture.