- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies degradation during self-distillation for structured tool calling and proposes PRISM, which masks the forward-KL loss at token positions where a frozen self-teacher has low confidence. Confidence combines normalized entropy, top-1 probability, and the top-1 or top-2 probability margin. The standalone teacher receives the query and reference answer; an optional extension also supplies an external model's rationale. Experiments use Qwen2.5-0.5B-Instruct, a 256 training and 68 test Tool-Bench subset, three seeds, and 120 training steps. The main table reports 88.7% final action accuracy for standalone PRISM versus 85.3% for uniform SDPO, accompanied by token-category and gradient diagnostics.
Critical strengths
-
Position-level selection is a useful design for structured outputs that mix predictable schema tokens with variable arguments. The proposed mask is simple, requires no external teacher queries in the standalone setting, and retains confident argument values instead of deleting the whole category.
-
The paper attempts substantial controls, including random masking, learning-rate scaling, single-score ablations, category masking, and privileged or non-privileged comparisons. The supplementary train/evaluation files and deterministic scorer make the task definition inspectable. Reporting complete-call matching alongside action selection is valuable, although the results and control implementations need the clarification below.
Critical weaknesses
-
The supplementary implementation is materially different from the reported protocol. Algorithm 1 samples a current-student rollout each iteration; the engine referenced by the advertised reproduction entry point instead uses frozen-teacher generations cached once and repeatedly samples those cached continuations. This changes the on-policy distillation question. The scorer compares parsed, normalized tool-call content and does not execute tools, so the claimed independent execution-based validation is unsupported by this supplied evaluator. These may be release discrepancies; they do not prove which code produced the paper's numbers. However, without a mapping to the actual runs, I cannot establish that the experiments evaluate the stated method and controls.
-
The mechanistic and superiority claims exceed the evidence. High entropy does not itself imply a harmful or high-variance exact KL gradient q-p; the static positive cosine 0.1503 and confidence-defined gradient partition do not establish subsequent structural damage. Masking removes selected contributions by definition, rather than proving an orthogonal parameter-space projection or retained-target correctness. The bounded logit-gradient argument also applies to one-hot cross-entropy. Empirically, Table 2's SDPO already retains 97.2% of peak accuracy versus PRISM's 97.8%; PRISM's exact match is 45.1%, below SFT's 47.1% and sample-drop's 49.5%, despite Section 5.3 claiming the highest exact match. The hard/soft comparison also changes teacher context; within the privileged setting, soft weighting scores 83.3% action accuracy versus hard masking's 80.9%.
Questions for authors
-
Which code and commands produced each principal result? Please reconcile the online student-rollout algorithm with the supplied offline teacher-cache engine, and provide the per-row configurations/run records needed to verify the comparisons.
-
Please reconcile category-mask exact match of 0% in Table 5 versus 32.8% in Tables 2/7. Also, Appendix B.2 attributes the 21.1%-21.6% results to shuffled token masking, while Table 2 labels them shuffled explanations. Which intervention generated those results?
-
Is hard masking better than soft weighting with identical teacher context and training settings? For the category control, were labels produced by the supplied token-string/keyword heuristic (annotate_token_categories), or a schema-aware annotation that actually preserves all tool names and keys? This matters for interpreting its failure as evidence against semantic-role masking.