acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Mitigating Distillation Collapse in Tool-Use LLMs via Position-Level Target Reliability Masking

Abstract

Self-Distillation Policy Optimization (SDPO) offers an appealing paradigm for aligning lightweight open-source language models (<=1B parameters) by matching the token-level distribution of a same-family self-teacher. However, when applied to structured agentic tasks such as tool use, SDPO exhibits a pervasive failure mode: late-training collapse, where policy capability peaks early and degrades as distillation continues. In this work, we uncover a critical pathology behind this collapse: structured tool-calling sequences feature an intrinsic non-homogeneous supervision topology, where closed-set syntax and tool identifiers enjoy near-certain teacher guidance (p approx 1.0, H approx 0), whereas open-vocabulary argument values suffer from diffuse predictive uncertainty (sigma_H = 1.4091, with entropy spikes exceeding 7.5 nats). Diagnostic probes on representation gradients reveal that uncalibrated argument positions exhibit large gradient dispersion (3.51x norm dominance) and lie at wide subspace angles (cos approx +0.1503) relative to structural updates, introducing transverse drift into shared representations during autoregressive training. Furthermore, we show that conventional sample-level filtering is structurally insufficient when reliable and unreliable supervision co-exist within the same sequence (100% co-occurrence in our diagnostic testbed), forcing an intractable compromise between structural starvation and parameter noise leakage. To resolve this dilemma, we introduce PRISM (Position-level Reliability Insulation and Selective Masking), a lightweight framework that operates directly at the position level without requiring any external teacher models: By evaluating continuous predictive confidence on the self-teacher distribution, PRISM constructs a dynamic binary gradient gate that surgically excises volatile, high-variance parameter updates while preserving the large majority of structural supervision: at tau_pos = 0.5 it retains 97.05% of STRUCT syntax delimiters, 96.88% of TOOL_NAME identifiers, and 88.17% of ARG_NAME schema keys. On controlled 120-step multi-seed evaluations across 14 competitive methods and ablation paradigms on real 4xH100 hardware, standalone PRISM achieves the global highest final action accuracy of 88.7% (+3.4 pp over vanilla SDPO, +5.9 pp over sample-level filtering, +15.7 pp over on-policy GRPO) with an outstanding 97.8% peak capability retention, zero external teacher queries, and smooth parameter convergence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
PRISM, Self-Distillation, Tool-Use LLMs, Policy Optimization, Position-Level Masking, Late-Training Collapse, Agentic Alignment, Knowledge Distillation, Near-Orthogonal Gradient Decoupling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iti5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies degradation during self-distillation for structured tool calling and proposes PRISM, which masks the forward-KL loss at token positions where a frozen self-teacher has low confidence. Confidence combines normalized entropy, top-1 probability, and the top-1 or top-2 probability margin. The standalone teacher receives the query and reference answer; an optional extension also supplies an external model's rationale. Experiments use Qwen2.5-0.5B-Instruct, a 256 training and 68 test Tool-Bench subset, three seeds, and 120 training steps. The main table reports 88.7% final action accuracy for standalone PRISM versus 85.3% for uniform SDPO, accompanied by token-category and gradient diagnostics.

Critical strengths
  1. Position-level selection is a useful design for structured outputs that mix predictable schema tokens with variable arguments. The proposed mask is simple, requires no external teacher queries in the standalone setting, and retains confident argument values instead of deleting the whole category.

  2. The paper attempts substantial controls, including random masking, learning-rate scaling, single-score ablations, category masking, and privileged or non-privileged comparisons. The supplementary train/evaluation files and deterministic scorer make the task definition inspectable. Reporting complete-call matching alongside action selection is valuable, although the results and control implementations need the clarification below.

Critical weaknesses
  1. The supplementary implementation is materially different from the reported protocol. Algorithm 1 samples a current-student rollout each iteration; the engine referenced by the advertised reproduction entry point instead uses frozen-teacher generations cached once and repeatedly samples those cached continuations. This changes the on-policy distillation question. The scorer compares parsed, normalized tool-call content and does not execute tools, so the claimed independent execution-based validation is unsupported by this supplied evaluator. These may be release discrepancies; they do not prove which code produced the paper's numbers. However, without a mapping to the actual runs, I cannot establish that the experiments evaluate the stated method and controls.

  2. The mechanistic and superiority claims exceed the evidence. High entropy does not itself imply a harmful or high-variance exact KL gradient q-p; the static positive cosine 0.1503 and confidence-defined gradient partition do not establish subsequent structural damage. Masking removes selected contributions by definition, rather than proving an orthogonal parameter-space projection or retained-target correctness. The bounded logit-gradient argument also applies to one-hot cross-entropy. Empirically, Table 2's SDPO already retains 97.2% of peak accuracy versus PRISM's 97.8%; PRISM's exact match is 45.1%, below SFT's 47.1% and sample-drop's 49.5%, despite Section 5.3 claiming the highest exact match. The hard/soft comparison also changes teacher context; within the privileged setting, soft weighting scores 83.3% action accuracy versus hard masking's 80.9%.

Questions for authors
  1. Which code and commands produced each principal result? Please reconcile the online student-rollout algorithm with the supplied offline teacher-cache engine, and provide the per-row configurations/run records needed to verify the comparisons.

  2. Please reconcile category-mask exact match of 0% in Table 5 versus 32.8% in Tables 2/7. Also, Appendix B.2 attributes the 21.1%-21.6% results to shuffled token masking, while Table 2 labels them shuffled explanations. Which intervention generated those results?

  3. Is hard masking better than soft weighting with identical teacher context and training settings? For the category control, were labels produced by the supplied token-string/keyword heuristic (annotate_token_categories), or a schema-aware annotation that actually preserves all tool names and keys? This matters for interpreting its failure as evidence against semantic-role masking.

Reviewer_DydmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates late-training degradation in self-distillation for structured tool calling. It proposes PRISM, which combines teacher predictive entropy, top-1 probability, and the top-1/top-2 probability margin into a position-wise confidence score, then masks the forward-KL loss at positions below a threshold. The standalone version uses a frozen same-model teacher conditioned on the input and reference answer; an optional extension supplies the teacher with explanations from an external model. Experiments with Qwen2.5-0.5B-Instruct on a 256 training/68 test Tool-Bench subset report higher final action accuracy than the uniform-distillation baseline. Token-category statistics and gradient probes are used to motivate and explain the method.

Critical strengths
  1. Structured tool calls combine schema-constrained decisions with open-ended argument values, making position-dependent supervision a reasonable design choice for this practically relevant problem. PRISM is straightforward to implement and does not require an external rationale-generating teacher in its standalone setting. This makes the approach potentially useful for small tool-use models.

  2. In the main configuration, table 2 reports a promising improvement over uniform distillation, from 85.3% to 88.7% in final action accuracy and from 43.1% to 45.1% in Exact Match. The evaluation also uses three seeds. And the controls for random masking, static category masking, and individual confidence components are worthwhile attempts to distinguish selective supervision from generic pruning.

Critical weaknesses
  1. In table 2, PRISM's Exact Match of 45.1% is below SFT at 47.1% and sample-drop at 49.5%, contrary to the highest-EM statement in Sec 5.3. In the central comparison, the drop from peak to final action accuracy is 2.5 points for SDPO and 2.0 for PRISM, which supports an accuracy gain more than collapse mitigation. The hard vs. soft comparison is also confounded: the soft baseline uses privileged explanations while standalone PRISM does not. With privileged context in both rows, soft weighting actually beats hard masking, 83.3% vs. 80.9% on action accuracy, with equal EM at 40.7%. Appendix B.2 attributes the 21.1% & 21.6% results to shuffled token masking, while table 2 and appendix B.8 attribute them to shuffled explanations; table 5 reports 0% EM for category masking vs. 32.8% in tables 2 and 7. These discrepancies need an explicit mapping from configuration to result, and they limit confidence in the principal comparisons given the small 68-instance test set and the insufficiently specified scoring protocol.

  2. The gradient groups are defined using the same confidence score used for masking, so eliminating the group labeled unreliable does not independently show that harmful supervision was removed. With fixed teacher and student distributions, full forward KL has the deterministic logit gradient q-p at T=1; high teacher entropy alone does not establish high stochastic gradient variance. A positive cosine of 0.1503 between aggregate gradients on the output projection likewise does not by itself establish destructive interference or explain the later collapse; the authors acknowledge this in appendix B.2 and B.10, but the main text still makes stronger causal claims. Token-loss masking also does not generally act as an orthogonal projection in shared parameter space, as suggested by eq. 19 and fig 7d. This gap matters because position-selective distillation already exists, for example in TSD-KD, and entropy-dependent objectives are explored in Entropy-Aware OPD. The frozen-teacher forward-KL baseline also differs from the evolving, regularized teacher and the stability-oriented objectives in the original SDPO. The contribution therefore needs stronger evidence beyond the masking heuristic and this particular baseline configuration.

Questions for authors
  1. Can the authors reconcile the shuffled-token vs. shuffled-explanation attribution of the 21.1% & 21.6% results and the category-mask EM discrepancy of 0% vs. 32.8%, identifying the exact configurations and per-seed runs behind these numbers? The text should state which results are the authoritative controlled comparisons.

  2. How are action accuracy, key accuracy, and EM defined and computed, and does EM compare text, canonicalized calls, or execution outcomes? The selection and split of the 256/68 subset need detail, as does whether thresholds and checkpoints were chosen on separate validation data. Paired prediction-level uncertainty for the main method differences would also be useful.

  3. Under the same standalone teacher context, loss normalization, and training protocol, does hard masking outperform soft weighting, and does the advantage persist against an SDPO baseline with the original work's teacher regularization and stability-oriented divergence choices? Please distinguish results already available from proposed additional experiments.

  4. What evidence independent of the confidence-based partition establishes that the removed supervision is harmful rather than merely uncertain? Checkpoint-level structural-margin or gradient diagnostics would help support the claimed temporal link to collapse, and the parameter-space projection interpretation in eq. 19 also needs clarification.

Reviewer_amz2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the late‑training capability collapse problem existing in Self‑Distillation Policy Optimization (SDPO). The authors attribute training collapse to substantial heterogeneity of supervision reliability across different token positions within tool‑call generation sequences. They argue that continual accumulation of these noisy gradients induces transverse drift of model representations, which ultimately degrades tool‑calling performance. The authors propose PRISM, a lightweight distillation framework for Position‑level Reliability Insulation and Selective Masking, to mitigate late‑training collapse at the token granularity. The authors conduct a set of experiments to demonstrate the effectiveness of their method.

Critical strengths

This paper examines the late‑training capability collapse problem at a finer granularity and conducts analytical experiments to validate the motivations behind the authors’ proposed method.

Critical weaknesses

The paper is suspected to have been submitted with only AI-generated content. The authors did not carefully revise and polish the paper. Here are several pieces of evidence:

  1. The overall writing style of the manuscript reads more like a technical report rather than a research paper. In particular, the abstract spans nearly one page, and the introduction suffers from disorganized formatting.
  2. The paper is interspersed with excessive parenthetical remarks, including section cross‑references and scattered numerical annotations.
  3. Nearly all figures exhibit distortion, making numerical values and axes hardly legible.
  4. Text such as "Reviewers may reasonably question…" (in Appendix B.6, line 914) is present in the manuscript. The following points are other weaknesses of the content:
  5. Explanations for the design choices of the proposed method are missing. For instance, why are these three terms combined in Section 4.1? Moreover, why is the top-1 probability term added separately after top-1 information has already been incorporated via the margin difference? Additionally, have the authors considered the sampling strategy of the teacher model?
  6. After accounting for per-token differences, do differences between individual trajectories/samples require no consideration at all?
  7. Why are results from a specific training step selected? For example, Table 2’s caption reports results taken at step 119. However, the authors claimed that all methods are evaluated across 120 training steps.
Questions for authors

See in Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.