acceptodds
Withdrawn from ICLR 2027

Distilling Productive Disagreement: Collision-Aware On-Policy Distillation for Self-Correcting Language Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Teacher disagreement is useful for on-policy distillation only when it exposes a decision-sensitive alternative that can be verified and acted on from the student's current state. We study this conditional learning signal and introduce Collision-Aware On-Policy Distillation (C-OPD). For each student trajectory, an axiom teacher identifies implicit assumptions, a counterfactual teacher minimally inverts high-risk assumptions, and task-native verification adjudicates the resulting branches. A Productive Collision Score (PCS) combines disagreement, counterfactual utility, evidence consistency, recoverability, and training-trajectory reward improvement, then attributes additional distillation weight to an intervention-linked reasoning span. Across three seeds on SWE-bench Verified, BFCL v3.0.1, MATH-500, and HotpotQA v1.1, C-OPD reaches 49.6±0.4% macro-average success, compared with 41.8±0.4% for standard on-policy distillation and 38.7±0.4% for response-level knowledge distillation under the same 800M teacher-token budget per seed. On SWE-bench Verified, resolved instances increase from 32.4±0.4% to 38.9±0.3%. On adversarial tool observations, unsupported continuation falls by 28.4%, while recovery after an incorrect intermediate decision rises from 46.1% to 61.3%. Weighting only disagreement, evidence consistency, or future reward underperforms full PCS by 5.2, 4.4, and 4.8 points, respectively, showing that future reward alone does not explain the gain. Retraining without collision-generated supervision absent from all pre-collision teacher responses lowers success from 49.6% to 43.9%, accounting for 72.6% of the observed improvement over standard OPD under this controlled removal-and-retraining intervention. The results show that verified, recoverable disagreement supplies supervision that raw disagreement and best-of-teacher selection do not capture.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Knowledge Distillation, Language Agents, Self-Correction, Teacher Disagreement, Counterfactual Reasoning, Process Supervision, Tool-Augmented Language Models.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8ddBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper distills a student from on-policy trajectories by turning one assumption into a second branch. The prefix is held fixed, and task checks assign a utility VV to each branch. The higher-utility branch is the preferred branch when it is supported and the two branches are not equivalent. The paper calls a collision productive when the branches disagree at a localized decision, the alternative can be tested, and the verified branch improves downstream reward without restarting the episode. A Productive Collision Score then combines disagreement, the utility gap, evidence, recoverability, and reward-to-go improvement, and that score scales the token weight on the intervention span and the preference term in Equation 9. The student is Qwen2.5-7B-Instruct. Every teacher role is Qwen2.5-72B-Instruct under a different prompt. The tasks are SWE-bench Verified, BFCL, MATH-500, and HotpotQA, and at test time the student acts alone.

Critical strengths
  1. A counterfactual branch must change one assumption and keep the prefix, and it is judged by unit tests, tool returns, SymPy, or retrieved evidence rather than by another language model’s preference. The training, calibration, and test splits are stated to be disjoint, and at test time the student acts alone. This makes the separation between training-time checks and test-time behavior something a reader can verify.

  2. The evaluation covers code repair, tool use, mathematics, and evidence-seeking question answering, and three seeds are reported. A reader can therefore see whether the same filter is applied across these domains.

Critical weaknesses
  1. Contribution (i) presents productive disagreement as the supervision source: a teacher conflict is useful when it reveals a verifiable, recoverable alternative at a student-visited decision. The paper’s definition of that event is localized disagreement, a testable alternative, and a verified branch that improves downstream reward without restarting the episode. The preferred branch in Equations 8 and 9 is the higher-VV branch, after unsupported collisions are discarded and equivalent branches receive no enhancement. Appendix F.1 states that DcD_c identifies a collision without deciding that it is useful, and Table 16 states that DcD_c does not establish that either branch is correct. DcD_c then enters the percentile sum. UcU_c is the absolute gap in VV. EcE_c is support for the preferred branch and is also the argument of the evidence gate. QcQ_c is recoverability of that preferred correction. FcF_c is the reward-to-go change of bc∗=arg⁡max⁡Vb_c^*=\arg\max V, which is what the definition of FcF_c specifies. The sum of these terms with the frozen coefficients is a scale on the branch the definition has already kept. Section 5.4 reports Spearman ρ=.58\rho=.58 with correction success and AUC .76.76 on these collisions. Those figures associate this scale with the outcome of the same pipeline.

  2. Both branches are generations of one Qwen2.5-72B-Instruct. The original branch is conditioned on the extracted assumption, and the alternative is conditioned on its inversion. The axiom, counterfactual, and verification roles are prompts to that model. Unit tests, tool returns, SymPy, and retrieved passages then choose between the two texts, and a text that fails the native record can be rejected. The assumption that is inverted, and the span that receives the weight, are named by the same model. Table 21 reports 50.2% when three hosted teachers fill the same roles and 49.6% for the shared model. The paper uses this comparison to show that the result does not require three hosted models. The control changes who fills the roles. The branch that is kept remains the higher-VV branch under the same support and equivalence filters.

  3. PCS converts (D,U,E,Q,F)(D,U,E,Q,F) to percentiles on the calibration set and adds them with (.15,.15,.25,.20,.25)(.15,.15,.25,.20,.25). The evidence gate 0.700.70, the retention threshold 0.620.62, λ=2\lambda=2, and wmax⁡=4w_{\max}=4 are frozen from that set. A new trajectory supplies inputs to this function, and the coefficients stay fixed. Table 2 removes components and retrains. Each row is another stipulated function of the same records. Fitting a new coefficient vector on the same calibration split replaces one frozen vector with another. Section 5.4 computes its correlation from the frozen score, so the association with correction success leaves the coefficients as they were set on the calibration split.

  4. The span ScS_c is stipulated as the tokens that express the assumption, select the affected action, and carry it forward until contradictory feedback. The paper states that this attribution is a structured heuristic anchored by the inversion and the earliest discriminating evidence, and that it is not a causal identification. The factor κ\kappa equals 1 inside ScS_c and decays with distance outside it. The elevated weight is the weight this span and this decay assign. The pipeline records which tokens express the unsupported inference and the corrective action, and it answers that question from stored fields. Those fields are the span this rule has already named. Table 18 reports a 500-collision audit on productive-collision detection, evidence consistency, recoverability, and preferred-branch correctness. Those four criteria judge the retained collision against the majority human judgment. The table contains no comparison between the elevated tokens and a set defined independently of ScS_c and κ\kappa.

Questions for authors
  1. The paper defines a productive collision as disagreement at a localized student decision, a testable alternative, and a verified branch that improves downstream reward without restarting the episode. UcU_c, EcE_c, QcQ_c, and FcF_c are functions of the branch VV has named, and DcD_c is the separation between the two texts. Which term in PCS is a measurement of productivity that this definition has not already fixed?

  2. Both branches are generations of one Qwen2.5-72B-Instruct, one conditioned on the extracted assumption and one on its inversion. Table 21 fills the same roles with three hosted teachers, keeps the same retention rule, and reports 50.2% against 49.6% for the shared model. Which recorded quantity in that control keeps a generation by a rule other than VV, the support filter, and the equivalence filter?

  3. Which observation, independent of correction success and of the Table 2 retrainings, rejects the frozen coefficients (.15,.15,.25,.20,.25)(.15,.15,.25,.20,.25), the evidence gate 0.700.70, or the retention threshold 0.620.62? These values are currently fixed on the calibration set.

  4. The pipeline answers, from stored fields, which tokens express the unsupported inference and the corrective action. Table 18 reports human agreement on productive-collision detection, evidence consistency, recoverability, and preferred-branch correctness. Which recorded field compares those tokens with a set not defined by ScS_c and κ\kappa?

Reviewer_46fGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Collision-Aware On-Policy Distillation (C-OPD) algorithm. Instead of a naive teacher (dis)agreement in OPD, the C-OPD teacher (1) evaluates the assumptions of the student, (2) inverts the high-risk assumptions, and (3) verifies the edited trajectory. Then, the distillation loss includes a productive collision score (PCS) combining disagreement, counterfactual utility, evidence consistency, recoverability, and future reward improvement. The loss objective combines PCS-weighted KL distillation on verified preferred branches, a DPO-style preference loss between competing branches, and KL regularization to limit policy drift. Evaluation across various benchmark shows the improvement over standard OPD.

Critical strengths
  1. The paper presents a new direction for on-policy distillation by leveraging structured teacher disagreement to construct and learn from verified counterfactual trajectories.

  2. The evaluation shows that C-OPD can improve naive baselines across four benchmarks.

Critical weaknesses
  1. The paper's presentation and clarity need substantial improvement.

    1. Figures 1 and 2 consist of small font size and dense textual information, limiting readability.
    2. The paper introduces numerous concepts and components (e.g., axiom extraction, counterfactual inversion, verification, PCS, and token-level attribution) without clear motivations, details, and contributions.
    3. The explanation of the training objective (Section 4.3) is insufficient. It is unclear how verified counterfactual trajectories are incorporated into KL distillation.
  2. The proposed method is overly complex, raising concerns about its practicality and the necessity of its components.

    1. PCS combines five components, but the necessity of each component is not sufficiently justified.
    2. The training objective combines PCS-weighted KL distillation, a DPO-style preference loss, and KL regularization. However, Table 2 is insufficient to explain the importance of each component.
    3. The contribution of PCS and counterfactual trajectory generation/verification is unclear. The paper does not include an experimental comparison with standard KL distillation based on counterfactual trajectories.
Questions for authors

Please check the weaknesses above.

Reviewer_wPDYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Collision-Aware On-Policy Distillation (C-OPD), which constructs assumption-level counterfactual branches from student-visited states, verifies competing continuations using task-native feedback, and assigns localized supervision through a Productive Collision Score (PCS). The method combines weighted token-level distillation with a branch-preference objective. Across SWE-bench Verified, BFCL, MATH-500, and HotpotQA, the paper reports a 7.8-point macro-average improvement over standard OPD under matched teacher-token budgets, together with additional ablations, compute controls, and recovery analyses. However, several evaluation-reporting inconsistencies and insufficient mechanism isolation weaken confidence in the central claims.

Critical strengths
  1. The paper tackles an important problem by explicitly distinguishing verifiable, recoverable decision conflicts from superficial teacher disagreement.

  2. The evaluation spans four domains and includes several useful ablations and compute-controlled experiments. The improvement under matched GPU-hours suggests that additional verification compute does not fully explain the reported gains.

Critical weaknesses
  1. Evaluation inconsistencies. Some per-seed MATH-500 accuracies (e.g., 57.1%) are incompatible with a single binary-scored 500-problem evaluation. The BFCL test-set composition and unusually identical seed-wise fluctuations across several ablations also require clarification.

  2. Insufficient mechanism isolation. C-OPD combines counterfactual generation, verification, data selection, localized weighting, and preference optimization. The experiments do not adequately distinguish the proposed collision mechanism from simpler verifier-filtered supervision.

  3. Limited evidence for self-correction. Recovery rates are measured on method-specific error states rather than matched failures. The qualitative examples are also schematic rather than actual trajectories, limiting validation of the claimed correction mechanism.

  4. Limited novelty differentiation. Closely related work has already explored step-wise supervision and feedback-guided intervention in OPD [1,2]. Although DemoPSD and CROP are compared, the reproduced results lack detailed implementation and per-run evidence.

  5. Presentation issues. Figures 1 and 2 contain excessively small text and labels that are difficult to read at normal viewing size. This substantially reduces the clarity of the main method illustrations.

[1] SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

[2] SOD: Step-wise On-policy Distillation for Small Language Model Agents

Questions for authors

Can the authors clarify how the per-seed MATH-500 accuracies were calculated and provide the exact composition of the 1,200 BFCL evaluation episodes? Per-instance evaluation records would help resolve these reporting concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.