acceptodds
Under review as a conference paper at ICLR 2027

Progress-Aware On-Policy Distillation for Multimodal Agents

Abstract

Standard on-policy distillation (OPD) applies a uniform objective across all states. In long-horizon multimodal tasks, however, local alignment with a teacher often decouples from actual student progress. We observe that this uniform supervision frequently traps compact agents in low-progress continuation—they repeatedly invoke tools or search without meaningfully advancing toward the final answer. We present -OPD, a simple yet effective progress-aware distillation method. Our core idea is to condition teacher supervision on the student's realized progress. We quantify this progress via the step-wise change () in answer readinessin \emph–how ready the student is to answer correctly at the current step. -OPD dynamically adapts the learning objective: it applies reverse KL for interactions that actively drive progress, while switching to forward KL for low-progress steps to explore teacher-supported alternatives. Furthermore, for verified-success trajectories, it naturally bypasses redundant interactions where readiness is already saturated. Importantly, -OPD introduces zero inference overhead. It operates entirely on student-generated trajectories without requiring additional rollouts. Evaluated on the OmniGAIA benchmark, -OPD improves the accuracy of a Qwen3.5-4B student from 40.8% under standard OPD to 45.4%, while reducing the budget-exhaustion rate by 35.9%. Ablation studies show that selecting where to apply forward KL matters beyond its overall frequency, while readiness-based masking provides further gains. Code is provided in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agent, On-policy distillation, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xbQJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Distilling long-horizon task performance from a strong teacher to a weaker student multimodal agent is particularly complex. Decision-making at any given step relies not only on current input, but also on the full history of past steps and anticipated future trajectories. A behavioral analysis of teacher-student distillation highlights these challenges: SFT on teacher trajectories leads to premature termination because the student inherits stopping behaviors without learning how to finalize a trajectory. Conversely, OPD addresses premature stopping but introduces another issue, unnecessarily prolonging interaction trajectories without progressing toward the answer. To address these limitations, the authors propose Δ\Delta-OPD. This approach improves upon standard OPD by applying supervisory signals only when the student’s interactions demonstrate meaningful progress toward the answer with reduced redundancy. Low progress or redundancy is probabilistically quantified via variations in answer-readiness, the likelihood of the student generating a correct answer immediately. Experiments across complex multimodal benchmarks demonstrate that Δ\Delta-OPD narrows the performance gap between small multimodal agents and proprietary models, outperforming open-source baselines of similar scale. Furthermore, direct comparisons with existing OPD methods show that Δ\Delta-OPD achieves the best average performance across multiple benchmarks.

Critical strengths
  • The proposed method yields meaningful performance improvements over baseline OPD methods and existing VLMs. Furthermore, the ablation study clearly demonstrates the specific contribution of each component technique.

  • The analysis of pathologies in SFT and OPD training, which forms the foundation of the proposed technique, is both clear and compellingly motivated.

Critical weaknesses
  • While the problem formulation targets multimodal agents, the proposed technique is not specific to multimodality in any respect and can be applied to text-only agents.

  • The paper lacks an in-depth analysis of how the key indicators are determined, as well as empirical verification of whether these indicators accurately detect and mitigate the intended pathologies.

Questions for authors
  • How were the specific thresholds and formulations for "answer-readiness" determined? Did you explore alternative indicators during development?

  • Have you conducted behavioral audits or qualitative case studies to verify that low-progress interactions identified by the model actually correspond to human-judged stalling or repetitive reasoning loops?

  • Is it possible to provide a baseline comparison against alternative supervision filters or progress-gating heuristics to demonstrate that the proposed Δ\Delta-OPD formulation is optimal?

  • The related work section seems overly brief and lacks depth. Is it sufficient to omit a broader literature survey and conceptual comparison across general LLM/VLM knowledge distillation or trajectory-level imitation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xbQJ (Part 1/3)

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. This part addresses Q1: the formulation, threshold selection, and alternative indicators. Responses to Q2–Q4 follow in subsequent comments.

Q1: How were the specific thresholds and formulations for "answer-readiness" determined? Did you explore alternative indicators during development?

A1: We fixed τ = 0.05 through offline calibration before Δ-OPD training and examined alternative progress indicators and gating strategies. Our empirical comparisons favor readiness-change magnitude over the alternatives considered.

Formulation. Answer readiness is the student's mean per-token log-likelihood of the reference answer at the current state (Eq. 2). It directly measures how strongly the current state supports the reference answer, reflects the student's own answer state rather than its local agreement with the teacher (Section 2.2), requires no learned critic, and is length-normalized across answers. We use its change rather than its absolute level because the latter is task- and state-dependent, whereas the change captures the effect of an individual interaction.

Magnitude rather than sign. Δ-OPD switches to forward KL to help the student escape stalled, self-reinforcing interaction patterns (Eqs. 9–10). A large change in either direction indicates substantial answer-state evolution rather than stagnation; we do not assume such changes are necessarily beneficial. Consistently, under identical masking and normalization, fully reverse-KL training achieves 43.3% on OmniGAIA, while signed gating that also routes large negative changes to forward KL achieves 44.0%, compared with 45.4% for magnitude-based gating (Table 4). Extending forward KL to large negative changes thus removes most of the gain from magnitude-based routing. Moreover, the signed variant yields fewer tool turns (3,652 vs. 4,145) yet lower accuracy, indicating that the gain of Δ-OPD is not explained by shortening interaction alone.

Threshold. We fixed τ = 0.05 before Δ-OPD training through offline behavioral calibration on multiple batches of student-generated trajectories collected during separate standard OPD training runs. We compared candidate thresholds {0.01, 0.025, 0.05, 0.1}, using exact tool-call repetitions as conservative indicators of stagnation rather than its complete definition. At τ = 0.05, the readiness-change criterion captured 73–81% of exact repetitions, while only 10–16% of flagged interactions exhibited clearly observable progress under conservative, coarse-grained rule-based checks, such as new evidence subsequently used, useful computation results, or successful recovery from tool failures. This offered a favorable balance between capturing repetitive behavior and limiting the inclusion of evidently productive interactions. The reported ranges reflect variation across separate standard OPD training runs rather than confidence intervals.

Since readiness is measured as mean per-token log-likelihood, τ = 0.05 corresponds to approximately a 5% relative change in the geometric-mean probability assigned to the reference answer. Subsequent analysis of 28,952 training tool-calling turns shows that 39% satisfy |Δt| ≤ 0.05 (Appendix C), and trajectories observed at these states fail more often than those at progressing states (72% vs. 58%). Threshold sensitivity experiments further show that tightening τ to 0.025 yields comparable OmniGAIA accuracy (45.0% vs. 45.4%), whereas loosening it to 0.1 reduces accuracy to 43.1% (Table 4). This is consistent with the design: a looser threshold may route interactions with non-negligible readiness changes to forward KL. Together, the offline calibration and subsequent sensitivity results support τ = 0.05 as a well-grounded operating point. For masking, we use the readiness at the state from which a verified correct answer is produced as a trajectory-specific reference, avoiding an additional global threshold.

Alternative indicators. We also examined several alternative indicators, which showed important limitations compared with readiness change. Local teacher–student discrepancy is substantially less predictive of downstream outcomes: its AUROCs for task failure, budget exhaustion, and future repetition are 0.47, 0.46, and 0.45, compared with 0.65, 0.64, and 0.56 for readiness change (Appendix C). Exact repetition accounts for only 1.3% of tool-calling turns, while 98% of near-zero-change turns are not exact repeats, so repetition alone misses most low-change interactions. Trajectory length alone is also insufficient: even after eight turns, 45% of subsequent turns still exhibit readiness changes above τ. Additional supervision-allocation baselines, including proportion-matched random routing, are compared in A3.

AuthorsOpenReview ↗

Response to Reviewer xbQJ (Part 2/3)

We next address the reviewer's question regarding behavioral validation (Q2) through a blinded human audit, post-training behavioral analysis, and qualitative case studies.

Q2: Have you conducted behavioral audits or qualitative case studies to verify that low-progress interactions identified by the model actually correspond to human-judged stalling or repetitive reasoning loops?

A2: Yes. Our blinded human audit provides direct evidence that low-change interactions frequently correspond to human-judged stagnation, while post-training behavioral analysis shows that Δ-OPD substantially reduces prolonged low-progress tool use.

Human behavioral audit. We uniformly sampled 100 low-change interaction windows from 586 eligible windows across 55 tasks, using the criterion |Δt| < 0.05 (Section 4.4). Annotators examined each window and its preceding history, blinded to readiness scores, routing decisions, ground-truth answers, and subsequent turns. Meaningful progress required new task-relevant evidence, hypothesis narrowing, useful computation, or error recovery.

Of these windows, 80 were judged stalled: 54 redundant searches, 14 failed tool calls, 7 instances of repeated reasoning, and 5 unsupported guesses. The remaining 20 exhibited meaningful progress. These results show that readiness change captures diverse forms of low-yield behavior beyond exact repetition, while acknowledging that not every flagged interaction is stagnant.

Mitigation after training. Table 4 reports one rollout per policy on 358 OmniGAIA tasks. Compared with standard OPD, Δ-OPD reduces low-progress tool turns from 3,995 to 2,611, with 94.2% of the reduction occurring after turn five. Tool turns not flagged as low-progress increase from 1,345 to 1,534, while budget exhaustion decreases from 17.6% to 10.9%. In the main four-run evaluation, budget exhaustion similarly decreases from 15.3% to 9.8%, alongside an accuracy improvement from 40.8% to 45.4%. Together, these results indicate that Δ-OPD mitigates prolonged stagnation while improving task-solving performance.

Qualitative case studies. Appendix D illustrates repeated reasoning over resolved results, failed retrieval during verification, and unsupported inferences that constrain subsequent searches. A boundary case further distinguishes redundant repetition from potentially useful verification, providing a more nuanced understanding of low-progress behavior.

AuthorsOpenReview ↗

Response to Reviewer xbQJ (Part 3/3)

We further address Q3 on alternative supervision strategies and Q4 on related work, while clarifying the multimodal relevance of our method. Together with Parts 1–2, which address indicator design and behavioral validation, these responses directly cover both critical weaknesses raised by the reviewer.

Q3: Is it possible to provide a baseline comparison against alternative supervision filters or progress-gating heuristics to demonstrate that the proposed OPD formulation is optimal?

A3: Yes. We compare Δ-OPD against existing distillation methods, alternative supervision filters, and progress-gating strategies.

Three-benchmark comparison. Tables 2 and 3 report four-run average accuracy across OmniGAIA, VideoDR, and VisualSimpleQA.

MethodAverage accuracy
Standard OPD50.7%
TCOD52.4%
SOD52.1%
TurnOPD50.9%
Zero-loss replacement of FKL (filtering)52.8%
Proportion-matched Random FKL52.1%
Δ-OPD (Ours)55.5%

Zero-loss filtering achieves 52.8% and proportion-matched Random FKL achieves 52.1%, versus 55.5% for Δ-OPD. Neither filtering nor frequency matching reproduces the gains of progress-conditioned supervision.

Alternative supervision and gating strategies. We further compare uniform KL objectives, fixed-ratio allocation, and signed gating on OmniGAIA (Table 4). These variants retain the same readiness-based masking and turn-level normalization as Δ-OPD, isolating the effect of supervision allocation.

StrategyAccuracy
Standard OPD40.8%
Uniform FKL42.5%
Uniform RKL43.3%
Fixed-ratio FKL/RKL43.3%
Signed gating44.0%
Δ-OPD (Ours)45.4%

Δ-OPD achieves the highest accuracy among the evaluated strategies, outperforming uniform objectives, randomized allocation, and signed gating. Together, these results provide strong empirical evidence that both the placement and the form of supervision matter, validating progress-conditioned KL routing as a key design choice for long-horizon agent distillation.

Q4: The related work section seems overly brief and lacks depth. Is it sufficient to omit a broader literature survey and conceptual comparison across general LLM/VLM knowledge distillation or trajectory-level imitation?

A4: We appreciate this suggestion and have expanded Section 5 to broaden the literature review and further clarify our contribution. The additions cover general LLM/VLM distillation and divergence selection (MiniLLM, GKD, DistiLLM, AKL), trajectory-level imitation (behavior cloning and DAgger), and recent multimodal distillation methods, including Vision-OPD, SpatialOPSD, visual-manipulation self-distillation, and visual-grounding-aware objectives.

Conceptual distinction. Existing methods primarily focus on matching teacher distributions, selecting divergence objectives, transferring privileged visual information, or addressing distribution shift. In contrast, Δ-OPD tackles a distinct challenge: how to allocate teacher supervision across student-visited states based on realized progress. Specifically, answer-readiness changes determine forward/reverse KL routing, while a trajectory-specific reference enables selective masking of low-progress supervision. A3 further shows that divergence-based weighting (SOD), fixed or uniform KL objectives, and signed gating do not reproduce Δ-OPD's gains. Our key contribution is therefore not merely choosing a different distillation objective, but adapting teacher supervision to the student's realized progress throughout long-horizon interactions.

Multimodal Relevance and Scope. We respectfully note that multimodal relevance does not require a modality-specific architectural mechanism. Δ-OPD targets a concrete failure mode identified in long-horizon multimodal agent distillation: agents must gather and integrate visual, video, and other task-dependent evidence through multi-step interactions, yet locally reasonable teacher-supervised actions can repeatedly fail to advance the student's answer state. Consistent with this mismatch, readiness change is more predictive of downstream task failure than local teacher–student discrepancy (AUROC 0.65 vs. 0.47; A1 and Appendix C), motivating progress-conditioned KL routing and selective masking.

This is not merely a general method evaluated incidentally on multimodal tasks. Our behavioral analysis (Section 2) identifies the failure mode, while experiments across OmniGAIA, VideoDR, and VisualSimpleQA establish the effectiveness of Δ-OPD in multimodal settings, outperforming standard OPD on all three benchmarks. The contribution lies in addressing a demonstrated multimodal agent learning problem, not in imposing modality-specific architectural constraints. Its potential applicability to text-only agents broadens the scope of the principle rather than diminishing its contribution to multimodal learning.

AuthorsOpenReview ↗

Follow-up on Author Responses to Reviewer xbQJ

Dear Reviewer xbQJ,

Thank you for your careful review and thoughtful feedback.

As the discussion deadline approaches, we would appreciate your feedback on whether our responses have addressed your concerns. We believe our responses address the issues you raised, and the additional evidence further validates the original design choices and supports the central claims of Δ-OPD.

If any issues remain unresolved, we would be happy to provide further clarification. We would also be grateful if you could reconsider your assessment in light of this evidence.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_N5VDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Δ-OPD, which adapts OPD based on changes in the student’s likelihood of the reference answer. It uses these changes and a trajectory-specific readiness reference to select reverse KL, forward KL, or zero loss for each interaction turn. Experiments on three multimodal agent benchmarks show improved average accuracy for Qwen3.5-2B and Qwen3.5-4B, with reduced budget exhaustion on OmniGAIA.

Critical strengths
  • The paper identifies a meaningful limitation of uniform teacher supervision: locally plausible interactions may no longer help the student make progress. The proposed design is intuitive and straightforward, preserves student-generated trajectories, and introduces no inference-time overhead.

  • The 4B results are encouraging, with improvements over several recent distillation methods across all three benchmarks, including two evaluated without benchmark-specific training. The frequency-matched random-FKL ablation is particularly useful, providing evidence that where forward KL is applied matters beyond its overall frequency.

  • The paper is very clearly written and easy to follow.

Critical weaknesses
  • The random-FKL ablation provides useful evidence that the placement of forward KL matters at the chosen frequency. However, comparisons with always-forward-KL training or a tuned fixed FKL/RKL mixture are missing. These would help establish whether progress-based routing offers an advantage over simpler alternatives to vanilla OPD.
  • Evidence for generality and robustness remains limited to 2B/4B students on multimodal answer-oriented tasks, leaving applicability to larger students and broader agent settings unclear. This may restrict the potential impact of the proposed approach.
  • The substantial regression on 2B VideoDR Hard also calls for more detailed analyses of the conditions under which the proposed method will help/regress.
Questions for authors
  • How does Δ-OPD compare with always-forward-KL training or a fixed FKL/RKL mixture? Replacing the selected forward-KL losses with reverse KL while keeping masking and normalization unchanged would also help isolate the contribution of switching directions.

  • What evidence supports treating large negative readiness changes as useful progress rather than distraction or confusion? A signed-versus-absolute-change comparison and an ablation of the separate Repeat criterion would help validate these design choices.

  • Could you provide confidence intervals or paired significance tests for the main results and ablations? For example, the last two rows in Table 3 are fairly close to the full method, and it's difficult to tell whether the differences are meaningful.

  • The main text and Appendix B describe KL computation using full next-token distributions, while Table 4 specifies top-k = 64. How are the support and renormalization handled for both forward and backward KL?

3 comments
AuthorsOpenReview ↗

Response to Reviewer N5VD (Part 1/2)

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. This part addresses Q1 and Q2; Q3 and Q4 follow in the next comment.

Q1: How does Δ-OPD compare with always-forward-KL training or a fixed FKL/RKL mixture? Replacing the selected forward-KL losses with reverse KL while keeping masking and normalization unchanged would also help isolate the contribution of switching directions.

A1: We compare Δ-OPD against three alternative KL strategies: fully forward KL (42.5%), a fixed, uniformly mixed FKL/RKL strategy (43.3%), and fully reverse KL (43.3%). All variants keep masking and turn-level normalization unchanged, varying only the KL supervision strategy (Table 4). Δ-OPD achieves 45.4%, outperforming all three. In particular, the fully reverse-KL variant implements your suggested control by replacing the FKL losses selected by Δ-OPD with RKL; progress-conditioned routing improves accuracy by 2.1 points over this control (43.3% → 45.4%). These controls cover uniform and fixed-mixture supervision strategies, none matching Δ-OPD's performance.

Furthermore, the proportion-matched Random FKL baseline preserves each trajectory's FKL allocation ratio but randomizes its placement; its three-benchmark average drops from 55.5% to 52.1%, showing that FKL effectiveness depends on progress-conditioned placement, not allocation ratio alone.

Q2: What evidence supports treating large negative readiness changes as useful progress rather than distraction or confusion? A signed-versus-absolute-change comparison and an ablation of the separate Repeat criterion would help validate these design choices.

A2: We use readiness-change magnitude to identify substantive answer-state updates rather than assuming that every change improves correctness. A large negative change need not be beneficial; the key question is whether its negative sign alone warrants classifying a turn as low-progress and routing it to FKL. Our behavioral audits and controlled comparisons indicate that it does not.

Why magnitude rather than sign. Under RKL, the corrective gradient toward a teacher-preferred action is scaled by the student's own probability qₖ (Eq. 9), so a student trapped in a self-reinforcing pattern receives only weak correction; FKL maintains this pressure (Eq. 10). FKL is therefore intended for stalled turns, which near-zero readiness change identifies well: it predicts eventual task failure (AUROC 0.65), outperforming local teacher–student discrepancy (0.47; Appendix C). A large negative change instead indicates that the answer state has moved substantially, and such turns continue to receive reverse-KL teacher supervision.

Controlled evidence. Under identical masking and normalization, fully reverse-KL training achieves 43.3% on OmniGAIA, compared with 44.0% for signed gating (Δₜ ≤ 0.05) and 45.4% for absolute gating (|Δₜ| ≤ 0.05) (Table 4). Extending FKL to large negative changes thus removes most of the gain from magnitude-based routing. Notably, the signed variant yields fewer tool turns (3,652 vs. 4,145) yet lower accuracy, indicating that the gain of Δ-OPD is not explained by shortening interaction alone.

Human behavioral validation. In a blinded human audit of 100 low-change windows (|Δₜ| < 0.05), sampled uniformly from 586 eligible windows across 55 tasks (Section 4.4), annotators examined each window and its preceding history, blinded to readiness scores, routing decisions, ground-truth answers, and future turns. Meaningful progress required new task-relevant evidence, hypothesis narrowing, useful computation, or error recovery; query reformulation alone did not qualify. 80/100 were judged stalled: 54 redundant searches, 14 failed tool calls, 7 instances of repeated reasoning, and 5 unsupported guesses.

To directly examine large negative changes, we further audited 100 windows with Δₜ < −0.05 under the same protocol, sampled from 707 eligible windows across 218 tasks. Only 19/100 were judged stalled (15 redundant searches, 3 instances of repeated reasoning, and 1 failed tool call); the remaining 81 exhibited meaningful updates. In these samples, the stalled rate was more than four times higher for low-change windows, supporting magnitude-based rather than sign-based routing.

The Repeat criterion. In our Δ-OPD training logs, the readiness-change criterion already captures 68.3% of exact-repeat turns, and repeat-only turns account for about 1% of low-progress turns. Consistently, removing the explicit Repeat condition yields 45.6% on OmniGAIA (mean over four evaluation runs), comparable to the full Δ-OPD (45.4%). Thus, readiness change is the primary low-progress signal; removing Repeat does not materially change aggregate accuracy, while Repeat serves as a lightweight safeguard for explicit repetitions that readiness change does not capture.

AuthorsOpenReview ↗

Response to Reviewer N5VD (Part 2/2)

We next address Q3 on statistical reliability and Q4 on the implementation of top-k KL approximation.

Q3: Could you provide confidence intervals or paired significance tests for the main results and ablations? For example, the last two rows in Table 3 are fairly close to the full method, and it's difficult to tell whether the differences are meaningful.

A3: Across four independent evaluation runs, Δ-OPD outperforms OPD on all three benchmarks, with average gains of 4.6, 7.0, and 2.6 percentage points, respectively (Table 2). To further quantify uncertainty in the closer component ablations, we report run-level 95% confidence intervals on OmniGAIA below.

The component ablations in Table 3 further assess the roles of masking and normalization. Removing masking lowers accuracy by 1.3, 1.7, and 0.9 percentage points across the three benchmarks, respectively, while removing normalization lowers accuracy by 0.7, 0.9, and 0.6 percentage points. On OmniGAIA, the four-run mean accuracy is 44.1% without masking (95% run-level t-CI: [43.4, 44.7]) and 44.7% without normalization (95% run-level t-CI: [43.4, 46.0]), quantifying uncertainty in the four-run means due to run-to-run variability.

Across all three benchmarks, Δ-OPD achieves higher mean accuracy than either ablation, with a larger observed contribution from masking than from normalization. The reported run-level confidence intervals quantify evaluation variability, while the consistency of the improvements across benchmarks further supports the contribution of both components.

Q4: The main text and Appendix B describe KL computation using full next-token distributions, while Table 4 specifies top-k = 64. How are the support and renormalization handled for both forward and backward KL?

A4: The equations in Section 3 describe the idealized full-vocabulary objectives; training uses top-64 approximations in both directions. Forward KL sums over the teacher's top-64 tokens, using probabilities computed from the full-vocabulary softmax without renormalizing the selected support. Reverse KL uses the student's top-64 tokens, renormalizing their weights within the selected support while retaining full-vocabulary log-probabilities in the KL terms. We clarify this in Section 3.3 and Appendix B.4 of the revision.

We hope these additional analyses and clarifications address the reviewer's concerns and further strengthen the evidence for Δ-OPD. We would be grateful if the reviewer could take our responses into consideration when revisiting the assessment, and we welcome any further questions.

AuthorsOpenReview ↗

Follow-up on Author Responses to Reviewer N5VD

Dear Reviewer N5VD,

Thank you for your thoughtful review and constructive feedback. We also sincerely appreciate your positive assessment of our motivation, methodological design, and experimental results.

As the discussion deadline approaches, we would appreciate your feedback on whether our responses have addressed your concerns. We believe the additional experimental evidence and implementation clarifications further strengthen the support for the proposed method.

If any concerns remain, we would be happy to provide further clarification. We would also be grateful if you could reconsider your assessment in light of the additional evidence.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_b5ykOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Δ-OPD, a progress-aware on-policy distillation method for long-horizon multimodal agents. The key idea is to measure interaction progress through changes in the student's likelihood of the reference answer and adapt the distillation objective accordingly. Progressing turns retain reverse KL, while low-progress turns receive forward KL or zero loss depending on a trajectory-specific readiness reference. Experiments on OmniGAIA, VideoDR, and VisualSimpleQA demonstrate improved average accuracy over standard OPD and several recent baselines, along with reduced interaction budget exhaustion.

Critical strengths
  1. The contrast between premature answering under SFT and excessive tool use under OPD provides a convincing motivation. Measuring progress through changes in the student's answer state is a useful perspective beyond trajectory length or repetition.

  2. The proposed KL routing is straightforward and requires no changes to the inference procedure. I particularly appreciate the random-FKL ablation, which helps establish that the choice of supervised turns matters, rather than simply the proportion of forward KL.

Critical weaknesses
  1. Ambiguity in the progress definition. Section 3.1 treats large positive and negative readiness changes equally as progress. While the authors motivate this choice, a substantial decrease in correct-answer likelihood could also indicate that the student is moving away from the correct answer.

  2. Potentially unreliable readiness reference. The masking criterion relies on the readiness of the final successful-answer state. However, a correct completion does not necessarily imply high readiness, so this reference might occasionally suppress useful earlier interactions.

  3. Limited behavioral analysis after training. The reduction in budget exhaustion is encouraging, but the paper provides relatively little direct evidence of how Δ-OPD changes the frequency of low-progress interactions after training. The connection between the proposed supervision changes and the observed interaction behavior could be clearer.

Questions for authors
  1. Have the authors examined whether large negative readiness changes correspond to productive belief updates or simply deteriorating answer states? It would be helpful to understand how these cases affect the routing decisions.

  2. Could the successful-answer reference become unreliable when the student happens to produce a correct answer despite low readiness? In such cases, how does the masking mechanism avoid suppressing potentially useful interaction supervision?

  3. Neither forward KL nor readiness-based masking explicitly teaches the student when to stop. What changes in the student's tool-use behavior primarily explain the lower budget-exhaustion rate?

5 comments
AuthorsOpenReview ↗

Response to Reviewer b5yk

We sincerely thank the reviewer for the time, effort, and insightful questions regarding the routing dynamics, masking reliability, and stopping behavior of Δ-OPD. To address these concerns, we provide targeted empirical analyses, including a blinded human audit of negative readiness changes (A1), a reference stress test on low-readiness successes (A2), and a turn-level behavioral breakdown (A3). We have also incorporated the relevant analyses and discussions into the revised manuscript. Our detailed responses follow.

Q1: Have the authors examined whether large negative readiness changes correspond to productive belief updates or simply deteriorating answer states? It would be helpful to understand how these cases affect the routing decisions.

A1: Yes. Large negative readiness changes predominantly correspond to meaningful answer-state updates rather than stagnation, although an individual update may move the answer state away from the reference answer.

Behavioral audit. Following the blinded human-audit protocol of Section 4.4, we uniformly sampled 100 interaction windows with large negative readiness changes (Δt < −0.05) from 707 eligible windows across 218 tasks; the sample spans 68 tasks. Meaningful progress required new task-relevant evidence, hypothesis narrowing, useful computation, or error recovery; query reformulation alone did not qualify.

Under the same criterion, only 19/100 large-negative-change windows were judged stalled (i.e., showing no meaningful progress: 15 redundant searches, 3 instances of repeated reasoning, and 1 failed tool call), compared with 80/100 low-change windows (|Δt| < 0.05) in Section 4.4. In these samples, the stalled rate was more than four times higher for low-change windows, supporting the use of readiness-change magnitude to distinguish substantive answer-state updates from stagnation.

Routing implications. Δ-OPD retains reverse KL for non-repeated large-negative-change turns, as in standard OPD: a decrease in readiness is neither rewarded nor exempted from teacher correction. In contrast, signed gating would classify all these negative-change windows as low-progress, including the 81% exhibiting meaningful behavioral updates, and route them to forward KL or zero loss depending on the readiness mask. This alternative achieves 44.0% accuracy versus 45.4% for Δ-OPD on OmniGAIA (Table 4). Together, these results support magnitude-based routing.

Q2: Could the successful-answer reference become unreliable when the student happens to produce a correct answer despite low readiness? In such cases, how does the masking mechanism avoid suppressing potentially useful interaction supervision?

A2: We agree that correctness and readiness are distinct: a correct completion can have low reference-answer likelihood. Importantly, masking requires both low progress and readiness above the successful-answer reference: zₜ = 1 and Vₜ > Vₐₙₛ. Consequently, every non-repeated turn with |Δₜ| > τ retains reverse-KL supervision, regardless of how low the reference is.

Reference stress test. Lowering the reference by 1.0 increases masked turns from 41 to 120, but all 79 additional masks remain confined to the low-progress group. Across the reported probes, low-readiness successes constitute 5.3–10.5% of successful trajectories, with only 0–7.7% of their eligible turns masked. These findings show that the progress gate constrains the effect of a low readiness reference, rather than allowing unrestricted masking.

Q3: Neither forward KL nor readiness-based masking explicitly teaches the student when to stop. What changes in the student's tool-use behavior primarily explain the lower budget-exhaustion rate?

A3: The primary behavioral change is a substantial reduction in repetitive and late-stage low-progress tool use, rather than uniformly shortening trajectories. In a diagnostic analysis of 358 matched OmniGAIA tasks (one rollout per policy), we observe:

MetricOPDΔ-OPD
Total tool turns5,3404,145
Low-progress turns3,9952,611
Exact-repeat fraction58.5%43.0%
Budget-exhaustion rate17.6%10.9%

Notably, 94.2% of the reduction in low-progress turns occurs after turn five, while non-low-progress turns increase from 1,345 to 1,534. This indicates that Δ-OPD primarily reduces prolonged stagnation while preserving productive interaction.

Without requiring an explicit stopping target, Δ-OPD encourages teacher-supported alternatives through forward KL, avoids reinforcing low-progress continuations through readiness-based masking, and retains reverse-KL supervision for terminal answers. In the main four-run evaluation, accuracy increases from 40.8% to 45.4% and budget exhaustion decreases from 15.3% to 9.8%, indicating that reduced looping comes with improved task success rather than merely shorter trajectories.

Thank you for the authors' detailed responses and additional empirical analyses. I appreciate the clarification of the routing and masking mechanisms, as well as the new evidence showing reduced repetitive tool use. However, my main concern remains that large readiness changes do not necessarily correspond to beneficial progress, and the rebuttal does not directly establish whether the resulting KL routing decisions are appropriate. The observed reduction in budget exhaustion also comes with a trade-off in premature stopping. While I find the approach interesting and the empirical results encouraging, these concerns remain important to my assessment, so I will maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer b5yk's Follow-up (1/2): Appropriateness of KL Routing

We thank the reviewer for the thoughtful follow-up. This response (1/2) addresses whether the resulting KL routing decisions are appropriate, using gradient analysis, a blinded human audit, and controlled comparisons; our next response (2/2) addresses premature stopping. The human audit below replaces our preliminary model-assisted analysis.

1. Are the resulting KL routing decisions appropriate?

The failure mode targeted by Δ-OPD. Standard OPD can induce prolonged low-progress interactions (Section 2.1). For student distribution q, teacher distribution p, and student logit sₖ, the idealized KL gradients are:

Reverse KL (Eq. 9): ∂KL(q‖p)/∂sₖ = qₖ[log(qₖ/pₖ) − KL(q‖p)]

Forward KL (Eq. 10): ∂KL(p‖q)/∂sₖ = qₖ − pₖ

For a teacher-preferred action with qₖ ≪ pₖ, reverse-KL correction is attenuated by qₖ, whereas forward KL retains a gradient near −pₖ. FKL can therefore help escape self-reinforcing interactions.

The key is where to intervene. Near-zero readiness change predicts task failure better than local teacher–student discrepancy (AUROC 0.65 vs. 0.47; Appendix C). Δ-OPD applies FKL or masking to low-progress turns identified by near-zero change or repetition, while other turns retain RKL supervision.

Answer-state change versus beneficial progress. Section 3.1 defines "progress" operationally as answer-state change regardless of direction, not guaranteed improvement in correctness. The sign indicates whether readiness increases or decreases, while the magnitude indicates whether the answer state has changed substantially. We will state this distinction more explicitly in Section 3.1 and Figure 2.

A large negative change may be unfavorable, but its sign alone does not establish stagnation. Non-repeated large-negative-change turns retain standard OPD's reverse-KL teacher supervision. Importantly, routing is determined at each turn: if subsequent interactions exhibit near-zero readiness change or repetition, Δ-OPD switches to FKL or masking to address emerging stagnation.

Blinded human behavioral audit. We uniformly sampled 100 low-change windows (|Δt| < 0.05) from 586 eligible windows across 55 tasks, and 100 large-negative-change windows (Δt < −0.05) from 707 eligible windows across 218 tasks.

Annotators examined each window and its preceding history, blinded to readiness scores, routing decisions, ground-truth answers, and future turns. Under the same criterion, meaningful progress required new task-relevant evidence, hypothesis narrowing, useful computation, or error recovery; query reformulation alone did not qualify.

  • Low-change: 80/100 were judged stalled (54 redundant searches, 14 failed tool calls, 7 cases of repeated reasoning, 5 unsupported guesses).
  • Large-negative-change: Only 19/100 were judged stalled (15 redundant searches, 3 cases of repeated reasoning, 1 failed tool call); the remaining 81/100 exhibited meaningful behavioral updates.

In these samples, the stalled rate was more than four times higher for low-change windows. Thus, large negative changes predominantly reflect substantive answer-state updates rather than stagnation, without necessarily improving correctness.

This directly affects supervision: signed gating would classify all large-negative-change windows as low-progress, including the 81/100 exhibiting meaningful updates. Magnitude-based routing instead retains RKL unless the explicit Repeat criterion is triggered. The audit thus supports the behavioral distinction underlying the routing decision.

Why KL direction matters. FKL encourages coverage of teacher-supported actions, including alternatives assigned low student probability. This helps escape stagnation but can unnecessarily alter action preferences when the answer state is already evolving. FKL is not a neutral replacement for RKL.

Controlled routing evidence. Under identical masking and normalization, fully reverse-KL training achieves 43.3% on OmniGAIA, compared with 44.0% for signed gating (Δₜ ≤ 0.05) and 45.4% for magnitude-based gating (|Δₜ| ≤ 0.05) (Table 4). The two gating variants differ only in whether non-repeated large negative readiness changes are routed to the low-progress branch. Signed gating is 1.4 percentage points lower in accuracy than magnitude-based gating, supporting the decision to retain RKL supervision for non-repeated substantial negative changes. Notably, signed gating yields fewer tool turns (3,652 vs. 4,145) yet lower accuracy, indicating that shorter interactions alone do not explain Δ-OPD's advantage.

Taken together, the gradient analysis explains why selective FKL is useful, the human audit identifies where stagnation occurs, and the controlled comparisons establish the consequences of alternative routing decisions. These results support retaining RKL for substantial negative answer-state updates while reserving FKL for detected low-progress interactions, rather than routing by sign alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.