acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Visual Evidence Distillation

Abstract

Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition (), reading (), or answer grounding () can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce (), an on-policy distillation method for visual agents. compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the , , and stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Agentic Visual Reasoning, Reflection on Visual Evidence, Token-Level Reweighting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kdnJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose REVUE, an on-policy distillation method for visual agents. By comparing multiple student rollouts, REVUE employs a critic model to construct visual-evidence reflections: it summarizes the correct evidence and reasoning from successful rollouts and diagnoses the first failed stage (Acquire, Read, or Ground) for incorrect ones. These reflections serve as additional context for the teacher to provide more targeted supervision. A token-level reweighting loss is further applied, assigning higher weights to positions where the reflections most change the teacher's predictions. Experiments on 11 benchmarks and two models demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper is well-organized and easy to follow.

  2. The proposed method is straightforward and intuitive. The Acquire/Read/Ground breakdown is well-motivated by the error analysis and illustrated with clear case studies. The critic is used only at training time, so there is no extra inference cost, and the trained students produce shorter responses with fewer tool calls.

  3. The experiments cover two model families and 11 benchmarks and show consistent gains over the evaluated OPD baselines. The ablations are comprehensive.

Critical weaknesses
  1. It is unclear whether the improvement comes from the extra information provided by the critic model and the reference answer. This also makes the comparison with other baselines unfair, since the proposed method effectively has a stronger “teacher” providing context, plus extra compute. GT-Privileged is not an information-matched control either. The results showing that the student surpasses the teacher also indicate that the improvements do not come purely from distillation from the RL expert.

  2. When reproducing the baseline methods, the authors replace the teacher with RL experts, which does not align with the original papers. Additionally, some reproduced numbers are even lower than those of cold-start SFT; therefore, these numbers say little about how the original methods would perform.

  3. The improvements on some benchmarks are marginal, but no statistical results are reported.

  4. All experiments are conducted using models that are not current-generation. It is unclear whether the proposed method remains beneficial when the student’s Acquire/Read error rates are lower and the model natively supports agentic image operations (e.g., Qwen3.5). In addition, GPT-4o is an outdated closed-source reference, so the claim that both students exceed GPT-4o is not very informative.

Questions for authors
  1. If the RL expert and the critic are available, what is the purpose of distilling into a same-size student? How does REVUE compare with using the critic’s reflections as RL rewards and continuing RL on top of the expert under matched compute?

  2. What is the number of student rollouts per sample for the OPD baselines? Are they matched to REVUE?

  3. How much of the gain remains if the critic does not see the reference answer or if a much smaller critic is used?

6 comments
AuthorsOpenReview ↗

Response to Reviewer kdnJ (1/5): Overview and W1

We thank the reviewer for the detailed and constructive comments. We ran new experiments to address each point on supervision attribution, baseline configurations, multi-seed consistency, current-generation models, and RL alternatives, and have incorporated all new results into the revised manuscript (highlighted in blue). Unless noted otherwise, all new runs train Qwen2.5-VL-7B on the same data and training steps as ReVuE, and report weighted category averages (%) for Perception, Math, and General, with full 11-benchmark tables added to Appendices C.1 and C.6–C.9.

Overview of new evidence

ConcernNew evidenceKey result
W1 Gains may come from critic information or computeControls with the same critic, reference answer, and critic callsCritic information alone recovers under half of the perception and math gains
W2 Baselines use an RL-expert teacherBaselines rerun under their original papers' teacher settingsOriginal settings score lower on all six averages; ReVuE remains best
W3 No statistical results3 random seeds (42, 43, 44)Every ReVuE seed beats every Vanilla OPD seed on all three category averages
W4 Older-generation modelsQwen3.5-35B-A3B → Qwen3.5-2B+3.50% Perception and +2.40% General over Vanilla OPD
Q1 Use the critic as an RL reward insteadSame data and steps (575.6 vs. 635.0 GPU-h)ReVuE leads on all three averages; RL drops Math below initialization
Q2 Rollout budget—M = 8 rollouts per query for all OPD methods, matched to ReVuE
Q3 No reference answer / smaller criticAnswer-free critic; 122B critic92% of the perception gain kept without answers; 122B critic matches or beats all baselines

W1: Do the gains reflect extra information from the critic and the reference answer?

GT-Privileged tests only whether giving the reference answer directly to the teacher suffices. To match ReVuE's full information budget, we add two controls using the same critic, the same reference answer, and the same number of critic calls. Even when the critic writes per-trajectory feedback, the teacher recovers less than half of ReVuE's perception and math gains: the advantage comes from structuring the diagnosis around the first failed stage and converting it into token-level supervision, rather than from raw critic information.

Information-matched controls (category averages in %; Δ vs. Vanilla OPD)

Teacher contextCriticRef. answerAnchorBreak PointPerceptionMathGeneral
Vanilla OPD––––62.6147.6753.28
GT-Privileged–✓––62.26 (−0.35)47.49 (−0.18)53.23 (−0.05)
Critic + generic context✓✓––63.74 (+1.13)47.99 (+0.32)53.39 (+0.11)
Anchor-only✓✓✓–63.63 (+1.02)48.03 (+0.36)53.86 (+0.58)
ReVuE✓✓✓✓65.01 (+2.40)48.49 (+0.82)54.14 (+0.86)

In Critic + generic context, the critic sees the same trajectory group and reference answer and writes unconstrained natural-language feedback—a shared summary of how successful rollouts solved the task plus trajectory-specific error feedback—omitting only ReVuE's stage labels, Anchor evidence, and Break Point; the teacher distills with uniform token weights.

  • The reference answer alone adds no gain. GT-Privileged scores at or slightly below Vanilla OPD on all three averages.
  • Unstructured critic feedback recovers less than half of the gain. With the same critic calls as ReVuE, the two matched controls recover 42%–47% of the perception gain and 39%–44% of the math gain.
  • Break Point localization drives the remaining jump. Holding the rest of the pipeline fixed, adding first-failure diagnosis to Anchor-only more than doubles the gain in Perception (+1.02% → +2.40%) and Math (+0.36% → +0.82%). We have added these controls to Section 6 (Table 6a) and Appendix C.6 (Table 17).
AuthorsOpenReview ↗

Response to Reviewer kdnJ (2/5): Q3 and W1 (continued)

Q3: How much gain remains without the reference answer, or with a much smaller critic?

Removing both the reference answer and per-trajectory correctness labels from the critic retains 92% of ReVuE's perception gain (+2.20% vs. +2.40%). Replacing the 397B critic with Qwen3.5-122B-A10B (10B active parameters) still matches or exceeds every baseline in Table 1 and the RL Expert teacher across all three categories.

(a) Reference answer × diagnosis (Perception / Math / General in %; Δ vs. Vanilla OPD)

No diagnosisWith diagnosis
No answerVanilla OPD: 62.61 / 47.67 / 53.28Answer-free critic: 64.81 (+2.20) / 47.60 / 53.02
With answerGT-Privileged: 62.26 / 47.49 / 53.23ReVuE: 65.01 (+2.40) / 48.49 (+0.82) / 54.14 (+0.86)

Answer-free critic: both the reference answer and per-trajectory correctness labels are removed; the critic sees only the query, student trajectories, and their observed images.

  • Visual verification needs no answer key. By checking visual facts directly against the images students observed, the answer-free critic retains 92% of the perception gain.
  • Math and general reasoning require both target and diagnosis. Catching a Ground-stage error needs a verifiable target, yet giving the answer directly to the teacher yields no gain (GT-Privileged); the answer helps only when converted into a failure diagnosis.
  • Shared training-set labels. Reference answers come from the standard Thyme-RL training split already used to train the RL Expert teacher and filter RFT demonstrations.

(b) Smaller critic. Stage-diagnosis reliability can be audited on a few hundred rollouts before training, following the agreement protocol in Appendix B.6 (revised Table 9):

Criticvs. Gemini 3.8 Flashvs. GPT-5.6 Luna
Qwen3.5-397B-A17B (paper)93.21% / κ 0.89292.80% / κ 0.886
Qwen3.5-122B-A10B92.26% / κ 0.87791.44% / κ 0.865
Qwen3.5-35B-A3B81.82% / κ 0.71581.11% / κ 0.708

Qwen3.5-122B-A10B closely tracks the 397B critic in stage agreement, whereas 35B-A3B drops by over 11%. Training ReVuE with the 122B critic reaches 64.07% / 48.13% / 53.66% (+1.46% / +0.46% / +0.38% over Vanilla OPD), matching or exceeding every Table 1 baseline and the RL Expert on all three averages (added to Section 6 and Appendix C.8, Table 19).


W1 (continued): Why can the student surpass the RL Expert?

ReVuE distills from the RL Expert conditioned on the reflection (qRq^R, Eq. 10) rather than the unconditioned expert (q0q^0). Surpassing the unconditioned policy is the standard objective of privileged-context distillation—Vision-OPD, for example, trains a student to outperform its own unprivileged policy. Part 1/5 shows that unstructured critic feedback recovers under half of this gain, and Part 3/5 shows that ReVuE with a frozen self-teacher (no RL Expert) still surpasses the RL Expert on Math (48.32% vs. 47.56%).

W1 (continued): Extra compute

  • Training-time only. Critic inference adds roughly 70 GPU-hours of service time to a 635 GPU-hour run (App. A), or 10.8% of mean step time; test-time inference cost is unchanged.
  • Compute-matched control. Critic + generic context in Part 1/5 makes the exact same critic calls yet recovers less than half of the perception and math gains.
  • Comparable teacher forward cost. ReVuE queries the teacher twice per position (q0q^0, qRq^R), matching the two-view teacher passes in V-Zero and VAD.
AuthorsOpenReview ↗

Response to Reviewer kdnJ (3/5): W2 and Q2

W2: Baselines were reproduced with an RL-expert teacher rather than their original settings

Table 1 uses the shared RL Expert across all OPD methods to hold teacher capacity constant and isolate the supervision signal. To evaluate the original configurations directly, we reran Vision-OPD and VAD with their original papers' self-teacher settings under the same data and training steps.

Original-setting reproductions (category averages in %)

MethodTeacherPerceptionMathGeneral
Cold-start–60.7246.3852.68
Vision-OPDRL Expert (Table 1)59.9845.4852.08
Vision-OPDoriginal: EMA self-teacher58.02 (↓)44.76 (↓)51.54 (↓)
VADRL Expert (Table 1)62.0845.6253.65
VADoriginal: frozen initial model60.83 (↓)45.30 (↓)51.53 (↓)
ReVuEfrozen self-teacher61.9748.3252.71
  • Using the RL Expert strengthened the baselines. Under their original self-teacher setups, Vision-OPD and VAD score lower on all six category averages than the RL-Expert versions in Table 1.
  • ReVuE remains best without an RL Expert. With the same frozen initial model as VAD's original setup, ReVuE leads on all three averages, and its self-teacher Math score (48.32%) exceeds every method in Table 1, including the RL Expert (47.56%).

Why some baselines fall below cold-start. Vision-OPD and VAD were designed for single-pass VQA without tool calls, building privileged views (evidence-centered crops or evidence-removed views) from the raw input image. In multi-turn agentic reasoning, the student generates new visual states through its own tool calls, so static views of the initial image often mismatch what the student observed. ReVuE instead selects its Anchor from images observed within the students' own trajectories (added to Section 6, Table 6b, and Appendix C.7, Table 18).


Q2: Number of student rollouts per sample for the OPD baselines

All OPD methods use M = 8 student rollouts per query, matched to ReVuE, with the same batch size, dataset, and step budget. This matches VAD's original configuration (8 rollouts per prompt), and RFT likewise samples 8 RL-Expert trajectories per prompt. ReVuE uses no extra rollout budget (clarified in Section 5.1, Appendix B.1, and Appendix B.3, Table 8).

Reviewer_oY2AOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an on-policy distillation framework for visual agents named ReVuE. Specifically, ReVuE consists of three core components: the Visual-Evidence Reflection module, the Token Impact module, and the Reflection-Aware Grouped Distillation module. The Visual-Evidence Reflection module encourages accurate evidence acquisition and reasoning through an evidence anchor and a diagnosis of the first failure across the Acquire, Read, and Ground stages. The Token Impact module supports targeted supervision via absolute shifts in teacher log probabilities with and without reflection. Finally, the Reflection-Aware Grouped Distillation module adaptively reweights token-level reverse-KL losses using impact-based grouping and group-wise normalization.

The experimental evaluation in this paper assesses the effectiveness of ReVuE across multiple benchmark datasets, including HRBench-4K, HRBench-8K, V* Bench, MathVista, MathVerse, and ChartQA-Pro. Results demonstrate that ReVuE achieves higher weighted-average task accuracy while reducing response lengths and tool-use frequency on visual perception benchmarks.

Critical strengths
  • The idea of using visual models’ own tool interaction trajectories to provide more targeted distillation supervision is considered novel.
  • The paper attempts to turn error analysis into trainable supervision, offering useful insights for the community.
Critical weaknesses
  • One of the paper’s motivations is to identify the first point of failure in the visual evidence chain and apply supervision where the error begins to propagate. However, the critic’s output consists of a stage label and a natural-language error description, without specifying the failed interaction step or the locations of erroneous code or tokens. My understanding is that the procedure is essentially as follows: the critic identifies which stage failed and what type of error occurred, and the teacher then determines how to adjust its assessment of each output token based on the reflection. Does this mean that correct stage classification does not necessarily ensure that the subsequent adjustments target the appropriate actions or reasoning? In other words, does the method rely too heavily on the teacher’s implicit capabilities?
  • The metric defined in Eq. (7) essentially measures how much the teacher changes its assessment of the token actually generated by the student after receiving the reflection. However, it does not account for how far the student’s current distribution is from the target distribution, and it uses different probability quantities from those involved in the actual distillation gradient. It seems that the incremental information provided by reflection is not fully aligned with the student’s overall learning needs.
Questions for authors
  • To what extent do the teacher’s own capabilities influence the distillation process? Do the method’s benefits primarily stem from richer hints from the teacher or from its specific framework for diagnosing the evidence chain?
  • Giving the teacher advance access to evidence from future steps or other trajectories may alter its judgment about which actions are necessary to acquire evidence. Could this introduce a mismatch between training and inference?
3 comments
AuthorsOpenReview ↗

Response to Reviewer oY2A (1/2): Overview and W1 & Q1

We thank the reviewer for the insightful assessment and constructive questions. All new runs train Qwen2.5-VL-7B on the same data, budget (M=8M = 8), and steps as ReVuE, reporting weighted averages (%) for Perception, Math, and General (blue revisions; 11-benchmark tables in Apps. C.1–C.2 & C.6–C.7).

ConcernEvidenceKey result
W1 & Q1 Localization, teacher, & hintsSelf-teacher, Critic + generic, & Anchor-onlySelf-teacher Math (48.32%) beats RL Expert; generic hints recover <50% while Break Point >2× gains
W2 Impact vs. learning needsEq. (8), Lemma 1 (App. D.1), & ablationsRKL scales updates by student–target gap; impact beats random (+1.37% / +1.07%)
Q2 Retrospective evidenceOn-policy prefixes, Eqs. 13–14, & Fig. 5Alters only qRq^{\mathcal R}; adaptive tool use (83.7% VisualProbe; +3.10% TreeBench)

W1 & Q1: Stage-to-token localization, teacher role, and hints vs. diagnosis

1. Decoupling trajectory diagnosis from token localization. Predicting raw token offsets directly is brittle across tokenizers and multi-turn code; ReVuE pairs closed-set diagnosis with prefix-aligned teacher scoring:

  • Step narrowing (App. A.5, Eqs. 13–14). The Anchor prefers the trajectory's own minimal-cost sufficient state (Ci≠∅\mathcal C_i \neq \emptyset): Acquire means no own state sufficed (faulty tool code / crop), while Read / Ground means an own sufficient state exists (post-observation error). Stage labels reach 96.88% human agreement (95.31% on errors; Table 5).
  • Token localization (Eqs. 5–7). Scoring the same prefix with vs. without Ri\mathcal R_i concentrates impact at the error origin—suppressing faulty crop targets (foreground, left; Fig. 18), misreadings (one; Fig. 3), and wrong relations (exceed vs. above; Fig. 19), while final answer tokens barely shift (Lucia; Fig. 3).

2. Reliance on implicit teacher capability (W1, Q1-a). Fixing the critic, data, and budget, we replace the RL Expert with the frozen cold-start checkpoint (student's initialization), differing only via Ri\mathcal R_i:

TeacherMethodPerceptionMathGeneral
—Cold-start60.7246.3852.68
Self (cold-start)Vanilla OPD60.4246.4552.11
Self (cold-start)GT-Privileged60.8147.0152.52
Self (cold-start)ReVuE61.9748.3252.71
RL Expert (frozen)—63.8947.5653.60
RL ExpertVanilla OPD62.6147.6753.28
RL ExpertReVuE65.0148.4954.14

Self-teacher Vanilla OPD gives no gain over cold-start, whereas self-teacher ReVuE gains +1.55% / +1.87% / +0.60% over self-OPD and beats self-teacher GT-Privileged in all categories. Without an RL Expert, its Math (48.32%) exceeds RL-Expert Vanilla OPD (47.67%) and the RL Expert (47.56%). Stronger teacher priors lift Perception (+3.04%), while converting diagnosis into token supervision works already at initial student capacity.

3. Richer hints vs. diagnosing the evidence chain (Q1-b). With the shared RL-Expert teacher, data, and budget, we compare progressively structured contexts:

Teacher contextCritic & answerWeightingPerceptionMathGeneral
Vanilla OPD–Uniform62.6147.6753.28
GT-PrivilegedAnswerUniform62.26 (−0.35%)47.49 (−0.18%)53.23 (−0.05%)
Critic + generic context✓Uniform63.74 (+1.13%)47.99 (+0.32%)53.39 (+0.11%)
Anchor-only✓Impact63.63 (+1.02%)48.03 (+0.36%)53.86 (+0.58%)
ReVuE (Anchor + Break Point)✓Impact65.01 (+2.40%)48.49 (+0.82%)54.14 (+0.86%)

Critic + generic context: same critic and answer give free-form rollout summaries and per-trajectory error feedback, omitting stage labels, Anchor, and Break Point. Anchor-only: keeps Anchor images, visual facts, grounding rules, and ReVuE impact reweighting, removing only Break Point (all-wrong groups without an Anchor use uniform base-teacher OPD).

  • Richer hints recover <50% of the gain. GT-Privileged adds no gain, and Critic + generic context recovers 47% / 39% / 13% of ReVuE's gains over Vanilla OPD.
  • Break Point drives the jump. Adding Break Point to Anchor-only adds +1.38% / +0.46% / +0.28%, more than doubling Perception and Math gains over Vanilla OPD and improving 9 of 11 benchmarks (+2.38% HRBench-8K, +2.14% VisualProbe; Anchor-only leads only on V* Bench, 84.29% vs. 82.20%, and InfographicVQA, 80.35% vs. 80.27%), matching App. C.6 (Table 16: 60.13% → 92.60%; corrections 62 → 124/139; flips 47 → 8/172; added in Secs. 4.1, 6 Table 6, & Apps. C.6–C.7 Tables 17–18).
AuthorsOpenReview ↗

Response to Reviewer oY2A (2/2): W2, Q2, and Closing

W2: Complementary roles of token impact and reverse KL in Eq. (8)

In Eq. (8), wi,tDKL(pi,tSuppi,t∥(qi,t∗)Suppi,t)w_{i,t} D_{\mathrm{KL}}\left(p_{i,t}^{\rm Supp_{i,t}} \Vert (q_{i,t}^*)^{\rm Supp_{i,t}}\right) separates what to learn from where reflection shifts supervision, pairing student–target gap with reflection sensitivity without double-counting:

  1. Reverse KL captures student–target distance. At each supervised token, DKL(pSupp∥(q∗)Supp)D_{\mathrm{KL}}(p^{\mathrm{Supp}} \Vert (q^*)^{\mathrm{Supp}}) scales update direction and magnitude by student–target gap: mastered tokens have near-zero loss and gradient, and Low retains half the trajectory weight (λ=0.5\lambda = 0.5). Because 20% of early-training tokens carry 97.6% of total impact (Fig. 4), wi,tw_{i,t} concentrates weight so sparse shifts are not diluted; folding student–target gap into wi,tw_{i,t} again would double-count discrepancy and pull weight toward generic high-entropy tokens.
  2. Full-vocabulary shifts at yi,ty_{i,t} match the shared-support gradient weight. By Lemma 1 (Eq. 6) and App. D.1 (Eq. 15), the support-renormalized KL difference is −∑v∈SupppSupp(v)d(v)+log⁡[qR(Supp)/q0(Supp)]-\sum_{v \in \mathrm{Supp}} p^{\mathrm{Supp}}(v) d(v) + \log[q^{\mathcal R}(\mathrm{Supp}) / q^0(\mathrm{Supp})]. The θ\theta-independent normalization ratio vanishes under ∇θ\nabla_\theta, leaving −Ev∼pSupp[d(v)∇θlog⁡pSupp(v)]-\mathbb E_{v \sim p^{\mathrm{Supp}}}[d(v)\nabla_{\theta} \log p^{\mathrm{Supp}}(v)], weighted by the full-vocabulary log-ratio di,t(v)d_{i,t}(v) of Eq. (5). Evaluating detached si,t=∣di,t(yi,t)∣s_{i,t} = |d_{i,t}(y_{i,t})| at yi,t∼pi,ty_{i,t} \sim p_{i,t} probes this shift on the student's path and stays defined even when an erroneous yi,t∉Suppi,ty_{i,t} \notin {\rm Supp_{i,t}} (App. A.3), while RKL updates the full distribution over Suppi,t{\rm Supp_{i,t}}.
  3. Controlled ablations verify impact ranking (Qwen2.5-VL-7B).
VariantWeighting under fixed q∗q^*PerceptionMathGeneral
Reflection-only (α=100\alpha = 100%)Uniform63.7047.7854.42
Random select 20%Same weights, random63.6447.4254.10
ReVuE (Impact top 20%)Same weights, impact-ranked65.0148.4954.14
  • At fixed weights (α=0.2,λ=0.5\alpha = 0.2, \lambda = 0.5), ReVuE beats Random select 20% by +1.37% Perception and +1.07% Math. Across 3 seeds (42, 43, 44), reweighting beats Reflection-only on every seed in Perception (mean +0.74%) and Math (mean +1.14%) and lifts the 3-seed General mean (53.60% → 54.07%).
  • Renormalizing retained weights to 1 after masking gives Mask low 20% (64.48% / 47.74% / 53.02%) > Mask random 20% (63.52% / 47.24% / 52.58%) > Mask top 20% (62.08% / 45.98% / 52.43%), all below ReVuE (updated in Secs. 4.3 & 6 Tables 2–3; Apps. C.1–C.2 Tables 10–12).

Q2: Retrospective teacher evidence and train–inference consistency

Conditioning the teacher on retrospective evidence is a training-time privileged-target design (App. A.3) that preserves train–inference alignment:

  1. Unchanged on-policy student states. Exposure bias stems from off-policy prefixes. In ReVuE, the student acts and is supervised on its own inference-time states and visible history hi,th_{i,t}; Ri\mathcal R_i enters only the frozen teacher to form qi,tRq^{\mathcal R}_{i,t}, matching Vision-OPD and GT-Privileged.
  2. Reachable visual states. Every image in Ai\mathcal A_i comes from the student's rollout group Gx\mathcal G_x on the same query, and the closed-set critic cannot invent crops—so qRq^{\mathcal R} never assumes unreachable evidence.
  3. Selective action revision without shortcutting needed tools. The Anchor prefers the trajectory's own earliest sufficient state at minimal tool cost (Eqs. 13–14): when earlier evidence suffices, lowering support for redundant calls is intended; when own acquisition failed (Ci=∅\mathcal C_i = \emptyset), sibling evidence pairs with Acquire to redirect where to crop (suppressing foreground / left; Fig. 18) rather than suppressing tool use. Conditioned on hi,th_{i,t}, the teacher does not shortcut to hindsight answers (Lucia, Fig. 3; GT-Privileged adds no gain).
  4. Adaptive test-time tool use. At inference (no critic or reflection), ReVuE beats all other plotted distillation baselines on HRBench-8K and V* Bench both with and without tools (Fig. 5, Sec. 5.2). It keeps high tool use where zoom-in is needed (83.7% VisualProbe; 42.91% → 44.66% vs. Vanilla OPD) while pruning redundant calls on TreeBench (64.7% → 15.8% vs. Vanilla OPD, +3.10% accuracy; moved from App. A.3 to Sec. 4.1).

Closing

We have added the self-teacher study, context ablations, and clarifications on localization, Eq. (8), and retrospective conditioning to the revised manuscript (Secs. 4.1, 4.3, 6–7; Apps. C.1–C.2 & C.6–C.7). Thank you again for the feedback, and we welcome further discussion.

AuthorsOpenReview ↗

Follow-Up to Reviewer oY2A

We sincerely thank you for the time and effort you have devoted to reviewing our work. In response to your valuable feedback, we have provided detailed responses and clarifications addressing each of the concerns raised.

As the discussion deadline is now less than two days away, we would greatly appreciate any further feedback on whether our revisions and clarifications have adequately addressed your concerns.

We remain fully committed to incorporating your suggestions to further improve the quality of our manuscript, and we look forward to your continued feedback and discussion.

Reviewer_ZyJnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes REVUE, an on-policy distillation method for visual agents. A critic compares student attempts at the same question, identifies useful visual evidence, and diagnoses failures in acquiring, reading, or grounding that evidence. These reflections guide the teacher, while changes in teacher token probabilities determine how distillation losses are weighted. Experiments on two model families and 11 benchmarks show improved category-average performance, alongside shorter responses and less frequent tool use.

Critical strengths

The distinction between finding evidence, reading it, and using it correctly is useful and well illustrated. More importantly, the paper goes beyond examples: it compares reflection and reweighting separately, tests impact-based versus random grouping, and checks critic diagnoses against human labels. The improvements across both model families are encouraging. I also appreciate that the authors report where reweighting hurts performance rather than presenting it as uniformly beneficial.

Critical weaknesses
  1. It is still difficult to separate the benefit of stage-specific diagnosis from receiving a richer explanation from a strong critic. GT-Privileged supplies only the answer, which is a substantially weaker form of guidance. The offline Anchor-only comparison helps, but a trained-student comparison would better establish how much the failure diagnosis contributes.
  2. The masking experiment needs clarification about loss normalization. With the stated weighting, masking the highest-impact 20% removes half the total loss weight, whereas masking the lowest-impact 20% removes about one eighth. Unless the remaining weights are renormalized, the comparison changes both token selection and overall loss scale.
  3. Several improvements are small, and the experiments use one random seed. This matters particularly for reweighting: it improves perception and math but slightly lowers the general-task average. Some indication of run-to-run variation would help establish which differences are reliable.
Questions for authors
  1. How does a student trained with Anchor-only teacher context compare with the full method? Does the Break Point diagnosis improve downstream performance beyond the offline teacher comparison?
  2. Are the retained losses renormalized after token masking? If not, does the ordering in Table 3 hold when total loss weight is matched?
  3. Have you repeated the main component ablation with another seed, particularly the reflection-only versus full-method comparison?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZyJn (1/2): Overview and W1 & Q1

We thank the reviewer for the thoughtful assessment and constructive questions. We ran new student-training experiments to address each point on Break Point attribution, loss normalization under token masking, and multi-seed reliability, and have incorporated all new results into the revised manuscript (highlighted in blue). Unless noted otherwise, all new runs train Qwen2.5-VL-7B on the same data, rollout budget, and training steps as ReVuE, and report sample-weighted category averages (%) for Perception, Math, and General, with full 11-benchmark tables added to Appendices C.1, C.2, and C.6.

Overview of new evidence

ConcernNew evidenceKey result
W1 & Q1 Contribution of Break Point beyond Anchor-onlyTrained student with Anchor-only contextBreak Point raises all three averages (+1.38% / +0.46% / +0.28%), more than doubling perception and math gains over Vanilla OPD
W2 & Q2 Loss normalization in token maskingMasking runs with retained weights renormalized to 1Mask low > Mask random > Mask top holds in all three categories; ReVuE beats all masking variants
W3 & Q3 Single-seed variance in component ablations3 random seeds (42, 43, 44) for Vanilla OPD, Reflection-only, and ReVuEReweighting improves Perception (+0.74%) and Math (+1.14%) on every seed, with no General drop (53.60% → 54.07%)

W1 & Q1: Contribution of Break Point diagnosis beyond Anchor-only guidance

Training a student with Anchor-only teacher context improves over Vanilla OPD, and adding the trajectory-specific Break Point diagnosis (ReVuE) raises downstream accuracy across all three categories—from 63.63% to 65.01% in Perception, 48.03% to 48.49% in Math, and 53.86% to 54.14% in General. Relative to Vanilla OPD, adding Break Point more than doubles the student's gain in Perception (+1.02% → +2.40%) and Math (+0.36% → +0.82%).

Trained-student comparison on Qwen2.5-VL-7B (category averages in %; Δ vs. Vanilla OPD)

MethodAnchorBreak PointPerceptionMathGeneral
Cold-start––60.7246.3852.68
Vanilla OPD––62.6147.6753.28
GT-Privileged––62.26 (−0.35)47.49 (−0.18)53.23 (−0.05)
Anchor-only✓–63.63 (+1.02)48.03 (+0.36)53.86 (+0.58)
ReVuE✓✓65.01 (+2.40)48.49 (+0.82)54.14 (+0.86)

Anchor-only uses the same critic outputs and impact-reweighting rule as ReVuE, keeping the Anchor's observed supporting images, verified visual fact, and grounding rule while removing the trajectory-specific Break Point diagnosis (all-wrong groups without an Anchor fall back to uniform clean-teacher OPD).

  • Downstream gain beyond rich reference guidance. GT-Privileged shows that the reference answer alone adds no gain, whereas Anchor-only already supplies verified visual facts and grounding rules. Adding Break Point still improves 9 of 11 benchmarks, led by Perception (+1.38% average; +2.38% on HRBench-8K, 71.62% → 74.00%, and +2.14% on VisualProbe, 42.52% → 44.66%), with Anchor-only higher only on V* Bench (84.29% vs. 82.20%) and InfographicVQA (80.35% vs. 80.27%).
  • Consistent with the offline paired comparison. On 311 paired rollouts (Appendix C.6, revised Table 16), adding Break Point to Anchor-only lifts offline teacher accuracy from 60.13% to 92.60%, doubling error corrections (62/139 → 124/139) while cutting regressions on initially correct answers nearly sixfold (47/172 → 8/172). Pinpointing the first failed stage helps the teacher correct errors without perturbing valid reasoning. We have added this trained-student comparison to Section 6 (Table 6a) and the full 11-benchmark breakdown to Appendix C.6 (Table 17).
AuthorsOpenReview ↗

Response to Reviewer ZyJn (2/2): W2 & Q2, W3 & Q3, and Conclusion

W2 & Q2: Loss normalization in token-masking comparisons

In the initial submission's Table 3 (and Appendix C.2), retained token losses were not renormalized after masking; when we renormalize retained weights to sum to 1 per trajectory, the ordering Mask low > Mask random > Mask top holds across all three categories. At α=0.2\alpha = 0.2 and λ=0.5\lambda = 0.5 (Eq. 9), per-token weights are 2.52.5 in High and 0.6250.625 in Low, so masking the top 20% removed 0.50.5 of total trajectory weight while masking the lowest 20% removed 0.2×0.625=0.1250.2 \times 0.625 = 0.125. To isolate token selection from loss scale, we reran all masking variants on Qwen2.5-VL-7B (same teacher, data, steps, and seed 42) with trajectory-level renormalization:

ℓi=∑t∈Tikeptwi,t ℓi,t∑t∈Tikeptwi,t,\ell_i = \frac{\sum_{t \in \mathcal{T}_i^{\text{kept}}} w_{i,t}\,\ell_{i,t}}{\sum_{t \in \mathcal{T}_i^{\text{kept}}} w_{i,t}},

matching the unit trajectory weight of ReVuE, Random select 20%, and Vanilla OPD.

Renormalized token-impact interventions on Qwen2.5-VL-7B (category averages in %)

MethodPerceptionMathGeneral
Base Model57.7746.3450.53
Cold-start60.7246.3852.68
Vanilla OPD62.6147.6753.28
Random select 20%63.6447.4254.10
Mask random 20% (w/ renorm.)63.5247.2452.58
Mask low 20% (w/ renorm.)64.4847.7453.02
Mask top 20% (w/ renorm.)62.0845.9852.43
ReVuE65.0148.4954.14
  • The Table 3 ordering holds under matched loss weight. Across all three categories, Mask low 20% > Mask random 20% > Mask top 20%. Mask low exceeds Mask top by +2.40% in Perception and +1.76% in Math (vs. +2.92% and +1.25% unnormalized); on General tasks, the margin is smaller (+0.59% over Mask top; +0.15% for Mask random over Mask top), consistent with the smaller sensitivity to position selection in Random select 20%.
  • Down-weighting outperforms hard dropping. Under unit trajectory weight, Mask low 20% scores below ReVuE on all three categories (64.48% vs. 65.01%, 47.74% vs. 48.49%, and 53.02% vs. 54.14%), while still exceeding Mask top 20% on all 11 benchmarks. Its +0.17% Perception edge in the initial Table 3 stemmed from unnormalized loss scale rather than harmful low-impact tokens, supporting soft reweighting over dropping. We have updated Table 3 (Section 6), Appendix C.2 (revised Tables 11–12), and the corresponding analysis, while retaining the unnormalized results in Appendix C.2 (Table 13) for reference.

W3 & Q3: Multi-seed reliability of the component ablation

We repeated the Qwen2.5-VL-7B component ablation (Vanilla OPD, Reflection-only, and ReVuE) with two new random seeds (43 and 44) alongside the default seed 42 (n=3n = 3 runs under identical settings). Impact reweighting improves Perception and Math over Reflection-only on every seed, and shows no General-task drop across seeds.

Three-seed component ablation across seeds 42, 43, and 44 (mean ± sample std, in %)

MethodPerceptionMathGeneral
RL Expert (frozen teacher)63.8947.5653.60
Vanilla OPD62.68 ± 0.5947.58 ± 0.2852.62 ± 0.57
Reflection-only (+Anchor & Break Point)64.15 ± 0.4547.75 ± 0.4353.60 ± 0.95
ReVuE (+impact reweighting)64.89 ± 0.1448.90 ± 0.7054.07 ± 0.06
  • Consistent end-to-end gains. Every ReVuE seed beats every Vanilla OPD seed in all three categories (64.74%–65.01% / 48.49%–49.71% / 54.02%–54.14% vs. 62.12%–63.30% / 47.27%–47.81% / 52.26%–53.28%), for mean gains of +2.21% in Perception, +1.32% in Math, and +1.45% in General.
  • Reliable reweighting gains in Perception and Math. Paired by seed against Reflection-only, reweighting is positive on all three seeds in Perception (+1.31%, +0.15%, +0.77%; mean +0.74%) and Math (+0.71%, +1.18%, +1.54%; mean +1.14%).
  • No systematic General trade-off. The −0.28% General dip in the initial Table 2 occurred only on seed 42 (54.14% vs. 54.42%); seeds 43 and 44 improve by +0.23% (54.04% vs. 53.81%) and +1.46% (54.02% vs. 52.56%), raising the three-seed mean from 53.60% ± 0.95% to 54.07% ± 0.06% with much lower variance. We have updated Table 2 and Section 6, and added the full per-seed, per-benchmark breakdown to Appendix C.1 (Table 10).

Closing

All three studies—the trained-student Anchor-only ablation, trajectory-renormalized token-masking runs, and three-seed component ablations (seeds 42, 43, 44)—have been incorporated into the revised manuscript (highlighted in blue). Thank you again for the constructive feedback. We hope these revisions resolve your remaining questions and welcome any further discussion.

AuthorsOpenReview ↗

Follow-Up to Reviewer ZyJn

We sincerely thank you for the time and effort you have devoted to reviewing our work. In response to your valuable feedback, we have provided detailed responses and clarifications addressing each of the concerns raised.

As the discussion deadline is now less than two days away, we would greatly appreciate any further feedback on whether our revisions and clarifications have adequately addressed your concerns.

We remain fully committed to incorporating your suggestions to further improve the quality of our manuscript, and we look forward to your continued feedback and discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Authors’ Thanks to Reviewer oY2A

Dear Reviewer oY2A,

Thank you for taking the time to read our rebuttal and follow up. Your question about the teacher’s role prompted the self-teacher experiment and strengthened the paper. Hearing back from you meant a great deal to us.

As the discussion draws to a close, we would be happy to clarify any remaining concerns. Thank you again for your thoughtful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.