- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work, the authors propose REVUE, an on-policy distillation method for visual agents. By comparing multiple student rollouts, REVUE employs a critic model to construct visual-evidence reflections: it summarizes the correct evidence and reasoning from successful rollouts and diagnoses the first failed stage (Acquire, Read, or Ground) for incorrect ones. These reflections serve as additional context for the teacher to provide more targeted supervision. A token-level reweighting loss is further applied, assigning higher weights to positions where the reflections most change the teacher's predictions. Experiments on 11 benchmarks and two models demonstrate the effectiveness of the proposed method.
Critical strengths
-
The paper is well-organized and easy to follow.
-
The proposed method is straightforward and intuitive. The Acquire/Read/Ground breakdown is well-motivated by the error analysis and illustrated with clear case studies. The critic is used only at training time, so there is no extra inference cost, and the trained students produce shorter responses with fewer tool calls.
-
The experiments cover two model families and 11 benchmarks and show consistent gains over the evaluated OPD baselines. The ablations are comprehensive.
Critical weaknesses
-
It is unclear whether the improvement comes from the extra information provided by the critic model and the reference answer. This also makes the comparison with other baselines unfair, since the proposed method effectively has a stronger “teacher” providing context, plus extra compute. GT-Privileged is not an information-matched control either. The results showing that the student surpasses the teacher also indicate that the improvements do not come purely from distillation from the RL expert.
-
When reproducing the baseline methods, the authors replace the teacher with RL experts, which does not align with the original papers. Additionally, some reproduced numbers are even lower than those of cold-start SFT; therefore, these numbers say little about how the original methods would perform.
-
The improvements on some benchmarks are marginal, but no statistical results are reported.
-
All experiments are conducted using models that are not current-generation. It is unclear whether the proposed method remains beneficial when the student’s Acquire/Read error rates are lower and the model natively supports agentic image operations (e.g., Qwen3.5). In addition, GPT-4o is an outdated closed-source reference, so the claim that both students exceed GPT-4o is not very informative.
Questions for authors
-
If the RL expert and the critic are available, what is the purpose of distilling into a same-size student? How does REVUE compare with using the critic’s reflections as RL rewards and continuing RL on top of the expert under matched compute?
-
What is the number of student rollouts per sample for the OPD baselines? Are they matched to REVUE?
-
How much of the gain remains if the critic does not see the reference answer or if a much smaller critic is used?
6 comments
Response to Reviewer kdnJ (2/5): Q3 and W1 (continued)
Q3: How much gain remains without the reference answer, or with a much smaller critic?
Removing both the reference answer and per-trajectory correctness labels from the critic retains 92% of ReVuE's perception gain (+2.20% vs. +2.40%). Replacing the 397B critic with Qwen3.5-122B-A10B (10B active parameters) still matches or exceeds every baseline in Table 1 and the RL Expert teacher across all three categories.
(a) Reference answer × diagnosis (Perception / Math / General in %; Δ vs. Vanilla OPD)
| No diagnosis | With diagnosis | |
|---|---|---|
| No answer | Vanilla OPD: 62.61 / 47.67 / 53.28 | Answer-free critic: 64.81 (+2.20) / 47.60 / 53.02 |
| With answer | GT-Privileged: 62.26 / 47.49 / 53.23 | ReVuE: 65.01 (+2.40) / 48.49 (+0.82) / 54.14 (+0.86) |
Answer-free critic: both the reference answer and per-trajectory correctness labels are removed; the critic sees only the query, student trajectories, and their observed images.
- Visual verification needs no answer key. By checking visual facts directly against the images students observed, the answer-free critic retains 92% of the perception gain.
- Math and general reasoning require both target and diagnosis. Catching a Ground-stage error needs a verifiable target, yet giving the answer directly to the teacher yields no gain (GT-Privileged); the answer helps only when converted into a failure diagnosis.
- Shared training-set labels. Reference answers come from the standard Thyme-RL training split already used to train the RL Expert teacher and filter RFT demonstrations.
(b) Smaller critic. Stage-diagnosis reliability can be audited on a few hundred rollouts before training, following the agreement protocol in Appendix B.6 (revised Table 9):
| Critic | vs. Gemini 3.8 Flash | vs. GPT-5.6 Luna |
|---|---|---|
| Qwen3.5-397B-A17B (paper) | 93.21% / κ 0.892 | 92.80% / κ 0.886 |
| Qwen3.5-122B-A10B | 92.26% / κ 0.877 | 91.44% / κ 0.865 |
| Qwen3.5-35B-A3B | 81.82% / κ 0.715 | 81.11% / κ 0.708 |
Qwen3.5-122B-A10B closely tracks the 397B critic in stage agreement, whereas 35B-A3B drops by over 11%. Training ReVuE with the 122B critic reaches 64.07% / 48.13% / 53.66% (+1.46% / +0.46% / +0.38% over Vanilla OPD), matching or exceeding every Table 1 baseline and the RL Expert on all three averages (added to Section 6 and Appendix C.8, Table 19).
W1 (continued): Why can the student surpass the RL Expert?
ReVuE distills from the RL Expert conditioned on the reflection (, Eq. 10) rather than the unconditioned expert (). Surpassing the unconditioned policy is the standard objective of privileged-context distillation—Vision-OPD, for example, trains a student to outperform its own unprivileged policy. Part 1/5 shows that unstructured critic feedback recovers under half of this gain, and Part 3/5 shows that ReVuE with a frozen self-teacher (no RL Expert) still surpasses the RL Expert on Math (48.32% vs. 47.56%).
W1 (continued): Extra compute
- Training-time only. Critic inference adds roughly 70 GPU-hours of service time to a 635 GPU-hour run (App. A), or 10.8% of mean step time; test-time inference cost is unchanged.
- Compute-matched control.
Critic + generic contextin Part 1/5 makes the exact same critic calls yet recovers less than half of the perception and math gains. - Comparable teacher forward cost. ReVuE queries the teacher twice per position (, ), matching the two-view teacher passes in V-Zero and VAD.
Response to Reviewer kdnJ (3/5): W2 and Q2
W2: Baselines were reproduced with an RL-expert teacher rather than their original settings
Table 1 uses the shared RL Expert across all OPD methods to hold teacher capacity constant and isolate the supervision signal. To evaluate the original configurations directly, we reran Vision-OPD and VAD with their original papers' self-teacher settings under the same data and training steps.
Original-setting reproductions (category averages in %)
| Method | Teacher | Perception | Math | General |
|---|---|---|---|---|
| Cold-start | – | 60.72 | 46.38 | 52.68 |
| Vision-OPD | RL Expert (Table 1) | 59.98 | 45.48 | 52.08 |
| Vision-OPD | original: EMA self-teacher | 58.02 (↓) | 44.76 (↓) | 51.54 (↓) |
| VAD | RL Expert (Table 1) | 62.08 | 45.62 | 53.65 |
| VAD | original: frozen initial model | 60.83 (↓) | 45.30 (↓) | 51.53 (↓) |
| ReVuE | frozen self-teacher | 61.97 | 48.32 | 52.71 |
- Using the RL Expert strengthened the baselines. Under their original self-teacher setups, Vision-OPD and VAD score lower on all six category averages than the RL-Expert versions in Table 1.
- ReVuE remains best without an RL Expert. With the same frozen initial model as VAD's original setup, ReVuE leads on all three averages, and its self-teacher Math score (48.32%) exceeds every method in Table 1, including the RL Expert (47.56%).
Why some baselines fall below cold-start. Vision-OPD and VAD were designed for single-pass VQA without tool calls, building privileged views (evidence-centered crops or evidence-removed views) from the raw input image. In multi-turn agentic reasoning, the student generates new visual states through its own tool calls, so static views of the initial image often mismatch what the student observed. ReVuE instead selects its Anchor from images observed within the students' own trajectories (added to Section 6, Table 6b, and Appendix C.7, Table 18).
Q2: Number of student rollouts per sample for the OPD baselines
All OPD methods use M = 8 student rollouts per query, matched to ReVuE, with the same batch size, dataset, and step budget. This matches VAD's original configuration (8 rollouts per prompt), and RFT likewise samples 8 RL-Expert trajectories per prompt. ReVuE uses no extra rollout budget (clarified in Section 5.1, Appendix B.1, and Appendix B.3, Table 8).
Response to Reviewer kdnJ (1/5): Overview and W1
We thank the reviewer for the detailed and constructive comments. We ran new experiments to address each point on supervision attribution, baseline configurations, multi-seed consistency, current-generation models, and RL alternatives, and have incorporated all new results into the revised manuscript (highlighted in blue). Unless noted otherwise, all new runs train Qwen2.5-VL-7B on the same data and training steps as ReVuE, and report weighted category averages (%) for Perception, Math, and General, with full 11-benchmark tables added to Appendices C.1 and C.6–C.9.
Overview of new evidence
GT-Privileged tests only whether giving the reference answer directly to the teacher suffices. To match ReVuE's full information budget, we add two controls using the same critic, the same reference answer, and the same number of critic calls. Even when the critic writes per-trajectory feedback, the teacher recovers less than half of ReVuE's perception and math gains: the advantage comes from structuring the diagnosis around the first failed stage and converting it into token-level supervision, rather than from raw critic information.
Information-matched controls (category averages in %; Δ vs. Vanilla OPD)
In
Critic + generic context, the critic sees the same trajectory group and reference answer and writes unconstrained natural-language feedback—a shared summary of how successful rollouts solved the task plus trajectory-specific error feedback—omitting only ReVuE's stage labels, Anchor evidence, and Break Point; the teacher distills with uniform token weights.