On-Policy Visual Evidence Distillation

Visual agents solve problems by interleaving reasoning with image operations, and on-policy distillation (OPD) provides guidance from a strong teacher on student-generated interaction trajectories. However, image operations change the evidence available for subsequent reasoning, so local errors in evidence acquisition ($Acquire$), reading ($Read$), or answer grounding ($Ground$) can propagate through the trajectory and lead to incorrect answers. Existing multimodal OPD methods primarily construct or contrast auxiliary views of the original image to strengthen supervision, without explicitly modeling the connections between student actions, resulting observations, and subsequent reasoning. This limits their ability to provide corrections tailored to different failure stages. We introduce $\underline{\mathbf{Re}}\text{flection on }\underline{\mathbf{V}}\text{is}\underline{\mathbf{u}}\text{al }\underline{\mathbf{E}}\text{vidence}$ ($\mathbf{ReVuE}$), an on-policy distillation method for visual agents. $\mathbf{ReVuE}$ compares multiple student-generated trajectories for the same query, summarizes the observed visual evidence, and diagnoses the first failure across the $Acquire$, $Read$, and $Ground$ stages. The resulting reflections provide training-time context for the teacher. We group and reweight token-level distillation losses according to how strongly these reflections affect the teacher's predictions. This design translates trajectory-level evidence diagnosis into targeted token-level supervision, guiding students to improve their visual evidence acquisition and reasoning. Across 11 benchmarks spanning the Qwen2.5-VL and InternVL3.5 model families, $\mathbf{ReVuE}$ outperforms all evaluated OPD baselines in weighted-average scores for perception, mathematical reasoning, and general tasks. $\mathbf{ReVuE}$ also reduces redundancy in reasoning and tool calls while improving tool-call accuracy and task accuracy.