- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies on-policy distillation (OPD) of multimodal LLMs. The authors show that current multimodal OPD fails to transfer how the teacher uses the image: the student's outputs converge to the teacher's, but the image regions behind its tokens stay wrong. They call this the evidence gap. Their method, SWIS, scores how much each token relies on each image patch. These scores form a joint token-patch distribution. SWIS adds one KL divergence between teacher and student on this distribution. By the chain rule, it splits into two terms: which tokens use the image, and where the evidence lands. No weighting between them is needed. Experiments cover 10 benchmarks, 2 student sizes, and 2 teacher setups. SWIS gains about 1.8 points of average accuracy over vanilla OPD. It also moves evidence onto the annotated regions, close to the teacher's level. A masking check that avoids the readout backs both claims.
Critical strengths
- The paper identifies a real gap in the field. Existing OPD methods match only the teacher's per-token output distribution, leaving how the image is used unsupervised. Cross-modal alignment is a central problem for current multimodal LLMs, so this question matters beyond distillation. The authors verify that the problem exists through a fairly detailed diagnostic analysis in the OPD setting.
- The experiments around the method are thorough.They show the gain and the bottleneck of each component: mass and placement terms separately and jointly (Tab. 2), the choice of readout (Tab. 3), the teacher's privileged input (Tab. 4), and the loss weight (Tab. F2).
- The paper is clearly written and easy to follow.The mass/placement framing is introduced early and used consistently from the diagnosis to the ablations, and most claims are tied to a specific figure or table.
Critical weaknesses
- The target quality is bounded by the teacher, and errors can propagate. The paper itself notes that "SWIS inherits where the teacher looks: its Energy-PG approaches the teacher's ... for right and wrong answers alike" (App. G.4). The teacher's own grounding is moderate (Energy-PG 33.8 vs. chance 14.9), so when the teacher attends to spurious regions, the loss teaches the student to do the same. Since the training data has ground-truth boxes, gating the evidence loss by teacher evidence quality seems worth exploring.
- The improvements in Table 1 are not clearly beyond noise.The paper claims "the best average accuracy in all four settings" and, at 4B, "the only method that improves on its vanilla recipe on all ten benchmarks" (Sec. 4.2). Yet on MMVP (n=300), a 2-point gain is about 6 questions; the 2B-OPD margin over the strongest baseline is 0.66 points; and vanilla OPD improves over the base model by only 0.21 points (Tab. 1). These claims are stronger than the margins support.
- The grounding gains may reflect optimization toward the metric.The abstract concedes the gap is closed "under the same readout that exposed it." The evidence loss, the Energy-PG metric, and the V verdict of the process evaluation ("V holds when the response's Energy-PG reaches τ", App. H) are all built from this readout, so the student's in-box share rises almost by construction. Consistently, at λ=0.2 Energy-PG reaches 33.4 while accuracy drops (Tab. F2). An evaluation the loss does not touch, such as an off-family attribution, would make the grounding claims convincing.
- Benchmark images overlap the training set. App. D states: "We did not remove benchmark images from this training set, which all compared methods share." The overlap is never quantified, and it is concrete: POPE is built on COCO images while the training set is largely COCO-derived, and Geometry3K is used in full and feeds MathVista. Because the evidence loss binds tokens to image regions, memorization may interact with evidence supervision rather than add to it, so sharing the data does not fully neutralize the comparison.
Taken together, these weaknesses suggest that the paper reliably shows the student can be made to look where the teacher looks; what is not yet established is that this transfer brings reliable gains on the task itself.
Questions for authors
- Why does better placement improve the task? The premise of the paper is that the student fails to inherit the teacher's cross-modal ability. Yet one could argue that both student and teacher largely exploit global image information, and the observed gap reflects only where the chosen readout concentrates its mass. Beyond the correlations in Figs. 6–7, please provide a logical account of why optimizing placement — making the evidence land in the annotated region — translates into task-level gains.
- Independent evaluation metrics. The evidence loss, the Energy-PG metric, and the process-evaluation verdict are all derived from the same readout, so part of the improvement may reflect optimization toward the metric itself. Please supplement an evaluation that the loss does not touch, for example re-scoring the trained student with an off-family attribution (patch ablation, LIME, or Grad-CAM), to rule out this misleading effect.
- Decontaminated comparison. Please remove the benchmark-overlapping portion from the evaluation and add the corresponding comparison experiments, so that the method's effect on the tasks is not distorted by images seen during training. The concrete overlaps are POPE–COCO and MathVista–Geometry3K. These three questions are central to my assessment; a convincing answer to each would raise my rating.
8 comments
Response to Reviewer 5WM2 (Part 2/4)
Q2. "The evidence loss, the Energy-PG metric, and the process-evaluation verdict are all derived from the same readout, so part of the improvement may reflect optimization toward the metric itself. Please supplement an evaluation that the loss does not touch, for example re-scoring the trained student with an off-family attribution (patch ablation, LIME, or Grad-CAM) [...]."
We ran the kinds of evaluation the review suggests: patch ablation, a perturbation test like LIME (Ribeiro et al., 2016); Grad-CAM (Selvaraju et al., 2017) with its high-resolution variant HiResCAM (Draelos & Carin, 2020); and the input gradient (Simonyan et al., 2014). None of them uses attention weights or Eq. (2).
Under patch ablation, HiResCAM and the input gradient, SWIS depends on the annotated region significantly more than vanilla OPD. On the share-based measures it covers 73–82% of the distance from vanilla OPD to the teacher, close to the 80% under Energy-PG.
(1) Patch ablation. On the same 2,951 held-out examples, we delete the visual tokens of the annotated region, of a random region of equal size, or of the whole image.
Table R2: Region deletion (Qwen3.5-4B, OPD). Drop: accuracy lost. Δ log p: log-likelihood of the model's own response lost per token (nats) without the region. Region share: Δ log p for the region over that for the whole image.
| Model | Acc. | Drop w/o region ↑ | Drop w/o random | Δ log p ↑ | Region share (%) ↑ |
|---|---|---|---|---|---|
| Teacher (9B) | 64.42 | 41.88 | 2.58 | 0.284 | 54.7 |
| Base Model | 57.40 | 14.94 | 1.93 | 0.125 | 29.8 |
| Vanilla OPD | 59.67 | 15.96 | 1.76 | 0.138 | 32.1 |
| SWIS (Ours) | 62.93 | 30.57 | 2.34 | 0.224 | 48.6 |
Without the region, SWIS loses 14.61 more accuracy points than vanilla OPD, while a random region makes no significant difference. The region share measures the same allocation as Energy-PG without the readout.
(2) Gradient-based attribution. We use Grad-CAM, HiResCAM, which keeps the gradient of every token instead of pooling it, and the gradient with respect to the pixels.
Table R3: Energy-PG (%) on the held-out split under attributions that the loss does not use (chance: 14.9).
| Model | Eq. (2) | Grad-CAM | HiResCAM | Input gradient |
|---|---|---|---|---|
| Teacher (9B) | 33.82 | 15.07 | 23.81 | 19.10 |
| Base Model | 24.70 | 14.89 | 18.98 | 16.64 |
| Vanilla OPD | 27.09 | 15.07 | 19.52 | 17.02 |
| SWIS (Ours) | 32.50 | 15.23 | 23.04 | 18.66 |
Grad-CAM stays within 0.4 points of chance for every model, the teacher included, so it cannot separate them. HiResCAM and the input gradient localize the region, and under them SWIS is above vanilla OPD by 3.52 and 1.64 points. The grounding gain therefore holds under evaluations that the loss never touches.
Revision. Tables R2 and R3 with their protocols are in Appendix J.3 (blue).
References
- Selvaraju et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. ICCV, 2017.
- Draelos & Carin. Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks. arXiv:2011.08891, 2020.
- Simonyan et al. Deep inside convolutional networks: Visualising image classification models and saliency maps. ICLR Workshop, 2014.
Response to Reviewer 5WM2 (Part 3/4)
W3. "The grounding gains may reflect optimization toward the metric. The abstract concedes the gap is closed 'under the same readout that exposed it.' The evidence loss, the Energy-PG metric, and the V verdict [...] are all built from this readout, so the student's in-box share rises almost by construction. Consistently, at λ=0.2 Energy-PG reaches 33.4 while accuracy drops (Tab. F2)."
The evaluations that do not use the readout are in Part 2/4.
The in-box share does not rise by construction: it follows the teacher's evidence in both directions. In the OPD setting, the loss never uses the box. It matches the student's evidence to the teacher's, while Energy-PG (Wang et al., 2020; applied to MLLMs by Gong et al., 2026) compares the evidence with a human annotation. The share rises by 12.7–13.9 points on the questions where the teacher's evidence hits the box and by only 2.9–3.7 where it does not (Table G2). It falls to 21.84, below the base model's 24.70, when the self-teacher is given a random box (Table 4 of the paper).
- λ = 0.2. As λ grows, Energy-PG approaches the teacher's 33.82 without exceeding it (now stated in Appendix F, blue). A larger weight has little evidence left to transfer, while the next-token loss, on which accuracy depends, loses relative weight.
- The V verdict is the only one of the four verdicts that uses the readout. Its counterpart without the readout, the region share of Table R2 (Part 2/4), gives the same picture as V in Fig. 7, and Appendix H now says so (blue).
- The abstract now ends: "... and narrows the evidence gap, under the readout that exposed it and under a region-deletion test that does not use it." (blue)
References
- Wang et al. Score-CAM: Score-weighted visual explanations for convolutional neural networks. CVPR Workshops, 2020.
- Gong et al. Saliency-R1: Enforcing interpretable and faithful vision-language reasoning via saliency-map alignment reward. CVPR, 2026.
Response to Reviewer 5WM2 (Part 1/4)
The evidence loss makes the student read the part of the image that the teacher reads, so the gain appears on the questions that a prior cannot answer. In these tasks the answer is read from a small part of the image. Next-token matching gives full credit to a token produced without reading that part, from a prior or from the context, and such a token is right only where the prior holds. All four are measured in accuracy, not with the readout.:
Table R1: Accuracy (%) on the questions that the base model answers correctly from a blank image (N = 3,881) and on the others (N = 8,994); gain over the vanilla recipe in parentheses. Qwen3.5-4B, seven benchmarks; the three binary ones are excluded, since a guess is right half of the time.
Under OPD, SWIS gains +2.25 points on the image-needed questions and +0.20 on the others; under OPSD the direction is the same. VA-OPD, the strongest OPD baseline, gains no more on the image-needed questions than on the others, so the concentration is a property of SWIS and not of the split.
Revision. This account and Table R1 are in Appendix J.6, and Analysis 3 of Sec. 5 refers to it (blue).
SWIS transfers the teacher's evidence by design, and Appendix G.4 reports what this implies. On the questions the teacher answers wrongly, SWIS passes on fewer errors than vanilla OPD. Where the teacher looks off the box, the student's evidence moves little, and its accuracy is above vanilla OPD (Table G2 of the paper).
Revision. The gate and a second direction, refreshing the self-teacher from the improved student, are added to Appendix I (blue).