- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Role-OPD, an on-policy self-distillation method for multimodal LLMs. A frozen copy of the model serves as the teacher and receives a privileged textual caption during training. The paper makes two claims: 1) An “evidence-utilization gap.” Models answer far more accurately from targeted textual descriptions than from the original images, which suggests textual privilege is promising for self-distillation. 2) A “privileged-context attribution problem.” When privileged text is placed in the user turn, the student learns to produce artifacts such as “based on the description,” even though no description is available at inference time. To address the second issue, Role-OPD places the caption in an intermediate assistant turn, framing it as the teacher’s own visual understanding. Experiments on Qwen3.5-4B/9B across six fine-grained, high-resolution benchmarks and five held-out benchmarks show improvements over several baselines.
Critical strengths
-
A concrete problem with a simple and effective solution. The paper identifies the privileged-context attribution problem, a practically relevant failure mode of textual privilege, and makes it measurable through the PCAR metric. It then resolves the problem by relocating the privileged caption to an assistant turn, which adds no inference overhead. This is the main reason I lean toward acceptance: the problem is real and clearly defined, and the fix is simple, cost-free, and easy to adopt in other self-distillation frameworks.
-
Strong controls for the role-aware design. The ablation separating privilege form from placement is clear. The additional controls, an explicit no-attribution instruction and a three-turn all-user context, show that the gain comes from role assignment itself, not from suppressing attribution phrases or from the multi-turn structure. This is consequential because it makes the paper’s core claim about where to place privilege well supported, even though other parts of the evaluation are less rigorous.
Critical weaknesses
-
The motivation experiment does not support its conclusion. The captions are generated with access to the ground-truth answer, so the near-ceiling accuracy in the textual setting reflects answer leakage rather than better evidence utilization. The visual setting, by contrast, uses the full image without localization such as cropping. The comparison therefore conflates modality with answer leakage. This matters for my assessment because it undermines the paper’s first contribution, the claim that textual privilege is superior to visual privilege, and with it the rationale for choosing captions in the first place.
-
The gain over Vision-OPD cannot be clearly attributed to the role-aware design. The privileged captions come from a much larger, answer-aware model, so Role-OPD receives richer privileged content than the baselines. It is also unclear whether the on-policy baselines were trained with the same distillation recipe, even though the paper itself shows that such choices substantially affect accuracy.
Questions for authors
-
What fraction of the privileged captions contain the answer string or the correct option verbatim? How does the evidence-utilization comparison change if the visual setting uses evidence-centered crops and the textual setting uses captions generated without the answer?
-
How does Role-OPD perform when the captions are generated by the student model itself, rather than by Qwen3.5-397B?
-
Which baseline results were taken from prior work, and were OPSD and Vision-OPD trained with the same objective, teacher update strategy, and data as Role-OPD?
4 comments
[W2/Q2/Q3]: Attribution of performance gains
We thank the reviewer for requesting this clarification. We clarify that Vision-OPD also uses Qwen3.5-397B to construct its privileged information. Therefore, the original comparison does not provide Role-OPD with privileged information from a stronger generator than the baseline. We further evaluate Role-OPD using captions generated by the 4B student model itself. As shown below, it achieves an average performance of 77.79, outperforming the Vision-OPD result of 76.42 reported in our paper.
Performance of Role-OPD (4B) with external and self-generated captions.
| Caption Generator | V$^\star$ | ZoomBench | HR-Bench 4K | HR-Bench 8K | MME-RealWorld EN | MME-RealWorld CN | Avg. |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | 91.62 | 60.71 | 85.50 | 81.25 | 75.35 | 72.33 | 77.79 |
| Qwen3.5-397B | 92.15 | 61.18 | 85.75 | 81.62 | 75.61 | 72.62 | 78.16 |
The results for the Closed-Source Models (Single Forward Pass) and Open-Source Models (Single Forward Pass) groups in Table~1 of our paper are taken from the Vision-OPD. We will clarify their provenance in the revised manuscript. In our main experiments, OPSD and Role-OPD use the same training objective, teacher-update strategy, and training data. Vision-OPD uses the same training objective and data as Role-OPD but retains the best teacher-update strategy reported in its original work. We further report results under matched settings, as shown below.
Performance of OPSD, Vision-OPD, and Role-OPD under matched settings.
| Method | Param Size | V$^\star$ | ZoomBench | HR-Bench 4K | HR-Bench 8K | MME-RealWorld EN | MME-RealWorld CN | Avg. |
|---|---|---|---|---|---|---|---|---|
| OPSD | 4B | 86.39 | 54.32 | 82.38 | 79.62 | 74.81 | 72.52 | 75.01 |
| Vision-OPD | 4B | 88.48 | 60.47 | 82.88 | 79.50 | 74.72 | 70.66 | 76.12 |
| Role-OPD | 4B | 92.15 | 61.18 | 85.75 | 81.62 | 75.61 | 72.62 | 78.16 |
Thank you for the detailed responses. My concerns, especially those regarding the motivation, experiments, and attribution of the gains, have been addressed. Therefore, I will keep my positive score.
Dear Reviewer EnBF,
We are glad that our response has addressed your concerns. We sincerely appreciate your time and thoughtful feedback.
Best regards
We thank the reviewer for the constructive comments and suggestions. We hope our point-by-point responses below address your concerns.
[W1/Q1]: Clarification of motivation experiments
We thank the reviewer for requesting this clarification. We clarify that none of our captions contain explicit answer statements such as “The answer is A. Red” or correct-option labels. We inspected all captions and found that the answer string appears only as part of a description of visual facts rather than as a standalone answer label. The original images already contain the visual evidence needed to determine the correct answer. We use the ground-truth answer to guide caption generation to construct textual evidence sufficient to support answering the question, rather than to directly provide an answer label to the target model. The captions also contain question-irrelevant image content, so the target model must still identify and utilize the relevant evidence. Without answer guidance, the caption generator may omit critical evidence or introduce perceptual errors, conflating evidence sufficiency with the target model’s ability to utilize that evidence. Our original experiment therefore compares the utilization of answer-sufficient evidence presented in visual and textual forms, and our conclusion is restricted to this evidence-sufficient setting.
Following the reviewer’s suggestion, we conduct additional experiments using the evidence-centered crops provided by the dataset and captions generated without access to the ground-truth answer. The evidence-centered crops ensure that the visual input contains and localizes the evidence needed to answer the question. Nevertheless, their question-answering accuracy remains substantially lower than that obtained from answer-guided captions. We also manually inspect the captions generated without answer access and find that only 61.88% contain sufficient evidence to answer the corresponding question. This confirms that answer guidance primarily ensures evidence sufficiency. Despite this disadvantage, captions generated without answer access still achieve higher average accuracy than full-image inputs.
Question-answering accuracy (%) under different visual and textual evidence settings.