- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the effects of different forms of privileged information (PI) on multimodal on-policy self-distillation (OPSD). The authors introduce two metrics, "PI Access Gain" and "PI Effectiveness," to characterize the discrepancy between the direct performance gains obtained from PI and its effectiveness as a distillation supervision signal. Through experiments across four task types, the authors demonstrate that greater PI Access Gain does not necessarily translate into higher PI Effectiveness, and that the effectiveness of different PI conditions is highly task-dependent. The paper further analyzes how students trained under different PI conditions attend to and rely on task-relevant visual regions. Finally, the authors propose a validation-based, task-level PI routing strategy as a heuristic for selecting appropriate privileged information for different tasks.
Critical strengths
- The paper addresses a meaningful research question in multimodal OPSD by distinguishing PI Access Gain from PI Effectiveness, highlighting that stronger privileged information does not necessarily provide more effective supervision for PI-free students.
- The experiments systematically compare different PI conditions across four multimodal tasks using matched No-PI OPSD baselines, revealing task-dependent differences in PI effectiveness and providing valuable empirical insights.
- The paper is clearly structured and well presented. The Visual Overlap and Matched Occlusion analyses offer complementary perspectives on how different PI conditions influence students' visual attribution and reliance on task-relevant evidence.
Critical weaknesses
The paper's central observation is that the direct performance gains from PI do not necessarily translate into corresponding distillation benefits. However, prior work such as ViCuR has already highlighted the limitations of answer-based privileged information and demonstrated that visually grounded privileged supervision can lead to more effective distillation. Although this paper further defines PI Access Gain and PI Effectiveness and extends the investigation across multiple tasks and PI conditions, it remains unclear whether these results provide substantially new insights or mainly offer broader empirical validation of existing findings. Furthermore, the proposed PI Routing strategy appears to be more of a validation-based heuristic for selecting task-specific PI conditions. While it brings some performance improvements, it also requires evaluating different PI conditions on a validation set in advance to determine the routing configuration. The trade-off between performance gains and computational cost therefore deserves further discussion.
Regarding differences in PI Effectiveness, the analyses in Section 4 provide some insights into the potential mechanisms behind Hint-based OPSD but do not sufficiently explain why different PI conditions lead to different distillation outcomes. In Figure 2, apart from Hint, the Visual Overlap distributions of other PI conditions and baselines do not exhibit noticeable differences that correspond to their varying distillation gains. This raises questions about whether Visual Overlap can effectively explain differences in PI Effectiveness or mainly reflects the specific characteristics of Hint as localized visual privileged information. Moreover, other baselines in Figure 3 exhibit similar sensitivity to target-region occlusion as the Hint-trained model. Although this experiment demonstrates the models' reliance on task-relevant visual evidence, such behavior is largely expected in fine-grained perception tasks and does not sufficiently explain why different PI conditions produce different distillation outcomes. Overall, these analyses primarily characterize differences in student visual behavior rather than providing a convincing mechanistic explanation for the paper's central findings.
Questions for authors
-
How does the paper's central insight differ from ViCuR? What new insights does it provide beyond broader empirical validation across different PI conditions and tasks?
-
Could the authors more clearly articulate the key findings and claims of the mechanistic analyses in Section 4, and explain how they support the paper's central insight regarding PI Effectiveness?
8 comments
Response 2
W2: Regarding differences in PI Effectiveness, the analyses in Section 4 provide some insights into the potential mechanisms behind Hint-based OPSD but do not sufficiently explain why different PI conditions lead to different distillation outcomes. [...]
Overall, these analyses primarily characterize differences in student visual behavior rather than providing a convincing mechanistic explanation for the paper's central findings.
Q2: Could the authors more clearly articulate the key findings and claims of the mechanistic analyses in Section 4, and explain how these analyses support the paper's central insight regarding PI Effectiveness?
We thank the reviewer for this thoughtful comment. We would like to clarify that the purpose of Section 4 is not to establish a complete causal mechanism explaining the distillation differences across all PI conditions, but rather to investigate, from a student-side behavioral perspective, what visual evidence utilization behaviors are retained by PI-free students after training under different PI conditions. As a fine-grained perception task, FGP requires the utilization of visual evidence from target regions to answer correctly. Accordingly, we conduct two complementary analyses, Visual Overlap and Matched Occlusion, to examine whether students' visual attribution concentrates on task-relevant regions and whether the visual evidence in these regions actually affects model predictions.
1. Figure 2: Visual Attribution Reflects Differences in PI Modalities.
In FGP, ANSWER, TRAJECTORY, and EXPERIENCE provide textual PI, whereas HINT provides visual crops centered on question-relevant regions. Since Visual Overlap measures the concentration of a model's visual attribution over image regions, it primarily reflects the student's spatial localization behavior.
Specifically, ANSWER, as the most direct form of textual PI, provides the teacher with the correct answer but does not explicitly indicate where the supporting visual evidence is located in the image. HINT, in contrast, directly provides localized visual evidence. Although students trained with ANSWER also need to use the image to answer correctly, this does not necessarily mean that they develop the same target-region localization behavior. Our results show that, among the evaluated PI conditions, only HINT exhibits a clear enhancement in target-region attribution. This highlights the advantage of visual HINT over textual PI in promoting the target-region localization behavior required by FGP.
2. Figure 3: Target-Region Occlusion Verifies the Actual Impact of Visual Evidence on Predictions.
We note that sensitivity to target-region occlusion is expected for models that rely on task-relevant visual evidence in FGP. Our analysis aims to examine whether the relevant visual regions indeed contain evidence that affects model predictions. It complements Figure 2: while Figure 2 examines whether students attribute their visual attention to the target region during inference, Figure 3 further verifies the importance of these target regions for answering the questions correctly.
3. Overall Findings and Their Relationship to PI Effectiveness.
Importantly, our OPSD objective matches the next-token distributions between the teacher and student, without explicitly distilling or aligning Attention Maps. Therefore, the visual attribution patterns analyzed in Section 4 are not directly supervised during training, but are behaviors observed after OPSD training. Furthermore, all analyses are conducted on the held-out FGP test set, where students receive only the original images and questions, without access to any PI. Thus, we examine the visual behaviors actually retained by the resulting PI-free students after training under different PI conditions.
The relevance of these analyses to PI Effectiveness is that they provide a complementary characterization of the learning differences induced by different PI conditions from a student-side behavioral perspective. Figure 2 reveals differences in students' visual attribution behaviors, while Figure 3 examines the actual impact of the corresponding visual evidence on predictions. Considering the requirements of FGP, these results suggest that visual HINT, compared with the evaluated textual PI conditions, is more effective in promoting the target-region localization behavior required by the task. This also indicates that even when the teacher has access to the correct answer provided by ANSWER, it does not necessarily mean that the student will develop the same visual evidence utilization behavior as a student trained with visual HINT after OPSD.
Therefore, Section 4 characterizes the differences in student-side learning outcomes associated with different PI conditions, providing complementary evidence for understanding the transfer of PI benefits from teacher-side information access to student-side learning gains.
A friendly reminder
Dear Reviewer t7jy,
Thanks again for your valuable comments and precious time. We genuinely hope you could have a look at the new results and clarifications and kindly let us know if they have addressed your concerns. We would appreciate the opportunity to engage further if needed.
Best,
Authors of Paper 16735
Response 1
Dear Reviewer t7jy,
We sincerely thank you for your careful review and constructive feedback. We particularly appreciate your concerns regarding the novelty, mechanistic insights, and practical implications of our work. These comments have helped us further clarify our contributions and their distinctions from prior work. We respond to your concerns below.
1. Clarifying the Difference from ViCuR (W1, Q1)
We would first like to clarify an important detail about ViCuR: the privileged information provided to its teacher is actually a textual Visual Cue (denoted as S), rather than visual input itself. These cues describe question-relevant visual evidence, whose underlying information remains available in the student's original image. ViCuR further introduces a Sink-Token Cross-Attention module, allowing the student to aggregate relevant evidence from its own visual tokens into an internal Cue Representation.
The key difference between the two works lies in their research focus: ViCuR investigates the recoverability of textual cues grounded in visual evidence and their student-side implementation, whereas we systematically study how different forms of PI translate teacher-side access benefits into student-side learning gains, and how this transfer relates to student behavior and task requirements.
2. Insights Beyond Broader Empirical Comparisons (W1, Q1)
Our study provides three complementary insights.
First, PI Access Gain does not necessarily predict PI Effectiveness. We distinguish the direct performance benefit obtained by a fixed Base Model accessing PI from the incremental learning gain retained by the PI-free student after OPSD, relative to a matched No-PI baseline. Our FGP results show that ANSWER yields a greater PI Access Gain, whereas HINT achieves higher PI Effectiveness. This demonstrates that the direct benefits of PI access to the teacher do not necessarily translate into greater learning gains for the student.
Second, different PI conditions are associated with differences in student-side visual behavior. Through Visual Attribution and Matched Occlusion analyses, we investigate how students trained under different PI conditions utilize task-relevant visual evidence. These analyses provide a complementary characterization of the resulting students' behavior beyond final accuracy.
Third, PI Effectiveness is task-dependent rather than intrinsic to a particular PI format. Our cross-task analysis reveals that the relative effectiveness of different PI conditions changes with task requirements. For example, visual HINT is particularly beneficial for fine-grained perception, whereas ANSWER provides greater student-side gains in knowledge-intensive perception (KIP) and multi-hop spatial reasoning (MSR). In KIP, crop-based HINT helps the teacher localize the target region but does not provide the knowledge needed to interpret the identified content. In MSR, even a Spatial HINT specifically designed for the task does not necessarily yield greater student gains. These findings suggest that PI selection should consider whether the information provided matches the actual requirements of the target task, rather than assuming a generally superior PI format.
3. Practical Implications and Limitations of PI Routing (W1)
Building on our cross-task analysis, we further explore Task-level PI Routing as a preliminary investigation of the potential of selecting PI according to task requirements. The current routing strategy is a small-scale proof of concept conducted on subsets of the four tasks. We consider three candidate conditions (No-PI, ANSWER, and HINT) and use 200 validation examples per task to select the PI condition.
We acknowledge that the current strategy introduces additional computational costs. Importantly, the current PI Routing serves as a preliminary downstream application of our finding regarding the compatibility between task requirements and PI. We hope to further explore more computationally efficient PI selection methods and investigate the trade-off between performance gains and computational costs in practical applications.