Rethinking Privileged Information Effectiveness in Multimodal On-Policy Self-Distillation

On-policy self-distillation (OPSD) can provide the teacher with privileged information (PI) unavailable to the student, potentially improving teacher supervision during training. However, it remains unclear whether PI that provides a larger direct benefit to the teacher also leads to larger gains that persist in the PI-free student. We study this question in multimodal OPSD by explicitly separating two quantities: PI access gain, the target-task performance gain obtained when a fixed model is given direct access to PI, and PI effectiveness, the target-task performance gain retained by the PI-free student after OPSD relative to a matched no-PI baseline. In controlled fine-grained perception experiments, these quantities are not consistently aligned across PI conditions: Answer yields a larger PI access gain than Hint, whereas Hint achieves higher PI effectiveness. Student-side analyses further reveal condition-specific differences in the use of task-relevant visual evidence, with Hint showing the clearest target-region localization and selective sensitivity to target-region corruption. Extending the analysis across tasks further shows substantial variation in PI effectiveness across target tasks and concrete PI conditions. Overall, our results show that greater PI access gain does not necessarily translate into greater PI effectiveness, and that PI effectiveness is task- and condition-dependent and not intrinsic to a PI form. Motivated by this task dependence, a simple task-level PI routing strategy improves aggregate held-out performance over uniform PI baselines.