- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on On-Policy Self Distillation with privilege information. Notably, traditional approach consider privilege information to be the answer to the current question. In this work, this paper consider changing the privilege information to a fully derived problem from a different question. Throughout this process, this paper shows that improvement are still observed despite the absence of the answer in the privileged information given to the teacher during self distillation.
Critical strengths
The problem definition is particularly interesting and relevant. Especially, the authors claim that current evaluation mostly rely on accuracy gain, leaving little to no explanation as to where this gain comes from. Therefore, the motivation is sound and interesting.
Critical weaknesses
- Despite criticizing current literature for the lack of interpretation beyond accuracy (l.77-78), I am unsure that this study is much different. Notably:
- There is a discrepancy of performances gain between reasoning and non reasoning model performances; which is never discussed (see Table 1). Table 11 in the appendix confirms that when the model is reasoning, most conclusions of the paper are weakly supported by the experiments.
- Trivial context (1+1) makes self distillation decrease performances from around 30 to 5 (see table 3). Why is that? This should be discussed and analyzed.
- Why would OP2SD work better that OPSD? This should also be discussed.
- The breath of experiments seems a bit limited to support all claims. More variety in datasets, distillation policies, thinking and non-thinking models might help support the claims. It feels to me that the paper tells an intuitively convincing story but I am unsure that this generalizes.
Questions for authors
- What are the attentions maps for different contexts?
- What are the performances of the teacher for different contexts?
- What is the impact of using thinking vs non-thinking on the teacher output and why is the proposed method not working as well when the teacher is thinking? Does that mean that the current claim should be narrowed to non-thinking models?
- Can you clarify how 'context-induced teacher behavior' (l 166) is not just target-agnostic privilege information? It seems that the current presentation opposes target-specific privilege information to context-induced teacher behavior (see title), but it looks to me that what this paper actually studies is the origin of the privileged information rather than its existence.