- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper questioned the long standing problem of topk. it answers an interesting and important question that whether we can have better accuracy with less experts. it proposed to use KL from the teacher and hold the accuracy of the student and in the meantime reduce the number of experts.
Critical strengths
- It makes sense reducing the number of experts will not impact the scores in certain tasks which I believe.
- And it's clever to have a teacher model to help the student model
Critical weaknesses
- it's using teacher model and did test-time training so it's unclear to me where the gain coming from pruning or test time training (this is my main concern, if the authors can persuade me on this I can consider increase my score)
- the tables are missing the confidence interval, e.g. Human eval only has 164 problems 7.3% is just 12 problems.
Questions for authors
- is budget policy really useful? in table 4 with and without it doesn't seem very different.
- λ_o as a parameter do you need to tune it? how sensitive is it?
no responses?