- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a re-weighting of training sample in standard OPSD. For each example , the log probability ratio defines the score , which is used in an optimization problem to give the desired re-weighted distribution. Experiments examined the re-weighting mechanism and verified improvement over standard OPSD.
Critical strengths
The topic of re-weighting the examples is interesting and clear. The re-weighting is designed as a solution to an optimization problem that maximizes the expected score. Experiments and ablation studies are conducted to verify the effectiveness of the proposed re-weighted optimization problem.
Critical weaknesses
The paper lacks theoretical justifications of the proposed method. I don't understand why the score is defined in that way. This score represents the change in the confidence of the student's generated token after the teacher reads the solution. This does not capture the correctness or the value of the student's generated token, and the meaning of your score makes me really confused. The issue is: a very bad token that increases the probability from 1e-10 to 1e-9 still has a large score; a token that decreases the probability from 1e-1 to 1e-10 has very low score, but it doesn't mean the example is not important. The solution to the relative-entropy trust-region optimization is not guaranteed to improve the training. I don't understand why the new distribution is better than the original distribution, and why it gives a better direction of the OPSD problem. The paper does not solve the relative-entropy trust-region optimization directly, and instead uses an online algorithm that keeps updating the distribution during training. However, this online update algorithm is not guaranteed in the paper to approach the desired distribution. At least some guarantee should be given for your algorithm.
Questions for authors
I already wrote everything in the Critical Weaknesses section. Please refer to that.