- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies a finite-budget theory that shows a combination of reinforcement learning (RL) and on-policy distillation (OPD) can beat either methods alone. On the one hand, RL learns from rewards but is sensitive to reward model accuracy and can be noisy. On the other hand, OPD trains a weaker student to imitate the stronger teacher but may inherit teacher errors. The paper proposes a local nonlinear two-stage RL and OPD, and show the condition that the algorithm has fewer error than two pure methods. Finally, the paper implements the method in a real experiment, and shows the comparison against pure RL and OPD.
Critical strengths
I find the high level idea of this paper interesting, though I have not checked the whole proofs.
Critical weaknesses
- The current form of paper is hard to read. Theorem 4.1 takes a total of 12 lines, but one can hardly understand the central message it conveys. What is Equation (7)? Why do you assume it is positive? What is the interpretation? The authors simply define the math without any explanations, and it is unclear whether the condition is justified.
I also spot a lot of weird patterns in the writeup. Many sentences follow the pattern "X is Y, not Z" (e.g., Line 61, 71, 141, 147, 168, 192). For example, in Line 339, the authors write "These are theoretical certificates, not new experiments". Many sentences seem to answer questions the text has never asked, which read very unnatural.
- Experiments do not support the theory. Figure 1 (right panel) does not show the mixture method can beat pure RL or pure OPD, which raises the question why the experiments do not support the theory. The “not CI” text inside Figure 1 also reads odd.
Questions for authors
The paper repeatedly uses the word "certificate". What does it mean? Is it a common terminology in this field? If not, you should explain the meanings.