- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a bilevel framework for LLM self-play, which jointly optimizes the questioner and the solver. The optimization has two objectives: First, the questioner and the solver need to be paired in an equilibrium state, where updating either model undermines its own performance if the other model is fixed. Second, the equilibrium solver needs to satisfy a high-level incentive, which motivates a high success rate for the target task. In practice, this is achieved via a linear combination of the incentive loss function f and the equilibrium gap function. Results on math reasoning and formal theorem proving theorem-proving benchmarks suggest that BiGAP surpasses many baselines from prior work and also fixed-reward variants.
Critical strengths
- The idea of constraining self-improvement to equilibrium conditions is well-motivated, and the paper gives a feasible way to achieve it. Prior work does not formally require equilibrium between the Questioner and Solver, despite the fact that their roles are defined to be adversarial to each other. This paper introduces the formal constraints, and the proposed bilevel algorithm achieves this constraint optimization in a cost-efficient manner.
- The experiments are extensive, covering nine benchmarks, multiple ablations, and varying training iterations.
Critical weaknesses
- Improvements over the baselines are modest. Most improvements over the fixed-reward baseline are around 1 percent, which does not justify the additional complexities of BiGAP, including the UL solver loss and learning with the gap function. In addition, the fixed-reward baseline has multiple moving parts from BiGAP, so it is hard to identify the contribution of each component in the training objective.
- Theoretical convergence guarantee is disconnected from the practical algorithm. The convergence guarantee is established over exact best responses, exact gradient oracles, fixed prompt distributions, a fixed UL dataset, and a fixed penalty parameter . The strong assumptions do not hold in practice, and so the guarantee cannot be established on the actual algorithm used in practice.
Questions for authors
- Can you establish a new convergence guarantee for the algorithm used in practice, dropping the strong assumptions?
- How much does the UL solver objective contribute to the performance gain of BiGAP, and how much does the gap learning objective contribute?