acceptodds
Under review as a conference paper at ICLR 2027

Bilevel Incentive Optimization for Language Model Self-Improvement without Human Labels

Abstract

Self-improvement enables large language models to learn from self-generated problems and automated feedback, but its effectiveness depends on the incentives governing problem generation and solution learning. We propose a bilevel post-training paradigm that jointly adapts these incentives and the policies they induce without human-provided solution labels. At the lower level, a questioner and a solver interact in a general-sum game under role-specific rewards. At the upper level, a shared incentive model minimizes the equilibrium solver's expected task failure. We propose BiGAP, a bilevel game-aware penalty method that constructs policy-specific objectives from Nikaido–Isoda gap functions and aggregates both players' gaps for incentive learning. Its training protocol coordinates question generation, separate practice and target-response sampling, policy optimization, and incentive adaptation, with optional target-question refresh. For fixed distributions and a fixed penalty parameter, we show under suitable regularity assumptions that, after exact-oracle iterations, the average squared operator residual is . The analysis further relates optimization accuracy and penalty strength to approximate Nash equilibrium guarantees. Across nine mathematical reasoning and formal theorem-proving benchmarks, BiGAP consistently outperforms fixed-incentive self-improvement. These gains are achieved without human-provided solution labels during self-improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-improvement, Bilevel optimization, Incentive design, Large language models.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XaD7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a bilevel framework for LLM self-play, which jointly optimizes the questioner and the solver. The optimization has two objectives: First, the questioner and the solver need to be paired in an equilibrium state, where updating either model undermines its own performance if the other model is fixed. Second, the equilibrium solver needs to satisfy a high-level incentive, which motivates a high success rate for the target task. In practice, this is achieved via a linear combination of the incentive loss function f and the equilibrium gap function. Results on math reasoning and formal theorem proving theorem-proving benchmarks suggest that BiGAP surpasses many baselines from prior work and also fixed-reward variants.

Critical strengths
  • The idea of constraining self-improvement to equilibrium conditions is well-motivated, and the paper gives a feasible way to achieve it. Prior work does not formally require equilibrium between the Questioner and Solver, despite the fact that their roles are defined to be adversarial to each other. This paper introduces the formal constraints, and the proposed bilevel algorithm achieves this constraint optimization in a cost-efficient manner.
  • The experiments are extensive, covering nine benchmarks, multiple ablations, and varying training iterations.
Critical weaknesses
  • Improvements over the baselines are modest. Most improvements over the fixed-reward baseline are around 1 percent, which does not justify the additional complexities of BiGAP, including the UL solver loss and learning with the gap function. In addition, the fixed-reward baseline has multiple moving parts from BiGAP, so it is hard to identify the contribution of each component in the training objective.
  • Theoretical convergence guarantee is disconnected from the practical algorithm. The convergence guarantee is established over exact best responses, exact gradient oracles, fixed prompt distributions, a fixed UL dataset, and a fixed penalty parameter λ\lambda. The strong assumptions do not hold in practice, and so the guarantee cannot be established on the actual algorithm used in practice.
Questions for authors
  • Can you establish a new convergence guarantee for the algorithm used in practice, dropping the strong assumptions?
  • How much does the UL solver objective contribute to the performance gain of BiGAP, and how much does the gap learning objective contribute?
Reviewer_uUmnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BiGAP, a framework for self improvement of LLMs without human labels. At the lower level, a questioner and a solver are parameterized by the same incentive model and optimize their role specific rewards. At the upper level, the goal is to learn incentives that minimize the solver's expected failure, while the policies of the solver and questioner form a Nash equilibrium. BiGAP uses Nikaido-Isoda gaps to model the equilibrium constraints and learn the incentives. The authors also give convergence guarantees under regularity assumptions for an exact-oracle theoretical version of BiGAP. They also provide experiments on nine benchmarks showing that BiGAP improves over a baseline, although the improvements seem modest.

Critical strengths
  1. The formulation connecting the design of incentives with game theory and the self-improvement of LLMs seems interesting. The modeling of questioner and solver as players in the general-sum game makes sense as their objectives are coupled. Optimizing them independently doesnt guarantee improvement in performance. Additionally, the use of Nikaido-Isoda gaps gives a nice formalization to reason about the equilibrium.

  2. The evaluation covers many benchmarks with two different domains (math QA and formal theorem proving).

Critical weaknesses
  1. The motivation is to improve the solver's task performance by learning relevant incentives. The task loss however is constant with respect to the incentive parameters at each update. Thus, the incentive is trained only by minimizing the empirical reward gaps without somehow estimating the effect of the incentives on the performance. Also, the gaps compare the maximum and mean rewards of the candidates and do not evaluate best responses of the policies, while also not taking into consideration the KL regularization. This might lead to uniform reward scores and not necessarily give better incentives / policies. Finally the convergence assumes exact best-response, gradient oracles, fixed disributions, etc.

  2. The comparison in the experiments is with a baseline that does not have the incentive updates as well as the upper level solver updates. The improvements could be explained by the additional training on target questions and not by the incentive learning. Also the gains are modest and the results do not include uncertainty estimates.

Questions for authors
  1. How much of the improvement is specifically from the incentive learning? Could the authors provide an experiment keeping the upper level solver training while the incentive model is fixed?
Reviewer_RsGkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose a method for language model self-improvement via bilevel optimization. At the lowest level, a teacher generates questions and a student must solve them, coming to an equilibrium. At the highest level, a supervisor/incentive model updates the incentives (encoded in the lower level reward signal) given to the student and teacher to encourage the arrived-at equilibrium to improve downstream task performance. The authors propose a framework and theoretically analyze its convergence for both the UL optimality and LL convergence to a Nash equilibrium. Through experiments, they show gains in performance when using the bilevel optimization strategy across multiple benchmarks.

I think this paper, with the notation cleaned up, would be a great inclusion at the conference. I am more than willing to increase my score.

Critical strengths

The method at a high level is intuitive, and is substantiated by a detailed algorithm proposal and theoretical treatment. The empirical results prove the efficacy of the approach as well.

Critical weaknesses

The notation at times is confusing. For instance, in Eqn. 5 lowercase rr appears twice with different subscripts on both sides of the equation. I understand the need to save space but indexing all these functions by i∈{Q,S}i \in \{Q, S\} also generally makes the math less readable. For instance, why introduce this notation only to have KS=∅\mathcal{K}_S = \emptyset?

The authors have results proving the convergence of the algorithm, yet in practice find that the last iteration has worse performance and report results on the third or fourth iterate instead.

Section 5.4 is a little light on writing, and with the additional page count I would like to see more discussion in this section.

Line 258: "Let CiC_i contain its contexts" is a little confusing as well. Can you explain what this means?

Notation, Line 1364: line 4 -- should this be HλH_\lambda isntead of FλF_\lambda?

Questions for authors

Line 232: why is the player-specific objective still f as well? Shouldn't it be gQg_Q and gSg_S?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.