acceptodds
Under review as a conference paper at ICLR 2027

Multi-LLM Collaborative Alignment via Stackelberg Games

Abstract

A pool of language models can collaborate and improve collectively by learning from one another's responses. These interactions depend on the instructions used during training. Existing methods typically sample instructions uniformly, even though their usefulness may change as the models improve: an instruction on which models' responses once differed in quality may later be answered equally well, while a previously difficult instruction may begin to provide a useful learning signal. We propose Stackelberg Alignment, a game-theory-inspired leader–follower framework that turns instruction selection into an adaptive curriculum. An EXP3 bandit acts as the leader, allocating a fixed sampling budget across instructions and updating its sampling distribution using a reward that combines instruction difficulty and response discriminability. The language models act as followers: they respond to the selected instructions, evaluate one another's responses, and learn from the resulting preference signals through DPO or GRPO. The framework uses Elo-style reputation-weighted peer judgment and reputation-based opponent matching to support reliable and competitive model interactions. Experiments across three heterogeneous model pools and 12 benchmarks spanning scientific discovery, reasoning, code, instruction following, and knowledge show that Stackelberg Alignment achieves the highest macro-average across three diverse model pools, outperforming the strongest training-time baseline by up to 7.4% and the best static inference baseline by 12–25%. Analysis confirms that the adaptive leader concentrates duels on the most informative instructions, and ablations show that both reputation-weighted judgment and reputation-based matching improve the effectiveness of multi-LLM evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model collaboration, collaborative alignment, stackelberg games

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yZWTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Stackelberg Alignment, a multi-LLM collaborative training framework that uses EXP3 to adaptively select training prompts based on model competition. The framework combines reputation-based opponent matching and peer evaluation with DPO or GRPO training. Experiments across three heterogeneous model pools and 12 benchmarks demonstrate improvements over existing baselines.

Critical strengths

The experiments cover diverse model pools and benchmarks, showing consistent improvements in overall performance.

Critical weaknesses
  1. Under the GRPO setting, the method largely degenerates into adaptive prompt selection combined with multi-judge GRPO, a relatively standard setting. Comparisons with existing adaptive curriculum and prompt selection methods are missing.

  2. The DPO setting relies heavily on off-policy preference data generated by other models, inheriting the limitations of off-policy learning.

Questions for authors
  1. Why is it necessary to jointly train multiple LLMs, given that each model can perform inference independently?

  2. Does collaborative multi-LLM training outperform independent training under the same computational budget?

4 comments

We would like to thank the reviewer for their thoughtful comments and feedback.

Under GRPO, the method largely reduces to adaptive prompt selection combined with multi-judge GRPO, a fairly standard setting, and comparisons with existing adaptive curriculum and prompt selection methods are missing.

We compare against three existing adaptive curriculum and prompt selection methods, replacing only the EXP3 leader in Stackelberg (GRPO) and keeping everything else identical: the same followers, reputation-weighted peer-judge rewards, opponent-matching schedule, reputation system, and generation and update budget (T=8, D=64, G=4, LoRA, learning rate 1e-6).

  • (i) DAPO dynamic sampling (Yu et al., 2026): instructions whose G responses all receive the same peer reward (zero reward variance, hence zero GRPO advantage) are filtered out and resampled.
  • (ii) SEC (Chen et al., 2025): a non-stationary bandit whose reward is the learning progress of the follower, measured by the mean absolute GRPO advantage; since our instruction pools have no category labels, we run the bandit over individual instructions, the same arms as our leader.
  • (iii) CurES (Zeng et al., 2026): sampling probabilities allocated by each instruction's estimated success rate, which we compute from the normalized peer scores of the followers' responses.

Results on model Pool 1 (Avg is the mean over the four tasks):

Instruction selectionBixBenchLabBenchSMDDAssayBenchAvg
Uniform (Sparta-GRPO)0.3120.3110.1950.0260.211
DAPO dynamic sampling0.3400.2580.1990.0140.203
SEC0.3300.2970.2240.0220.218
CurES0.3010.2910.2330.0200.211
EXP3 leader (Stackelberg GRPO)0.3490.3220.2370.0230.233

The EXP3 leader outperforms all three prompt selection methods (0.233 vs. 0.203, 0.218, and 0.211), confirming that the gains do not only come from adaptive prompt selection in general, but also from selecting instructions by how the followers interact on them. SEC gains slightly over uniform sampling (0.218 vs. 0.211), CurES matches it (0.211), and DAPO dynamic sampling falls below it (0.203). This suggests that single-learner signals such as reward variance, advantage magnitude, or success rate are not by themselves informative under peer-judge rewards, whereas the leader's signals capture whether a duel separates the followers and whether the peer judges can tell their responses apart. This comparison is now in Appendix B.10 of the revised PDF.

The comparison also reflects how our setting differs from single-model prompt selection. Existing methods score an instruction by the progress of one learner on it (pass rate, advantage, or gradient signal), typically under verifiable rewards. Our leader instead scores an instruction by the interaction among the followers: whether the dueling pair is separated by it (difficulty) and whether the peer judges can tell their responses apart (preference quality, §2.1). The rewards come from reputation-weighted peer judges rather than from a verifier or an external reward model, and the leader also selects the opponent for each duel by the reputation gap (§2.2), which prompt selection methods do not have. We have expanded the related work to discuss DAPO, SEC, and CurES along these lines.

The DPO setting relies heavily on off-policy preference data generated by other models, inheriting the limitations of off-policy learning.

We do not use a fixed preference dataset generated by other models. In Stackelberg (DPO) the data is on-policy with respect to the follower pool, since the preference pairs are regenerated at every iteration by the current followers themselves: the leader draws D=64 duels, the two current followers in each duel respond to the instruction, the other current followers judge the responses, and the resulting pairs are used for that iteration's DPO update only (Algorithm 1). As the followers improve, so does their training data, which avoids the distribution drift of offline DPO on a static dataset.

Each follower does train on responses of the other followers. This is the collaborative mechanism rather than a side effect: a follower learns from the better responses of its peers, which is how complementary strengths transfer between models in the pool, as in Sparta Alignment.

For the fully on-policy setting, Stackelberg (GRPO) trains each follower only on its own sampled responses, scored by the peer judges. It further improves over Stackelberg (DPO) on model Pool 1 (macro-average 0.563 vs. 0.548; Table 1), and the leader improves results under both objectives (Sparta Alignment 0.524 → Stackelberg (DPO) 0.548; Sparta-GRPO 0.542 → Stackelberg (GRPO) 0.563, in our response to Reviewers gcGc and RUb3).

Why is it necessary to jointly train multiple LLMs if each model performs inference independently, and does collaborative multi-LLM training outperform independent training under the same computational budget?

Joint training does not change how the models are used at inference: the output of our method is a single trained model (the best checkpoint on the validation split), which performs inference independently at the cost of one model, unlike inference-time collaboration such as MoA or Multiagent Debate. The pool is needed during training, because it provides two learning signals that a model trained alone does not have:

  • Judges other than itself. Each follower is rewarded by the other followers as reputation-weighted peer judges, without an external reward model or verifier. A model trained alone has to judge its own responses, which is prone to self-preference bias; this is the m=1 special case of our framework, which recovers single-model self-alignment such as Self-Rewarding and SPIN (§2).
  • Peers' better responses. Each follower learns from duels in which a peer's response is preferred, which transfers complementary strengths between models in the pool. The leader's signal also exists only with multiple models: whether an instruction separates the dueling pair, and whether the peer judges can tell their responses apart (§2.1).

More broadly, a growing body of work studies the collaboration and collaborative training of multiple LLMs and shows the benefits of this direction over any single model, e.g., [1-8]. Our work builds on this direction and studies how to select the training instructions for collaborative training.

We additionally compare collaborative and independent training under the same computational budget. In independent training, each model in the pool is trained alone with the same EXP3 leader, judging its own responses, under both objectives: with DPO, the model generates two responses per instruction and is trained on the self-judged preference pair; with GRPO, it is trained on its own G samples scored by itself. Each uses the same number of iterations, generation and judge forward passes, and training steps as Stackelberg (DPO) and Stackelberg (GRPO), respectively, and we report the best model on the validation split, as for our method. Results on model Pool 1 (Avg is the mean over the four tasks):

MethodBixBenchLabBenchSMDDAssayBenchAvg
Independent training (DPO)0.1940.2730.1020.0050.144
Independent training (GRPO)0.2910.3150.2090.0180.208
Collaborative training (Stackelberg DPO)0.2330.3110.2040.0240.193
Collaborative training (Stackelberg GRPO)0.3490.3220.2370.0230.233

Collaborative training outperforms independent training under the same budget with both objectives (0.193 vs. 0.144 with DPO; 0.233 vs. 0.208 with GRPO), confirming that the gains do not only come from additional training of each model, but also from the peer judges and peer responses that only the pool provides. This suggests that collaboration helps most where a single model's own signal is weakest: with DPO, a self-judged pair of the model's own two responses offers little contrast and inherits its self-preference bias, and collaborative training improves by 0.049, compared with 0.025 under GRPO, where the model already obtains contrast from its G samples. This experiment is now in Appendix B.11 of the revised PDF.

[1] Du and Kaelbling. Position: Compositional Generative Modeling: A Single Model is Not All You Need.

[2] Subramaniam et al. Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains.

[3] Feng et al. Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems.

[4] Jiang et al. Sparta Alignment: Collectively Aligning Multiple Language Models through Combat.

[5] Zhao et al. The Majority is Not Always Right: RL Training for Solution Aggregation.

[6] Park et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning.

[7] Motwani et al. MALT: Improving Reasoning with Multi-Agent LLM Training.

[8] Estornell et al. ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration.

We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.

Thank you for the additional experiments. However, my core concern remains unresolved. Since the final output is a single LLM used independently at inference time, what is the fundamental advantage of jointly training multiple LLMs over simply training one policy with rewards provided by multiple external LLM judges through voting or consensus? Such a simpler approach could also benefit from diverse model judgments without requiring joint policy updates. The authors' comparison against independent self-judging models does not address this distinction, as it conflates the benefits of external peer evaluation with those of joint training. The paper still lacks a convincing theoretical or intuitive justification for why joint training itself is necessary or preferable.

Thank you for the follow-up. We ran the comparison the reviewer describes: one policy trained with rewards from multiple fixed external LLM judges, without joint policy updates.

We train each model in the pool on its own with GRPO, and the other m−1 models of the pool serve as fixed judges that are never updated. Everything else is identical to Stackelberg (ours): the same EXP3 leader, number of iterations, instructions per iteration, group size (G=4), and generation and judge forward passes. The only difference is that in Stackelberg (ours) all m models are updated jointly, so the judges and the dueling peers improve during training, while here they stay fixed. Columns 1 to m give the result of training each model of the pool this way (Pool 1: m=9; Pool 2: m=8; model order as in Appendix B.12), compared with Stackelberg (ours):

Dataset, pool123456789Stackelberg (ours)
BixBench, Pool 10.2910.3110.2520.2720.3010.3010.2330.2520.2820.349
LabBench, Pool 10.2980.2750.3030.3110.2990.3010.2910.2980.2840.322
BixBench, Pool 20.1750.2140.2230.2330.1940.1840.1940.194–0.291
LabBench, Pool 20.3200.3320.3220.2990.3220.3390.3320.329–0.355

Stackelberg (ours) outperforms every model trained with fixed external judges on both datasets and both pools, including the best one in each row, confirming that the gains do not only come from external peer evaluation, but also from training the policies jointly. This suggests that joint training helps because the judges and the dueling peers improve together with the policy: fixed judges stay at their initial ability, so as the policy improves, their consensus can no longer tell its responses apart, while jointly trained peers keep providing informative rewards and stronger responses to learn from. This experiment is now in Appendix B.12 of the revised PDF.

Using a single model at inference is actually a strength of our approach in efficiency: it costs one model's inference, whereas multi-LLM inference methods such as Majority Vote, MoA, Multiagent Debate, and Heterogeneous Swarms run multiple models for every query. Stackelberg (ours) also outperforms all of them in macro-average on all three pools (Table 1), thanks to the strength of collaborative training. It reaches 0.563, 0.544, and 0.609 on Pools 1–3, compared with 0.464, 0.482, and 0.487 for the strongest multi-LLM inference method in each pool (Majority Vote, Heterogeneous Swarms, and MoA).

We made these edits and uploaded a revised PDF to OpenReview.

Reviewer_RUb3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds an EXP3 instruction selector to a multi-LLM alignment framework built on pairwise duels, reputation-weighted peer judgments, and DPO or GRPO training. It also schedules opponent selection by reputation gap. Experiments cover three heterogeneous model pools and twelve tasks, with the GRPO variant achieving the highest reported macro-average in each pool.

Critical strengths
  1. Adaptive instruction selection is well motivated: useful training examples change as the models improve. Appendix F provides a useful DPO control by keeping Sparta’s instruction pool, opponent schedule, and reputation system fixed while using uniform sampling.
  2. The evaluation spans different model pools and tasks. GRPO achieves macro-averages of .563/.544/.609 versus Sparta’s .524/.533/.579. DPO also improves in the first two pools, although it falls slightly below Sparta in the third. These results suggest practical value beyond a single configuration.
Critical weaknesses
  1. The connection to a Stackelberg equilibrium remains underdeveloped. Section 2.4 acknowledges that alternating updates are an approximation, but does not specify the follower best-response problem or establish how the leader anticipates follower adaptation. The practical curriculum is understandable; its relationship to the claimed equilibrium is not established. This weakens the game formulation as a theoretical contribution.
  2. My main empirical concern is the missing uniform-sampling GRPO control. Appendix F matches gradient updates and other settings for Sparta, but Sparta uses DPO. The strongest results therefore change both instruction selection and the training objective. The DPO comparison supports some benefit from adaptive sampling, but cannot establish how much of the GRPO improvement comes from the leader.
Questions for authors
  1. How does uniform sampling perform with the same GRPO objective, peer rewards, opponent schedule, and generation/update budget? This comparison would most directly affect my recommendation.
  2. How does K=64K=64 in Appendix C relate to the 80% training pool in Appendix E? Please also clarify how duel rewards are aggregated in Eq. (5), given Algorithm 1’s update after all duels, and whether the opponent schedule is linear or cosine.
  3. Can you provide paired intervals for the key method–baseline differences? Appendix B.1 reports individual-method intervals, but comparing their lower bounds with baseline point estimates does not account for baseline uncertainty.
3 comments

We would like to thank the reviewer for their thoughtful comments and feedback.

The connection to a Stackelberg equilibrium is underdeveloped: Section 2.4 calls the alternating updates an approximation but does not specify the follower best-response problem or how the leader anticipates follower adaptation, which weakens the game formulation as a theoretical contribution.

We do not claim a theoretical equilibrium result: we describe the method as "a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum" (Abstract), and we use the Stackelberg game to frame the training loop. However, we agree that this was not consistently clear throughout the paper; as the reviewer notes, §2.4 described the alternating updates as an approximation of the equilibrium, which overstated the connection. We have made the following edits:

  • §2.4: renamed "Connection to the Stackelberg equilibrium" to "Relation to the Stackelberg equilibrium" and rewrote it to state that our procedure does not compute the equilibrium, since EXP3 reacts to the observed informativeness of past duels rather than anticipating how the followers adapt, and a round of DPO or GRPO updates is not a follower best response. We state that we provide no convergence guarantee and use the formulation as motivation and a design principle for the alternating leader-follower updates.
  • §2: the sentence "the leader's optimal strategy anticipates the followers' most informative responses" now refers to the idealized game, followed by "We use this game as a design principle for the training loop rather than as an equilibrium that we compute."
  • §1, contributions: "a Stackelberg game formulation" → "a Stackelberg-game-inspired leader-follower formulation."
  • §6: "By casting the training loop as a Stackelberg game solved via EXP3" → "By structuring the training loop as a Stackelberg-inspired leader-follower game with an EXP3 leader."

To make the relationship precise, we specify the idealized game here. Given the leader's instruction distribution πL∈Δ(X)\pi_L \in \Delta(\mathcal{X}) and the other followers' policies π−i\pi_{-i}, follower ii's best response is

πi∗(πL,π−i)∈arg⁡max⁡πiEx∼πLEy∼πi(⋅∣x)[sˉi(x,y;π−i)]−βKL(πi∥πiref),\pi_i^*(\pi_L, \pi_{-i}) \in \arg\max_{\pi_i} \mathbb{E}_{x \sim \pi_L} \mathbb{E}_{y \sim \pi_i(\cdot \mid x)} \big[\bar{s}_i(x, y; \pi_{-i})\big] - \beta \mathrm{KL}\big(\pi_i \Vert \pi_i^{\mathrm{ref}}\big),

where sˉi\bar{s}_i is the reputation-weighted peer score (Eq. 8), which depends on π−i\pi_{-i} through both the opponent and the peer judges. The followers' joint response π∗(πL)\pi^*(\pi_L) is a profile in which every follower best-responds to the others. The leader anticipates this response and solves

max⁡πL∈Δ(X)Ex∼πL[r(x;π∗(πL))],\max_{\pi_L \in \Delta(\mathcal{X})} \mathbb{E}_{x \sim \pi_L}\big[r\big(x; \pi^*(\pi_L)\big)\big],

where rr is the leader's instruction reward (difficulty and preference quality, §2.1) evaluated on the followers' responses. Our algorithm departs from this problem at both levels: (i) each follower takes one round of DPO or GRPO updates from its current policy on duels drawn from πLt\pi_L^t, rather than solving for πi∗\pi_i^*; (ii) EXP3 maximizes r(x;πt)r(x; \pi^t) under the current followers πt\pi^t, rather than r(x;π∗(πL))r(x; \pi^*(\pi_L)) under their best response, so the leader tracks the followers' adaptation instead of anticipating it; and (iii) there is no convergence guarantee for the coupled system. The revised §2.4 states these departures in words; we give the formal statement of the idealized game in this response.

We have existing results on how the leader responds to the followers' adaptation. Figures 4 and 5 show that the leader's distributions change as the followers change: the instruction weights start near uniform and concentrate into a difficulty curriculum by iteration 5 (Figure 5), and the opponent weights shift away from the emerging dominant model, differently for each task (Figure 4). Table 5 (Appendix B.6) compares EXP3 with an LLM leader (Gemini) that selects instructions conditioned on the followers' state (model embeddings and current reputation scores). The LLM leader underperforms EXP3 on GPQA (0.263 vs. 0.364), MBPP (0.581 vs. 0.643), and AlpacaEval (1.65 vs. 8.08), and ties on TruthfulQA (0.682), suggesting that explicitly reasoning about the followers' state does not by itself give a better curriculum than tracking the outcomes of their duels.

We additionally test whether tracking the followers matters, by fixing the leader's instruction distribution in Stackelberg (GRPO): frozen after iteration 2, and the final learned distribution of the adaptive run, fixed from iteration 1. Results on model Pool 1 (Avg is the mean over the four tasks):

LeaderBixBenchLabBenchSMDDAssayBenchAvg
Uniform (Sparta-GRPO)0.3120.3110.1950.0260.211
Frozen after iteration 20.3010.2160.1790.0120.177
Final distribution, fixed throughout0.3590.3110.2030.0190.223
Adaptive (Stackelberg GRPO)0.3490.3220.2370.0230.233

The adaptive leader outperforms both fixed distributions (0.233 vs. 0.177 and 0.223), including the final learned one, confirming that the gains do not only come from finding a good instruction distribution, but also from updating it as the followers adapt. This suggests that the leader's value lies in tracking the followers: a distribution frozen early (0.177) falls even below uniform sampling (0.211), as it keeps concentrating on instructions that were informative for the initial followers but no longer are once they improve, while the final distribution (0.223) recovers most of the gain but still trails the adaptive leader, as it presents the instructions suited to the trained followers before the followers are ready for them. This experiment is now in Appendix B.9 of the revised PDF.

The uniform-sampling GRPO control is missing: Sparta uses DPO, so the strongest results change both instruction selection and the training objective. How does uniform sampling perform with the same GRPO objective, peer rewards, opponent schedule, and generation/update budget?

We train a new baseline, Sparta-GRPO: Sparta with uniform instruction sampling, trained with GRPO on online peer-judge rewards. We keep the experiment setting identical to Stackelberg (GRPO): the same GRPO objective, reputation-weighted peer-judge rewards, opponent-matching schedule, reputation system, and generation and update budget (T=8 iterations, D=64 duels per iteration, G=4 samples per instruction, LoRA, learning rate 1e-6). Results on model Pool 1:

MethodBixBenchLabBenchSMDDAssayBenchGPQA-DiaMATHHumanEvalMBPPAlpacaEvalIFEvalTruthfulQACulturalBenchAvg
Sparta Alignment (DPO, uniform)0.3010.3080.1680.0170.3130.8200.7190.6347.5280.7070.6860.6560.524
Stackelberg (DPO)0.2330.3110.2040.0240.3640.8570.7630.6438.0810.7490.6820.7470.548
Sparta-GRPO (GRPO, uniform)0.3120.3110.1950.0260.3400.8730.7890.6576.9210.7090.6760.7000.542
Stackelberg (GRPO)0.3490.3220.2370.0230.3530.8770.8160.6537.8190.7300.6840.7240.563

Avg is the macro-average over the 12 datasets with AlpacaEval min-max normalized to [0,1] within the pool, as in Table 1. Sparta-GRPO reaches a macro-average of 0.542, compared with 0.563 for Stackelberg (GRPO), confirming that the gains do not only come from switching DPO to GRPO, but also from the leader's adaptive instruction selection, which improves results under GRPO as it does under DPO (0.524 → 0.548). This suggests that the leader's gain under GRPO (0.542 → 0.563) is comparable to its gain under DPO, and accounts for about half of the gain of Stackelberg (GRPO) over Sparta (0.021 of 0.039). The fixed-leader comparison in our response above also isolates the leader under GRPO. This experiment is now in Appendix B.7 of the revised PDF.

How does K=64 in Appendix C relate to the 80% training pool in Appendix E? How are duel rewards aggregated in Eq. (5), given Algorithm 1's update after all duels? Is the opponent schedule linear or cosine?

Instruction pool size K. We first split each development set into a training split (80%) and a validation split (20%; Appendix E). The leader's instruction pool is a fixed subset of K=64 instructions from the training split. The validation split is never seen by the leader or used for training, and is only used for checkpoint selection. We have clarified this in Appendices C and E.

Reward aggregation in Eq. (5). The leader is updated once per iteration, after all D duels, as in Algorithm 1. Within iteration tt the sampling distribution ptp^t is fixed, and the rewards of all duels on instruction xkx_k are importance-weighted and summed:

log⁡wkt+1=log⁡wkt+1K∑d∈Dktrk,dpkt,\log w_k^{t+1} = \log w_k^t + \frac{1}{K} \sum_{d \in \mathcal{D}_k^t} \frac{r_{k,d}}{p_k^t},

where Dkt\mathcal{D}_k^t is the set of duels in iteration tt that drew xkx_k and rk,dr_{k,d} is the reward of duel dd. An instruction drawn several times accumulates the rewards of all its duels, and an instruction that is not drawn receives no update. We have revised Eq. (5) and the text below it accordingly, and corrected Appendix H, which described the update as per duel.

Opponent schedule. The schedule is linear: the target reputation gap is g∗_t=1−τg^*\_t = 1 - \tau with τ=t/(T−1)\tau = t/(T-1) (Eq. 3, §2.2), which decreases linearly from 1 (mismatched opponents) to 0 (near-peer opponents). The target score gap in the leader's preference-quality reward follows the same linear schedule, from 0.6 to 0.15.

Can you provide paired intervals for the key method-baseline differences? Appendix B.1 reports per-method intervals, and comparing their lower bounds with baseline point estimates ignores baseline uncertainty.

We compute paired bootstrap intervals for the differences between Stackelberg (GRPO) and the three strongest baselines by macro-average across the three pools: Sparta Alignment, MoA, and Trained Router. Since all methods are evaluated on the same test items, we resample test items with replacement (10,000 resamples), recompute the score difference Δ = Stackelberg (GRPO) − baseline on each resample from per-item scores (correctness for accuracy tasks, reward-model scores for AlpacaEval), and mark the differences whose 95% interval excludes 0. Unlike the per-method intervals in Appendix B.1, this accounts for the uncertainty of both methods and for the correlation between them on the same items, and does not assume a variance for AlpacaEval. Score differences Δ (AlpacaEval in reward-model score units; Avg as in Table 1); †: the paired 95% interval excludes 0:

ComparisonBixBenchLabBenchSMDDAssayBenchGPQA-DiaMATHHumanEvalMBPPAlpacaEvalIFEvalTruthfulQACulturalBenchAvg
Pool 1 vs. Sparta+0.048+0.014+0.069†+0.006†+0.040+0.057†+0.097†+0.019+0.291†+0.023−0.002+0.068†+0.039†
Pool 1 vs. MoA+0.126†+0.035†+0.142†+0.007†0.000+0.044†0.000+0.064†+11.667†+0.023+0.102†−0.005+0.112†
Pool 1 vs. Trained Router+0.116†+0.059†+0.226†+0.008†+0.040+0.142†+0.193†+0.142†+6.101†+0.099†+0.079†+0.091†+0.135†
Pool 2 vs. Sparta+0.019+0.025+0.018†+0.007†+0.120†+0.031†+0.027+0.006−1.066†+0.008−0.0060.000+0.011
Pool 2 vs. MoA+0.039+0.045†+0.070†+0.010†+0.009+0.058†+0.079†+0.018+5.731†−0.001+0.096†−0.006+0.086†
Pool 2 vs. Trained Router+0.077†+0.094†+0.064†+0.011†+0.080†+0.058†+0.114†+0.018+2.141†+0.031+0.012+0.026+0.068†
Pool 3 vs. Sparta−0.009+0.112†+0.059†−0.006†+0.040+0.0150.000+0.023+0.029+0.016+0.010+0.092†+0.030†
Pool 3 vs. MoA+0.049+0.073†+0.050†+0.004+0.049−0.037†+0.368†+0.058†+8.371†+0.022+0.010−0.012+0.122†
Pool 3 vs. Trained Router+0.097†+0.157†+0.280†+0.012†+0.132†+0.044†+0.087†+0.078†+1.250†+0.202†+0.099†+0.174†+0.124†

Results confirm the significance of improvements in an item-paired setting. This analysis is now in Appendix B.1 of the revised PDF.

We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.

Reviewer_gcGcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Stackelberg Alignment extends Sparta Alignment by adapting how training instructions are selected during collaborative LLM fine-tuning. An EXP3 bandit prioritizes prompts using a reward that combines instruction difficulty with the response-quality gap between models. The method also schedules opponents based on their reputations and uses peer feedback for DPO or GRPO training. Experiments cover three heterogeneous model pools and 12 benchmarks. The reported results favor the proposed method over existing collaborative training and inference-time approaches, with the strongest results coming from the GRPO variant.

Critical strengths
  1. Instruction selection is a problem worth studying. As the participating models improve, a fixed sampling strategy may become less useful. Adapting the training distribution is a reasonable way to address this.

  2. The paper is organized well enough to follow the overall procedure. The figures and algorithm descriptions help explain how instruction sampling and model training interact.

  3. The evaluation covers several model combinations and a broad range of tasks. Testing three heterogeneous pools across 12 benchmarks gives a useful picture of performance across different settings.

Critical weaknesses
  1. My main concern is that the experiments do not clearly separate the benefit of instruction selection from the benefit of GRPO. The best-performing variant uses GRPO, while the Sparta Alignment baseline uses DPO. How would Sparta perform with GRPO under the same conditions? Its peer judgments could also provide rewards for GRPO, so I would like to see a Sparta+GRPO baseline with uniform instruction sampling and the same training budget. Without this comparison, I find it difficult to attribute the gains to the proposed instruction-selection strategy.

  2. The ablations leave the contribution of individual components unclear. Several changes are introduced together, and the experiments do not fully separate their effects. For example, the sampling reward combines difficulty and preference quality, but neither term is tested on its own. The opponent-matching comparisons are also difficult to interpret because the training budgets differ. Testing the reward terms separately and comparing opponent-selection strategies under the same budget would make these results more informative.

  3. I am not convinced that the training procedure approximates a Stackelberg equilibrium. Alternating between EXP3 updates and model optimization does not, by itself, establish convergence to such an equilibrium. DPO or GRPO updates need not produce follower best responses, and EXP3’s regret guarantees do not establish convergence for the combined system. I could not find theoretical guarantees or empirical evidence supporting the equilibrium interpretation. Unless the authors can provide that support, I suggest presenting the game-theoretic formulation as motivation and explaining what it contributes beyond adaptive curriculum learning.

Questions for authors

N/A

3 comments

We would like to thank the reviewer for their thoughtful comments and feedback.

The experiments do not separate the benefit of instruction selection from the benefit of GRPO, since Sparta Alignment uses DPO; how would a Sparta+GRPO baseline with uniform instruction sampling and the same training budget perform?

We train a new baseline, Sparta-GRPO: Sparta with uniform instruction sampling, trained with GRPO on online peer-judge rewards. We keep the experiment setting identical to Stackelberg (GRPO): the same GRPO objective and group size (G=4), reputation-weighted peer-judge rewards, opponent-matching schedule, reputation system, and training budget (T=8 iterations, D=64 duels per iteration, LoRA, learning rate 1e-6). Results on model Pool 1:

MethodBixBenchLabBenchSMDDAssayBenchGPQA-DiaMATHHumanEvalMBPPAlpacaEvalIFEvalTruthfulQACulturalBenchAvg
Sparta Alignment (DPO, uniform)0.3010.3080.1680.0170.3130.8200.7190.6347.5280.7070.6860.6560.524
Stackelberg (DPO)0.2330.3110.2040.0240.3640.8570.7630.6438.0810.7490.6820.7470.548
Sparta-GRPO (GRPO, uniform)0.3120.3110.1950.0260.3400.8730.7890.6576.9210.7090.6760.7000.542
Stackelberg (GRPO)0.3490.3220.2370.0230.3530.8770.8160.6537.8190.7300.6840.7240.563

Avg is the macro-average over the 12 datasets with AlpacaEval min-max normalized to [0,1] within the pool, as in Table 1. Sparta-GRPO reaches a macro-average of 0.542, compared with 0.524 for Sparta and 0.563 for Stackelberg (GRPO), confirming that the gains do not only come from switching DPO to GRPO, but also from the leader's adaptive instruction selection. This suggests that the leader accounts for about half of the gain of Stackelberg (GRPO) over Sparta (0.021 of 0.039), and that adaptive instruction selection with DPO (Stackelberg (DPO), 0.548) already outperforms uniform sampling with GRPO. This experiment is now in Appendix B.7 of the revised PDF.

More ablation on individual components and training budgets.

Reward terms. We train Stackelberg (GRPO) with each term of the leader's sampling reward on its own: difficulty only (λ1=1, λ2=0) and preference quality only (λ1=0, λ2=1), keeping everything else identical. Together with uniform sampling (Sparta-GRPO, i.e., neither term) and the default combination (λ1=0.3, λ2=0.7), this gives a full ablation of the two terms. Results on model Pool 1 (Avg is the mean over the four tasks):

Leader rewardBixBenchLabBenchSMDDAssayBenchAvg
None (uniform; Sparta-GRPO)0.3120.3110.1950.0260.211
Difficulty only0.2520.2770.2010.0090.185
Preference quality only0.3400.3110.1990.0180.217
Both (Stackelberg GRPO)0.3490.3220.2370.0230.233

Difficulty alone underperforms uniform sampling (0.185 vs. 0.211), preference quality alone gives a small gain (0.217), and the combination is best (0.233), confirming that the gains do not only come from either term alone, but also from combining them. This suggests that the two terms are complementary: difficulty alone concentrates sampling on hard instructions where both followers score low and the peer judges give little contrast between them, while the preference-quality term keeps the selected instructions discriminative, so that difficulty helps only when paired with it. This is also consistent with our default weighting, which puts more weight on preference quality (λ2=0.7). This experiment is now in Appendix B.8 of the revised PDF.

Opponent selection under the same budget. We compare all opponent-selection strategies on MATH (model Pool 2) under the same budget of 8 iterations, and add two more strategies: Sparta's reputation-proximity matching and uniformly random matching.

Opponent selectionMATH
Random0.840
Reputation proximity (Sparta)0.866
Lowest difference0.875
Highest difference0.861
Scheduled, decreasing0.865
Scheduled, increasing0.878

Random matching performs worst (0.840), below even the single-model baseline (0.857), while all reputation-based strategies outperform the single model (0.861-0.878), confirming that the gains do not only come from dueling against peers, but also from choosing each follower's opponent by reputation. Reputation-proximity matching (0.866) is on par with our default schedule (0.865), and scheduled-increasing (0.878) and lowest-difference (0.875) matching perform best. This suggests that informed opponent selection is what matters, while the best schedule among reputation-based strategies is task-dependent and not tuned by our default. The two new strategies are now in Appendix B.3 of the revised PDF.

Alternating EXP3 updates with DPO/GRPO updates does not establish convergence to a Stackelberg equilibrium, and there is neither theoretical nor empirical support for the equilibrium interpretation; present the game-theoretic formulation as motivation and explain what it adds beyond adaptive curriculum learning.

We agree that the procedure is not shown to reach a Stackelberg equilibrium, and we do not claim a theoretical equilibrium result: we describe the method as "a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum" (Abstract), and we use the Stackelberg game to frame the training loop. However, we agree that this was not consistently clear throughout the paper; in particular, the paragraph at the end of §2 stated that the alternating updates approximate the equilibrium. We have made the following edits:

  • §2.4: renamed "Connection to the Stackelberg equilibrium" to "Relation to the Stackelberg equilibrium" and rewrote it to state that our procedure does not compute the equilibrium, since EXP3 reacts to the observed informativeness of past duels rather than anticipating how the followers adapt, and a round of DPO or GRPO updates is not a follower best response. We state that we provide no convergence guarantee and use the formulation as motivation and a design principle for the alternating leader-follower updates.
  • §2: the sentence "the leader's optimal strategy anticipates the followers' most informative responses" now refers to the idealized game, followed by "We use this game as a design principle for the training loop rather than as an equilibrium that we compute."
  • §1, contributions: "a Stackelberg game formulation" → "a Stackelberg-game-inspired leader-follower formulation."
  • §6: "By casting the training loop as a Stackelberg game solved via EXP3" → "By structuring the training loop as a Stackelberg-inspired leader-follower game with an EXP3 leader."
  • §2: added the comparison with curriculum learning below.

Beyond adaptive curriculum learning, the leader-follower formulation contributes the following:

  • A unified view of collaborative alignment. Fixing the leader to uniform sampling recovers single-model self-alignment methods such as Self-Rewarding and SPIN (m=1) and Sparta Alignment (m>1); our method is the instance with an adaptive leader. This makes instruction selection an explicit, tunable component of the training loop that prior methods leave fixed.
  • A curriculum driven by interaction among followers. Standard curriculum learning scores training examples by a single learner's loss or difficulty. Here, the leader's payoff is defined by the followers' interaction: how difficult an instruction is for the dueling pair, and how well the peer judges can discriminate between their responses. The curriculum therefore targets the learning signal that arises from multi-model collaboration, and shifts as the followers' relative capabilities change.
  • A commitment structure that fixes the algorithm design. The leader commits to an instruction distribution before the followers respond and is updated only from their outcomes, which determines the order of the training loop and the leader's feedback. It also makes it straightforward to replace EXP3 with leaders that model follower adaptation, such as lookahead, which we leave to future work.

We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.

Thank you for the additional experiments and clarifications. Adding the grpo comparison is appreciated, though I am still concerned about the strength of the empirical contribution. With GRPO held fixed, adaptive instruction selection improves the reported scores from 0.542->0.563, and varies across benchmarks. So I am unsure whether the gains justify the added complexity; there is some benefit from adaptive sampling, but it provides limited evidence for the value of the full framework. So I increase the soundness score but tend to remain with my overall assessment.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your contributions to the review process. As the author–reviewer discussion period ends on November 18, I would like to kindly remind everyone to review the authors' rebuttals and engage in the discussion.

In particular, please indicate whether your main concerns have been adequately addressed and clarify any remaining issues that may affect your final assessment.

Your feedback will be very helpful for the subsequent reviewer–AC discussion and meta-review process.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.