- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Stackelberg Alignment, a multi-LLM collaborative training framework that uses EXP3 to adaptively select training prompts based on model competition. The framework combines reputation-based opponent matching and peer evaluation with DPO or GRPO training. Experiments across three heterogeneous model pools and 12 benchmarks demonstrate improvements over existing baselines.
Critical strengths
The experiments cover diverse model pools and benchmarks, showing consistent improvements in overall performance.
Critical weaknesses
-
Under the GRPO setting, the method largely degenerates into adaptive prompt selection combined with multi-judge GRPO, a relatively standard setting. Comparisons with existing adaptive curriculum and prompt selection methods are missing.
-
The DPO setting relies heavily on off-policy preference data generated by other models, inheriting the limitations of off-policy learning.
Questions for authors
-
Why is it necessary to jointly train multiple LLMs, given that each model can perform inference independently?
-
Does collaborative multi-LLM training outperform independent training under the same computational budget?
4 comments
Why is it necessary to jointly train multiple LLMs if each model performs inference independently, and does collaborative multi-LLM training outperform independent training under the same computational budget?
Joint training does not change how the models are used at inference: the output of our method is a single trained model (the best checkpoint on the validation split), which performs inference independently at the cost of one model, unlike inference-time collaboration such as MoA or Multiagent Debate. The pool is needed during training, because it provides two learning signals that a model trained alone does not have:
- Judges other than itself. Each follower is rewarded by the other followers as reputation-weighted peer judges, without an external reward model or verifier. A model trained alone has to judge its own responses, which is prone to self-preference bias; this is the m=1 special case of our framework, which recovers single-model self-alignment such as Self-Rewarding and SPIN (§2).
- Peers' better responses. Each follower learns from duels in which a peer's response is preferred, which transfers complementary strengths between models in the pool. The leader's signal also exists only with multiple models: whether an instruction separates the dueling pair, and whether the peer judges can tell their responses apart (§2.1).
More broadly, a growing body of work studies the collaboration and collaborative training of multiple LLMs and shows the benefits of this direction over any single model, e.g., [1-8]. Our work builds on this direction and studies how to select the training instructions for collaborative training.
We additionally compare collaborative and independent training under the same computational budget. In independent training, each model in the pool is trained alone with the same EXP3 leader, judging its own responses, under both objectives: with DPO, the model generates two responses per instruction and is trained on the self-judged preference pair; with GRPO, it is trained on its own G samples scored by itself. Each uses the same number of iterations, generation and judge forward passes, and training steps as Stackelberg (DPO) and Stackelberg (GRPO), respectively, and we report the best model on the validation split, as for our method. Results on model Pool 1 (Avg is the mean over the four tasks):
| Method | BixBench | LabBench | SMDD | AssayBench | Avg |
|---|---|---|---|---|---|
| Independent training (DPO) | 0.194 | 0.273 | 0.102 | 0.005 | 0.144 |
| Independent training (GRPO) | 0.291 | 0.315 | 0.209 | 0.018 | 0.208 |
| Collaborative training (Stackelberg DPO) | 0.233 | 0.311 | 0.204 | 0.024 | 0.193 |
| Collaborative training (Stackelberg GRPO) | 0.349 | 0.322 | 0.237 | 0.023 | 0.233 |
Collaborative training outperforms independent training under the same budget with both objectives (0.193 vs. 0.144 with DPO; 0.233 vs. 0.208 with GRPO), confirming that the gains do not only come from additional training of each model, but also from the peer judges and peer responses that only the pool provides. This suggests that collaboration helps most where a single model's own signal is weakest: with DPO, a self-judged pair of the model's own two responses offers little contrast and inherits its self-preference bias, and collaborative training improves by 0.049, compared with 0.025 under GRPO, where the model already obtains contrast from its G samples. This experiment is now in Appendix B.11 of the revised PDF.
[1] Du and Kaelbling. Position: Compositional Generative Modeling: A Single Model is Not All You Need.
[2] Subramaniam et al. Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains.
[3] Feng et al. Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems.
[4] Jiang et al. Sparta Alignment: Collectively Aligning Multiple Language Models through Combat.
[5] Zhao et al. The Majority is Not Always Right: RL Training for Solution Aggregation.
[6] Park et al. MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning.
[7] Motwani et al. MALT: Improving Reasoning with Multi-Agent LLM Training.
[8] Estornell et al. ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration.
We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.
Thank you for the additional experiments. However, my core concern remains unresolved. Since the final output is a single LLM used independently at inference time, what is the fundamental advantage of jointly training multiple LLMs over simply training one policy with rewards provided by multiple external LLM judges through voting or consensus? Such a simpler approach could also benefit from diverse model judgments without requiring joint policy updates. The authors' comparison against independent self-judging models does not address this distinction, as it conflates the benefits of external peer evaluation with those of joint training. The paper still lacks a convincing theoretical or intuitive justification for why joint training itself is necessary or preferable.
Thank you for the follow-up. We ran the comparison the reviewer describes: one policy trained with rewards from multiple fixed external LLM judges, without joint policy updates.
We train each model in the pool on its own with GRPO, and the other m−1 models of the pool serve as fixed judges that are never updated. Everything else is identical to Stackelberg (ours): the same EXP3 leader, number of iterations, instructions per iteration, group size (G=4), and generation and judge forward passes. The only difference is that in Stackelberg (ours) all m models are updated jointly, so the judges and the dueling peers improve during training, while here they stay fixed. Columns 1 to m give the result of training each model of the pool this way (Pool 1: m=9; Pool 2: m=8; model order as in Appendix B.12), compared with Stackelberg (ours):
| Dataset, pool | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Stackelberg (ours) |
|---|---|---|---|---|---|---|---|---|---|---|
| BixBench, Pool 1 | 0.291 | 0.311 | 0.252 | 0.272 | 0.301 | 0.301 | 0.233 | 0.252 | 0.282 | 0.349 |
| LabBench, Pool 1 | 0.298 | 0.275 | 0.303 | 0.311 | 0.299 | 0.301 | 0.291 | 0.298 | 0.284 | 0.322 |
| BixBench, Pool 2 | 0.175 | 0.214 | 0.223 | 0.233 | 0.194 | 0.184 | 0.194 | 0.194 | – | 0.291 |
| LabBench, Pool 2 | 0.320 | 0.332 | 0.322 | 0.299 | 0.322 | 0.339 | 0.332 | 0.329 | – | 0.355 |
Stackelberg (ours) outperforms every model trained with fixed external judges on both datasets and both pools, including the best one in each row, confirming that the gains do not only come from external peer evaluation, but also from training the policies jointly. This suggests that joint training helps because the judges and the dueling peers improve together with the policy: fixed judges stay at their initial ability, so as the policy improves, their consensus can no longer tell its responses apart, while jointly trained peers keep providing informative rewards and stronger responses to learn from. This experiment is now in Appendix B.12 of the revised PDF.
Using a single model at inference is actually a strength of our approach in efficiency: it costs one model's inference, whereas multi-LLM inference methods such as Majority Vote, MoA, Multiagent Debate, and Heterogeneous Swarms run multiple models for every query. Stackelberg (ours) also outperforms all of them in macro-average on all three pools (Table 1), thanks to the strength of collaborative training. It reaches 0.563, 0.544, and 0.609 on Pools 1–3, compared with 0.464, 0.482, and 0.487 for the strongest multi-LLM inference method in each pool (Majority Vote, Heterogeneous Swarms, and MoA).
We made these edits and uploaded a revised PDF to OpenReview.
We would like to thank the reviewer for their thoughtful comments and feedback.
We compare against three existing adaptive curriculum and prompt selection methods, replacing only the EXP3 leader in Stackelberg (GRPO) and keeping everything else identical: the same followers, reputation-weighted peer-judge rewards, opponent-matching schedule, reputation system, and generation and update budget (T=8, D=64, G=4, LoRA, learning rate 1e-6).
Results on model Pool 1 (Avg is the mean over the four tasks):
The EXP3 leader outperforms all three prompt selection methods (0.233 vs. 0.203, 0.218, and 0.211), confirming that the gains do not only come from adaptive prompt selection in general, but also from selecting instructions by how the followers interact on them. SEC gains slightly over uniform sampling (0.218 vs. 0.211), CurES matches it (0.211), and DAPO dynamic sampling falls below it (0.203). This suggests that single-learner signals such as reward variance, advantage magnitude, or success rate are not by themselves informative under peer-judge rewards, whereas the leader's signals capture whether a duel separates the followers and whether the peer judges can tell their responses apart. This comparison is now in Appendix B.10 of the revised PDF.
The comparison also reflects how our setting differs from single-model prompt selection. Existing methods score an instruction by the progress of one learner on it (pass rate, advantage, or gradient signal), typically under verifiable rewards. Our leader instead scores an instruction by the interaction among the followers: whether the dueling pair is separated by it (difficulty) and whether the peer judges can tell their responses apart (preference quality, §2.1). The rewards come from reputation-weighted peer judges rather than from a verifier or an external reward model, and the leader also selects the opponent for each duel by the reputation gap (§2.2), which prompt selection methods do not have. We have expanded the related work to discuss DAPO, SEC, and CurES along these lines.
We do not use a fixed preference dataset generated by other models. In Stackelberg (DPO) the data is on-policy with respect to the follower pool, since the preference pairs are regenerated at every iteration by the current followers themselves: the leader draws D=64 duels, the two current followers in each duel respond to the instruction, the other current followers judge the responses, and the resulting pairs are used for that iteration's DPO update only (Algorithm 1). As the followers improve, so does their training data, which avoids the distribution drift of offline DPO on a static dataset.
Each follower does train on responses of the other followers. This is the collaborative mechanism rather than a side effect: a follower learns from the better responses of its peers, which is how complementary strengths transfer between models in the pool, as in Sparta Alignment.
For the fully on-policy setting, Stackelberg (GRPO) trains each follower only on its own sampled responses, scored by the peer judges. It further improves over Stackelberg (DPO) on model Pool 1 (macro-average 0.563 vs. 0.548; Table 1), and the leader improves results under both objectives (Sparta Alignment 0.524 → Stackelberg (DPO) 0.548; Sparta-GRPO 0.542 → Stackelberg (GRPO) 0.563, in our response to Reviewers gcGc and RUb3).