acceptodds
Under review as a conference paper at ICLR 2027

Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning

Abstract

Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. For models inside systems, what matters most is not how well it performs on its own, but how much the system gains from its presence. We call this a model’s compositional strength, formalize it as the model’s Shapley value in a cooperative game over coalitions of a model pool, and estimate it with leave-one-out marginals costing n+1 system runs rather than 2n. Evaluating both quantities across five model pools, twelve collaboration methods, and twelve datasets, we find them largely uncorrelated: in 12 of 15 (pool, method-family) settings individual strength bears no significant relationship to compositional strength, and the coupling is weakest among general-purpose models that practitioners are most likely to deploy. Individual benchmarks and optimizing for them thus mean little for models’ values in collaborative systems. We then propose compositional reinforcement learning (C-RL), which places a policy inside a live multi-model collaborative system and rewards it with that system’s outcome rather than its own correctness. The leave-one-out baseline in Shapley values cancels under a group policy gradient, so this costs no more than ordinary RL and never requires running a coalition without the policy. Across three collaboration protocols and eight tasks, C-RL produces the strongest collaborator in 21 of 24 settings, is the only method that never leaves the system worse off than the untrained policy, and carries over to unseen datasets, unseen collaboration protocols, and pools of frontier proprietary models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model collaboration, compositional strength, modular AI systems

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_1tajOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors ask whether a model’s standalone performance reflects its contribution to a collaborative system, and how training can improve that contribution. They define compositional strength using Shapley values, use leave-one-out contributions as a practical measure, and introduce C-RL to train one participant using the final system reward. The evaluation covers five model pools, twelve collaboration methods, and twelve datasets. Training experiments compare baselines across three protocols and eight tasks, with further tests of transfer across datasets and protocols.

Critical strengths

Separating standalone capability from system contribution across diverse pools, tasks, and collaboration methods offers a useful empirical perspective on model selection for collaborative systems.

Critical weaknesses

a. Figure 1 pools tasks and methods without explaining how repeated observations are handled, so it does not establish that standalone scores lack predictive value within a specific collaboration setting (section 2.3). b. Training uses system reward after baseline cancellation, although Appendix C still presents marginal reward as a distinction; without direct comparisons to existing collaborative RL, the methodological advance remains unclear (Eq. 7–9).

Questions for authors

a Does the relationship between standalone performance and leave-one-out contribution remain near zero within fixed datasets and collaboration methods, after accounting for repeated observations? b Can the complete coalition results already available in Appendix A.8 establish whether the main correlation finding also holds for exact Shapley values? c After baseline cancellation, what algorithmic additions does C-RL make over existing collaborative RL, beyond the setting of fixed heterogeneous collaborators? d Which protocol produced the collaborative RAFT training trajectories, and are its protocol matching and data budget comparable to C-RL’s (App A.1 and B.2)?

4 comments

We would like to thank the reviewer for their thoughtful comments and feedback.

Figure 1 pools tasks and methods without explaining how repeated observations are handled; does the standalone-vs-contribution decoupling hold within fixed datasets/methods, and does it hold for exact Shapley values (per Appendix A.8)?

We address this in four steps.

(1) Two more pools. We compute exact Shapley values for the 8-30B Open (n=7, 128 coalitions) and Diverse Academic (n=8, 256 coalitions) pools, following the same protocol as the existing Specialized Expert check (4 datasets x 4 representative methods, one per family: Prompt Routing, Multiagent Debate, LLM-Blender, AggLM):

PoolnSpearman rhoPearson r
Specialized Expert (existing, Fig. 7)40.4170.661
8-30B Open70.6270.691
Diverse Academic80.6920.713

Both new, larger pools show Spearman agreement with exact Shapley at least as strong as the existing n=4 Specialized Expert check (0.63 and 0.69 vs. 0.42). This suggests that LOO's agreement with exact Shapley improves, rather than degrades, as pool size grows, and that the comparatively moderate pooled correlation on the smallest pool was in part an artifact of having only 4 models to rank. We now include this comparison in Appendix A.8 of the revised PDF.

(2) Re-analysis of the existing correlation study, respecting repeated observations. The reviewer is correct that Figure 1 pools (model, dataset, method) points that are not exchangeable: the same models recur across datasets and methods within each panel. Using the same 64 (model, dataset, method) rows already collected for the Specialized Expert pool (Fig. 7), we computed, separately for each of the 16 (dataset, method) blocks, the within-block Spearman correlation between LOO and exact Shapley (and Banzhaf), and ran a block-respecting permutation test that shuffles the LOO value only within each block (rather than across all 64 rows) before recomputing the pooled statistic.

Within-block agreement is substantially stronger than the pooled number suggests (mean rho = +0.71, median +0.80 against exact Shapley, vs. pooled 0.417; 94% of blocks positive), and the pooled correlation is highly significant once the block structure is respected (p < 1e-4 against exact Shapley, 10,000 permutations). We read this as evidence that LOO is a sound proxy for ranking models within a fixed task and collaboration method, even though the single pooled number across heterogeneous settings understates this.

(3) Updated claim in the paper. We have updated both the main text and Appendix A.8 to report this fuller picture, while still not overclaiming equivalence to exact Shapley:

  • Section 1, "Evaluating LLMs by compositional strength": changed "...costing only n+1 runs, and tracking the exact value closely (Appendix A.8)" to "...costing only n+1 runs; it correlates only moderately with the exact value when pooled across settings, but substantially more closely in rank within a fixed task and collaboration method (Appendix A.8)."
  • Appendix A.8: added a new paragraph reporting the within-block means/medians and the block-respecting permutation p-values from (2), explicitly noting that the pooled-exchangeability assumption is why the headline number understates rank agreement.

(4) Additional robustness check on the main decoupling result itself. Beyond the estimator-agreement check above, we also stress-tested the central Figure 1 correlations (individual vs. compositional strength, not LOO vs. exact Shapley) with two more tools: Kendall's tau, which is more robust than Spearman rho at the panel sample sizes in Figure 1, and 95% confidence intervals from a block-level bootstrap that resamples (task, method) pairs with replacement within each panel, rather than relying on a single permutation p-value.

PoolMethod familynrho95% CI for rhotau
Specialized ExpertRouting192+0.186[+0.021, +0.354]*+0.129
Specialized ExpertAgentic192+0.058[-0.087, +0.211]+0.034
Specialized ExpertTraining192+0.286[+0.122, +0.442]*+0.184
8-30B OpenRouting336+0.003[-0.157, +0.166]+0.004
8-30B OpenAgentic315+0.083[-0.064, +0.239]+0.058
8-30B OpenTraining168-0.122[-0.339, +0.122]-0.060
>100B OpenRouting144+0.151[+0.021, +0.294]*+0.103
>100B OpenAgentic132+0.123[-0.094, +0.328]+0.084
>100B OpenTraining95+0.092[-0.100, +0.285]+0.070
Proprietary APIRouting111+0.043[-0.148, +0.244]+0.031
Proprietary APIAgentic129+0.146[-0.030, +0.330]+0.092
Proprietary APITraining63+0.112[-0.119, +0.319]+0.063
Diverse AcademicRouting384+0.009[-0.161, +0.191]+0.014
Diverse AcademicAgentic368-0.131[-0.306, +0.062]-0.087
Diverse AcademicTraining384+0.103[-0.047, +0.244]+0.066

(* bootstrap interval excludes zero.)

Across all 15 panels, tau tracks rho closely in both sign and magnitude (never differing by more than 0.10), and the bootstrap intervals corroborate the original picture: every panel remains consistent with weak association (|rho|, |tau| <= 0.29 throughout).

The bootstrap interval excludes zero in 3 of the 15 panels (Specialized Expert/Routing, Specialized Expert/Training, >100B Open/Routing), which only partly overlaps with the 3 panels flagged by the permutation test (Specialized Expert/Training, Diverse Academic/Agentic, Diverse Academic/Training); the two criteria agree only on Specialized Expert/Training, and every panel where they disagree sits close to one or the other threshold (|rho| <= 0.19 in all cases). This does not change the overall picture: no panel shows anything beyond a weak, borderline association by either criterion.

We also directly repeated the main decoupling test with exact Shapley values in place of LOO on the >100B Open pool, finding no significant relationship (rho = +0.12, p = 0.287), consistent with the pooled LOO-based result and directly answering this comment's question about whether the complete-coalition results in Appendix A.8 bear on the main correlation finding. Because exact Shapley requires 2^n coalition evaluations, we cannot run this direct check on every pool and method combination in Figure 1; given this, and the agreement we do observe wherever exact Shapley is computable, we believe the leave-one-out marginal is a reasonable proxy for the main decoupling claim. This is now included in the revised PDF, in the same Appendix A.8 paragraph and a new table.

After baseline cancellation C-RL trains on raw system reward, yet Appendix C still frames it as a marginal-contribution reward; without a direct comparison to existing collaborative RL, what does C-RL add algorithmically beyond the fixed-heterogeneous-collaborators setting

This raises three points, which we address in turn.

(1) The Shapley framing after cancellation. We have fixed the wording, matching the same correction already given to reviewers XYWv and hppM on this exact point. Equations 7-9 show explicitly that the leave-one-out baseline cancels out of the group-relative advantage, so C-RL trains on expected raw system reward for fixed collaborators; we never claimed otherwise in the derivation itself. The passage the reviewer is responding to is in Appendix C (Related Work), where the submitted version stated that "the reward is not the system's raw outcome but the policy's own marginal contribution to it." We have reworded this to state plainly that the training algorithm is a standard group-relative RL update on the shared system reward: the per-sample reward is the system's raw outcome itself, with the policy's marginal contribution recovered only at the group-relative advantage, never as a per-sample reward transformation.

(2) What C-RL adds. C-RL makes compositional strength itself the training objective: the per-sample reward is derived from the system's outcome under the leave-one-out/Shapley motivation (Section 2), rather than treating a shared collaborative reward as a given, as prior collaborative RL methods do.

This lets us broaden collaborative RL training to a setting and scale prior work has not covered: a single trainable policy embedded in a fixed pool of genuinely heterogeneous, independently-sourced collaborators (not copies of itself, not jointly initialized), trained once and shown to work across three structurally different collaboration protocols (LLM-Blender, Multiagent Debate, SLM-MUX) and eight tasks, and transferring zero-shot to unseen datasets, unseen protocols, and pools of frontier proprietary models never seen in training. Equation 9's turn-credit rule, broadcasting the single system-level reward to every one of the policy's generations in a rollout, is the mechanism that makes this work when the policy acts multiple times per rollout.

The result is a training recipe that needs no additional rollout cost beyond the collaborative rollouts it already runs (via the cancellation in point (1), which avoids the extra collaborators-only rollout a naive LOO baseline would otherwise require) and that transfers this broadly, which existing collaborative RL approaches, whether they co-train every participant (e.g. MAPoRL (Park et al., 2025), which co-trains all participating LLMs together via multi-turn discussion and reward) or re-derive the advantage grouping for shared prompts (e.g. AT-GRPO), do not establish.

(3) Direct empirical comparison. We agree a direct empirical comparison strengthens the claim, matching the same request from reviewer q9xn (who separately raised AT-GRPO, 2026). We ran two comparisons against this training setting, matched on protocol and system reward where applicable: one isolates the grouping-mechanism question by holding C-RL's frozen pool fixed and swapping only the advantage-grouping rule, and the other isolates the co-training question (MAPoRL and AT-GRPO) by instead reproducing each method in its own native regime:

ConditionLLM-BlenderMultiagent DebateSLM-MUX
C-RL (per-question grouping, as in paper).620.614.646
C-RL w/ AT-GRPO-style per-turn grouping (new, policy only).587.592.601
Full AT-GRPO (co-trained pool, role/turn grouping).608.615.623
MAPoRL (co-trained multi-agent RL, all roles trained).614.582.610

C-RL outperforms both existing collaborative RL methods under LLM-Blender and SLM-MUX, and is within 0.001 of (co-trained, pool-wide) Full AT-GRPO under Multiagent Debate, its smallest margin of the three protocols.

Swapping only the advantage-grouping rule, from C-RL's per-question grouping to AT-GRPO-style per-turn grouping, while holding the frozen pool, protocol, and system reward fixed, costs 0.033-0.045 points in every protocol; this suggests that per-question grouping, not just the reward definition, is itself well-matched to C-RL's fixed-collaborator setting, and that per-turn grouping does not transfer benefit here even though it is motivated by a different setting (co-trained, role/turn-varying agents). We now include this ablation in Appendix B.2 of the revised PDF.

Which protocol produced the collaborative RAFT training trajectories, and is its protocol matching and data budget comparable to C-RL's?

Collaborative RAFT is trained separately per protocol, matching C-RL exactly on this point: for each of the three protocols (LLM-Blender, Multiagent Debate, SLM-MUX), RAFT's training transcripts are generated by running that same protocol with the untrained policy, and only transcripts from that protocol are used to fine-tune the corresponding RAFT checkpoint. Its data budget also matches C-RL's: up to the same 6,400 rollouts per protocol (N=8 questions x K=8 protocol executions per step, up to 100 steps), with RAFT keeping only the subset of these transcripts whose collaborative outcome scored well for supervised fine-tuning.

We also caught and corrected an inconsistency this question surfaced: Appendix A.1's individual-performance table reports collaborative RAFT's individual-mode score as a single row, with text describing it (incorrectly) as protocol-invariant like the other baselines. Since RAFT is in fact trained separately per protocol, its single reported row is the average of the three protocol-specific checkpoints' individual-mode scores, not a protocol-invariant number. We have corrected the wording in Appendix A.1 and the table caption to state this precisely, and added the protocol-matching and data-budget detail above to Appendix B.2, in the revised PDF.

We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.

Thanks for the response. I have updated my evaluation.

Reviewer_q9xnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates models by their contribution to a collaborative system rather than their standalone performance. It defines compositional strength using Shapley values, approximates it with leave-one-out contributions, and proposes C-RL, which trains a model using the final system reward. Experiments cover several model pools, collaboration methods, and tasks.

Critical strengths
  1. Evaluating what a model adds to a system is useful, which covers a broad range of collaborative settings.

  2. Training with system-level rewards gives consistent improvements, with tests on unseen tasks and collaboration settings.

Critical weaknesses
  1. The training contribution needs clearer positioning against existing collaborative RL [1]. Once the collaborators-only baseline cancels, C-RL becomes policy optimization with a shared system reward. This is sensible, but the Shapley framing does not introduce a new credit-assignment mechanism. Comparisons mainly with solo-reward RL, prompting, and RAFT do not establish the advantage over existing collaborative RL.

  2. I am not convinced that the evaluation supports the broad claim that individual and compositional strength are unrelated. The experiments mainly use LOO contributions, whose rank agreement with exact Shapley is only moderate in Appendix A.8. Figure 1 also pools different datasets and methods. Weak pooled correlations and non-significant tests do not establish that standalone performance is uninformative within a given setting.

References

[1] Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs, 2026.

Questions for authors

None

2 comments

We would like to thank the reviewer for their thoughtful comments and feedback.

The training contribution needs clearer positioning against existing collaborative RL (specifically Stronger-MAS/AT-GRPO), since after baseline cancellation C-RL is policy optimization with a shared system reward, and comparisons mainly with solo-reward RL, prompting, and RAFT don't establish the advantage over existing collaborative RL.

We already cite this work (as Zhao et al., 2025, in the submitted version); we have updated the citation to the published ICLR 2026 version and sharpened the comparison, and we agree the distinction is worth making explicit.

(1) Positioning against Stronger-MAS/AT-GRPO. Stronger-MAS's contribution is a grouping mechanism: AT-GRPO forms agent- and turn-wise groups so that a group-relative baseline remains valid when prompts differ by role and by turn, typically while co-training several agents (role-shared or role-specialized) at once. Our contribution is a reward definition, not a grouping mechanism: we keep a single fixed group per question (the frozen collaborator pool plus the policy), train only the policy within it, and broadcast one constant, question-level system reward to every one of the policy's turns (Eq. 9).

Because the group composition and the reward target never vary by role or turn in our setting, the role/turn-wise grouping problem that AT-GRPO solves does not arise for us; conversely, nothing in AT-GRPO's grouping mechanism specifies what reward a multi-agent system should optimize, which is the question C-RL addresses via the leave-one-out/Shapley motivation. The two contributions are complementary, not competing, and in principle AT-GRPO-style grouping could be applied on top of a C-RL-style reward once more than one agent is being trained.

We have added one sentence making this distinction explicit in Appendix C (Related Work), alongside the corrected citation; we have kept the change minor, consistent with the fact that Stronger-MAS was already discussed there.

(2) Comparisons with existing collaborative RL. We agree a direct empirical comparison strengthens the claim, and added an ablation that isolates the grouping-mechanism question from the reward question. We take our own training setting (same frozen pool, same protocol, same system reward) and replace C-RL's per-question pooled group with AT-GRPO-style per-turn/per-role grouping for the advantage baseline, training only the policy and holding the reward itself fixed:

ConditionLLM-BlenderMultiagent DebateSLM-MUX
C-RL (per-question grouping, as in paper).620.614.646
C-RL w/ AT-GRPO-style per-turn grouping (new, policy only).587.592.601
Full AT-GRPO (co-trained pool, role/turn grouping).608.615.623
MAPoRL (co-trained multi-agent RL, all roles trained).614.582.610

C-RL outperforms both existing collaborative RL methods under LLM-Blender and SLM-MUX, and is within 0.001 of (co-trained, pool-wide) Full AT-GRPO under Multiagent Debate, its smallest margin of the three protocols.

Swapping only the advantage-grouping rule, from C-RL's per-question grouping to AT-GRPO-style per-turn grouping, while holding the frozen pool, protocol, and system reward fixed, costs 0.033-0.045 points in every protocol; this suggests that per-question grouping, not just the reward definition, is itself well-matched to C-RL's fixed-collaborator setting, and that per-turn grouping does not transfer benefit here even though it is motivated by a different setting (co-trained, role/turn-varying agents). We now include this ablation in Appendix B.2 of the revised PDF.

Not convinced individual and compositional strength are truly unrelated, since LOO's agreement with exact Shapley is only moderate (Appendix A.8); weak pooled correlations don't establish that standalone performance is uninformative within a given setting.

We address this in four steps.

(1) Two more pools. We compute exact Shapley values for the 8-30B Open (n=7, 128 coalitions) and Diverse Academic (n=8, 256 coalitions) pools, following the same protocol as the existing Specialized Expert check (4 datasets x 4 representative methods, one per family: Prompt Routing, Multiagent Debate, LLM-Blender, AggLM):

PoolnSpearman rhoPearson r
Specialized Expert (existing, Fig. 7)40.4170.661
8-30B Open70.6270.691
Diverse Academic80.6920.713

Both new, larger pools show Spearman agreement with exact Shapley at least as strong as the existing n=4 Specialized Expert check (0.63 and 0.69 vs. 0.42). This suggests that LOO's agreement with exact Shapley improves, rather than degrades, as pool size grows, and that the comparatively moderate pooled correlation on the smallest pool was in part an artifact of having only 4 models to rank. We now include this comparison in Appendix A.8 of the revised PDF.

(2) Re-analysis of the existing correlation study. The pooled correlation in Fig. 7/App. A.8 mixes points that are not exchangeable: the same four models recur across datasets and methods, and the pooled Spearman conflates differences in score scale across these settings with genuine rank disagreement. Using the same 64 (model, dataset, method) rows already collected for Fig. 7, we instead computed, separately for each of the 16 (dataset, method) blocks, the within-block Spearman correlation between LOO and exact Shapley (and Banzhaf), and ran a block-respecting permutation test that shuffles the LOO value only within each block (rather than across all 64 rows) before recomputing the pooled statistic.

Within-block agreement is substantially stronger than the pooled number suggests (mean rho = +0.71, median +0.80 against exact Shapley, vs. pooled 0.417; 94% of blocks positive), and the pooled correlation is highly significant once the block structure is respected (p < 1e-4 against exact Shapley, 10,000 permutations). We read this as evidence that LOO is a sound proxy for ranking models within a fixed task and collaboration method, even though the single pooled number across heterogeneous settings understates this.

(3) Updated claim in the paper. We have updated both the main text and Appendix A.8 to report this fuller picture, while still not overclaiming equivalence to exact Shapley:

  • Section 1, "Evaluating LLMs by compositional strength": changed "...costing only n+1 runs, and tracking the exact value closely (Appendix A.8)" to "...costing only n+1 runs; it correlates only moderately with the exact value when pooled across settings, but substantially more closely in rank within a fixed task and collaboration method (Appendix A.8)."
  • Appendix A.8: added a new paragraph reporting the within-block means/medians and the block-respecting permutation p-values from (2), explicitly noting that the pooled-exchangeability assumption is why the headline number understates rank agreement.

(4) Additional robustness check on the main decoupling result itself. Beyond the estimator-agreement check above, we also stress-tested the central Figure 1 correlations (individual vs. compositional strength, not LOO vs. exact Shapley) with two more tools: Kendall's tau, which is more robust than Spearman rho at the panel sample sizes in Figure 1, and 95% confidence intervals from a block-level bootstrap that resamples (task, method) pairs with replacement within each panel, rather than relying on a single permutation p-value.

PoolMethod familynrho95% CI for rhotau
Specialized ExpertRouting192+0.186[+0.021, +0.354]*+0.129
Specialized ExpertAgentic192+0.058[-0.087, +0.211]+0.034
Specialized ExpertTraining192+0.286[+0.122, +0.442]*+0.184
8-30B OpenRouting336+0.003[-0.157, +0.166]+0.004
8-30B OpenAgentic315+0.083[-0.064, +0.239]+0.058
8-30B OpenTraining168-0.122[-0.339, +0.122]-0.060
>100B OpenRouting144+0.151[+0.021, +0.294]*+0.103
>100B OpenAgentic132+0.123[-0.094, +0.328]+0.084
>100B OpenTraining95+0.092[-0.100, +0.285]+0.070
Proprietary APIRouting111+0.043[-0.148, +0.244]+0.031
Proprietary APIAgentic129+0.146[-0.030, +0.330]+0.092
Proprietary APITraining63+0.112[-0.119, +0.319]+0.063
Diverse AcademicRouting384+0.009[-0.161, +0.191]+0.014
Diverse AcademicAgentic368-0.131[-0.306, +0.062]-0.087
Diverse AcademicTraining384+0.103[-0.047, +0.244]+0.066

(* bootstrap interval excludes zero.)

Across all 15 panels, tau tracks rho closely in both sign and magnitude (never differing by more than 0.10), and the bootstrap intervals corroborate the original picture: every panel remains consistent with weak association (|rho|, |tau| <= 0.29 throughout).

The bootstrap interval excludes zero in 3 of the 15 panels (Specialized Expert/Routing, Specialized Expert/Training, >100B Open/Routing), which only partly overlaps with the 3 panels flagged by the permutation test (Specialized Expert/Training, Diverse Academic/Agentic, Diverse Academic/Training); the two criteria agree only on Specialized Expert/Training, and every panel where they disagree sits close to one or the other threshold (|rho| <= 0.19 in all cases). This does not change the overall picture: no panel shows anything beyond a weak, borderline association by either criterion. This is now included in the revised PDF, in the same Appendix A.8 paragraph and a new table.

We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.

Reviewer_hppMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a model's compositional strength as its Shapley value in a cooperative game, where v(S)=Pm,t(S)v(S)=P_{m,t}(S) is the performance of collaboration method mm on task tt using model coalition SS. It estimates this with the leave-one-out (LOO) marginal ϕ^i=v(N)−v(N∖{i})\hat\phi_i=v(\mathcal N)-v(\mathcal N\setminus\{i\}), which needs n+1n+1 evaluations instead of 2n2^n.

  1. Evaluation: across 5 pools, 12 methods and 12 datasets, solo strength and LOO strength are not significantly correlated in 12/15 (pool, method-family) cells (∣ρ∣≤0.29|\rho|\le 0.29).
  2. C-RL: a policy is trained with group-relative policy gradients inside a system of frozen collaborators, rewarded by the system's output. The collaborators-only baseline cancels, so no extra rollouts are needed.
  3. Results: Qwen 3.6 27B with three frozen collaborators, on 3 protocols × 8 tasks:
    • best in 21/24 cells;
    • beats the untrained policy in all 24 (+4.0 to +5.4 pts on average);
    • reported to transfer to held-out datasets and protocols, and to a proprietary-model pool (after retraining).
Critical strengths
  1. A model's marginal value inside multi-model systems matters in practice. The 5×12×12 sweep in one toolkit is a reusable resource, even if its result is read narrowly as "value added to an existing pool".
  2. C-RL needs only the scalar system reward and works with closed-API collaborators. It beats the untrained policy in all 24 cells, and the comparison includes collaboration-aware baselines (prompt nudge, RAFT).
Critical weaknesses

W1. The game-theoretic framing does less work than claimed, and the headline evaluation result is specific to LOO.

(a) Training. The collaborators are frozen, so Pm,t(C)P_{m,t}(\mathcal C) in Eq. 5 does not depend on θ\theta, and arg⁡max⁡θϕ^πθ=arg⁡max⁡θPm,t(C∪{πθ}).\arg\max_\theta \hat\phi_{\pi_\theta} = \arg\max_\theta P_{m,t}(\mathcal C\cup\{\pi_\theta\}).

  • The cancellation in Eq. 7 is just the standard invariance of the policy gradient to an action-independent baseline. C-RL is therefore GRPO on the system reward with frozen partners.
  • Appendix C claims the reward is "not the system's raw outcome but the policy's own marginal contribution." This contradicts Eq. 8.
  • Eq. 6 is a difference reward in the sense of Wolpert & Tumer (cf. COMA's counterfactual baseline). The per-sample counterfactual that would make it useful is never used. For LLM-Blender and SLM-MUX it would be cheap: re-run only the ranker/fuser or selector on cached collaborator outputs. It would also address free-riding. Currently the policy is fully rewarded on "rescue" items where its own answer is wrong, which is 11–18% of items (Table 5).
  • The claim that a Shapley objective would be more expensive is incorrect. With Shapley weights w(S)=∣S∣! (n−∣S∣−1)!n!w(S)=\frac{|S|!\,(n-|S|-1)!}{n!}, ∇θϕπθ=ES∼w, S⊆C[∇θ v(S∪{πθ})],\nabla_\theta \phi_{\pi_\theta} = \mathbb E_{S\sim w,\ S\subseteq\mathcal C}\big[\nabla_\theta\, v(S\cup\{\pi_\theta\})\big], so it needs one rollout per sampled coalition. Notably, this objective includes the solo coalition with weight 1/n1/n.
  • In a two-model oracle team (CoRE, Thm. 4.1), ϕ^i=P(Ci=1,Cj=0)≤min⁡{pi, 1−pj}\hat\phi_i = P(C_i=1, C_j=0) \le \min\{p_i,\, 1-p_j\}, whereas ϕi=pi−pij/2≥pi/2\phi_i = p_i - p_{ij}/2 \ge p_i/2 with pij=P(Ci=Cj=1)p_{ij}=P(C_i=C_j=1): near ceiling, LOO →0\to 0 regardless of pip_i, while Shapley stays at least half the solo accuracy.

(b) Evaluation. The Shapley value contains the term 1n [v({i})−v(∅)]\tfrac1n\,[v(\{i\})-v(\emptyset)], so individual strength is mechanically a 1/n1/n share of compositional strength as the paper defines it. For the n=3n=3 proprietary pool, where decoupling is called "sharpest," that share is 1/31/3.

  • LOO is the semivalue that puts zero weight on this term. Eq. 4 argues that LOO is "the ∣S∣=n−1|S|=n-1 term up to 1/n1/n." The same argument applies to individual strength, which is the S=∅S=\emptyset term with the same weight.
  • Appendix A.8 shows weak rank agreement between LOO and exact Shapley (Spearman 0.31 and 0.42). Yet exact Shapley is cheap for three of the five pools (232^3 or 242^4 coalitions).
  • A.8's own explanation, that LOO "suppresses the credit assigned to models whose competence is duplicated," predicts near-zero LOO in capable, redundant pools near ceiling. That is exactly where decoupling is reported to be strongest.

The statistics also do not support "uncorrelated":

  • Points are pooled across datasets, which conflates dataset difficulty with model differences.
  • Points that share v(N)v(\mathcal N) are not exchangeable, so the permutation test should permute within blocks.
  • Each LOO value is a difference of two (apparently single-run) scores at temperature 0.7. Its reliability is low, which attenuates correlations toward zero.
  • Non-significance is not evidence of zero correlation; there are no CIs or equivalence tests.

Consequence: both main contributions rest on this framing. The evaluation claim may not hold under the paper's own definition. The training contribution reduces to RL on a system reward in a frozen pool, and its novelty over Router-R1, AggLM, MLPO, CollabLLM, SWEET-RL, and difference rewards is not established.

W2. The C-RL experiments do not establish improved compositional strength or transferable collaboration skill.

  • The optimized quantity is never reported. There is no Pm,t(C)P_{m,t}(\mathcal C) and no solo score for the collaborators, so we cannot tell whether ϕ^π>0\hat\phi_\pi>0 even after training.
  • Rigor.
    • There is one training seed, one rollout per test item at temperature 0.7, and CIs only for C-RL rather than for paired differences.
    • In Table 4, 13 of 24 per-dataset cells have C-RL's lower CI bound at or below Base. A.2 says these exceptions are "concentrated on the two smallest and noisiest tasks," but they span 7 of 8 datasets.
    • Under Debate, the margin over the best baseline is +1.1.
    • Each checkpoint is the best of up to ~33 evaluations on 16 dev questions. It is unclear whether baselines were selected on the same (collaborative) criterion, and compute is not matched.
  • Confounds. Under LLM-Blender and Debate, C-RL is also individually stronger than GRPO-indiv (Table 3: .547/.544 vs .526). Its solo gains (+3.9 to +6.3) are as large as its collaborative gains (+4.0 to +5.4). This points to an under-tuned baseline, or to in-context distillation from larger collaborators, rather than collaboration skill. Other mechanisms are untested:
    • truncation, given a 16,384-token sequence with a 15,000-token response budget, and a 34k-character base candidate shrinking to ~4k after training (Table 6);
    • aggregator bias, since gpt-oss-120b is both a collaborator and the ranker;
    • learned conformity to the majority;
    • KernelBench reward hacking with no KL anchor.
  • Generalization is overstated.
    • The frontier-pool result requires retraining, yet the abstract says C-RL "carries over."
    • Held-out sets are small (APEX-Shortlist: 40 items, where C-RL gains 3 problems; BixBench: 103 items), and the checkpoint–dataset pairings appear selected.
    • MoA is structurally close to LLM-Blender.
    • Prompt-collab is excluded from held-out evaluation even though its prompt transfers trivially.
    • Consistency across protocols comes from separately trained policies, so it does not show that one policy's skills transfer.

Consequence: the paper's central empirical claim, that C-RL produces better collaborators with generalizable skills, depends on these comparisons.


Other issues

Claims that do not match the paper's own numbers.

  • The intro says "close to three times the gain of the strongest baseline," but under Debate the ratio is 1.4× (+4.0 vs +2.9).
  • "Each of the five baselines regresses somewhere": there are only four non-Base methods.
  • From the three-decimal tables I count 18 (GRPO), 20 (DAPO), and 13 (RAFT) strict improvements over Base, not 19/21/14.
  • The "three execution-verified tasks" include AssayBench, which Table 11 and B.2 say needs no execution.
  • The +27% vs +7.4% contrast is a relative-gain artifact of near-zero baselines. In absolute points, answer-matching tasks gain more (≈+5.2 vs ≈+3.9).
  • Table 3 calls the baselines protocol-independent, but RAFT-collab and prompt-collab are defined per protocol.
  • In Table 7, the GPQA responses do not "agree on the whole reaction sequence": one forms a cyclohexene, the other the correct five-membered aldol ring.
  • A.3 calls Table 6's second item "drag," but there the policy's candidate is wrong, which contradicts Table 5's definition.
  • §2.4 singles out LLM-Blender and AggLM, while Fig. 2 shows Heterogeneous Swarms (0.61) and SLM-MUX (0.56) highest. The rates also imply ≈210 top-3 placements over 60 pairs, versus at most 180 without ties, so ties are frequent and should be reported.
  • "Collaboration is effective (+11.5)" (Table 8, which equals Fig. 5 ×100) compares the full coalition with leave-one-out coalitions, maximized over (dataset, method). It does not compare against the best single model max⁡iv({i})\max_i v(\{i\}) or against compute-matched self-consistency.
  • The "∼20–30B" pool is mostly 8–14B models.
  • "Collaborators larger than the policy" needs qualifying: gpt-oss-120b has ≈5B active parameters.
  • The eight tasks are described three different ways in the abstract, intro, and §3.2.
  • The reproducibility statement announces two caveats but gives only one.

Missing related work.

  • Credit assignment: difference rewards (Wolpert & Tumer; Agogino & Tumer); COMA (Foerster et al., 2018); Shapley-based MARL credit (Shapley Q-value, AAAI 2020; SHAQ, NeurIPS 2022; Li et al., KDD 2021); the lazy-agent problem (Sunehag et al., VDN).
  • Training an agent for a fixed or unseen team: ad hoc teamwork (Stone et al., 2010; Mirsky et al. survey), Other-Play, Fictitious Co-Play.
  • Prior art for the thesis itself: Bansal et al., "Is the Most Accurate AI the Best Teammate?" (AAAI 2021); Wilder et al., "Learning to Complement Humans" (IJCAI 2020).
  • Ensembles: boosting and negative correlation learning (Liu & Yao, 1999); Krogh & Vedelsby's ambiguity decomposition; ensemble selection (Caruana et al., 2004); Shapley values of classifiers in ensemble games (Rozemberczki & Sarkar, CIKM 2021).
  • LLM collaboration: CollabLLM; SWEET-RL; Sirius; CoRE (Mishra et al., ICML 2026; arXiv:2601.21600), which trains models to complement each other via peer hints and complementarity rewards; Self-MoA; debate critiques (Smit 2024; Wang 2024); self-consistency/More Agents.
  • Whether mixing helps at all: Self-MoA (Li et al., 2025) finds that mixing different LLMs often underperforms ensembling the best single model. Also debate critiques (Smit et al., ICML 2024; Wang et al., ACL 2024) and compute-matched baselines (self-consistency; "More Agents Is All You Need").
  • RL baselines and LLM judges: RLOO (Kool et al., 2019; Ahmadian et al., 2024); Dr. GRPO (Liu et al., 2025); self-preference and position bias in LLM judges (Panickssery et al., 2024; Wang et al., 2023).
  • Related work currently appears only in the appendix.

Benchmarks and setup.

  • Several section 2 datasets are at ceiling (ARC-C, MMLU-Redux, HumanEval) or at floor (AssayBench, KernelBench) for capable pools.
  • Splits are nonstandard subsets: GPQA-Diamond 99/198, HumanEval 114/164, TheoremQA 400/800, KernelBench 125/250.
  • CoCoNot normally uses an LLM judge, so the rule-based scorer should be described. The KernelBench reward is custom.
  • Decoding settings (thinking mode, reasoning effort, API sampling restrictions) are unspecified.
  • Qwen2.5-Math-7B is a base model with, to my knowledge, a 4K context window.
  • section 2 and section 3 use different setups under the same protocol names: Debate runs 3 vs 2 rounds, and the LLM-Blender ranker is Qwen2.5-7B vs gpt-oss-120b.
  • The release status of the Diverse Academic pool is not stated.
  • experiments on new more capable models like qwen 3.5-4b(,9b,27b) and gemma4-e2b(,e4b,12b,31b) are also required.

Presentation and references.

  • Wording:
    • "clear enough" → "clearly enough" (section 3.1).
    • "Sampling KK independent rollouts … and denote TqT_q" is a sentence fragment.
    • The footnote mark in "C-RL¹" reads as part of the name.
    • "spanning reasoning, scientific, multilingual, and pluralism" is not parallel.
    • "empowers small policy" → "empowers a small policy".
    • "enhance the policy as better collaborators" → "make the policy a better collaborator".
    • There are contractions and informal phrasing, and US/UK spelling is mixed.
  • Equations, figures, and tables:
    • Eq. 4 ends with a comma before a new sentence.
    • Tick labels in Figs. 1–2 read 0.2/0.8 where the ticks appear to be at 0.25/0.75.
    • The Fig. 1 caption says rows 3–4, while §2.3 says rows 2–4.
    • Table 1 (AssayBench, SLM-MUX) marks the two .109 entries differently.
    • Table 2 uses "percentage points" for non-percentage metrics.
    • Table 11 cites a "scoring split of section 3.2" that does not exist.
  • Naming and notation:
    • Method names are inconsistent: prompt nudge vs prompt-collab, collaborative RAFT vs RAFT-collab.
    • NN denotes both the pool and the number of questions per step.
    • A.7's efficiency axiom should read ∑iϕi=v(N)−v(∅)\sum_i\phi_i = v(\mathcal N)-v(\emptyset).
  • References:
    • Shapley (1953) is sole-authored.
    • Du et al. is listed at the 41st ICML but dated 2023.
    • NeurIPS vol. 38 entries are dated 2026.
    • "Brouwer" in the text vs "De Brouwer" in the bibliography.
    • Several entries lack venues.
Questions for authors
  1. What are Pm,t(C)P_{m,t}(\mathcal C) (the system without the policy) and each collaborator's solo score for every cell of Tables 1–2? Is ϕ^π=Pm,t(C∪{π})−Pm,t(C)\hat\phi_\pi = P_{m,t}(\mathcal C\cup\{\pi\})-P_{m,t}(\mathcal C) positive before and after C-RL?
  2. On the 64 Specialized-Expert observations in A.8, what is the Spearman correlation between individual strength and exact Shapley, compared with LOO? Can you compute exact Shapley for the n=3n=3 and n=4n=4 pools and redo Fig. 1 within (dataset, method) cells?
  3. How were baseline checkpoints selected: on individual or collaborative dev score, and with the same budget as C-RL? How many training seeds were run?
  4. Which roles does the policy play in each protocol? Who is the fuser in the §3 LLM-Blender setup? Is candidate order randomized? How is Debate's final answer chosen?
  5. Given the 16,384-token sequence and 15,000-token response budget, what fraction of policy and collaborator outputs are truncated or unparseable, before and after C-RL?
  6. How are RAFT-collab and prompt-collab protocol-independent in Table 3? Which protocol's transcripts were used to train RAFT-collab?
  7. In Table 8, is the baseline the leave-one-out score? How does v(N)v(\mathcal N) compare with max⁡iv({i})\max_i v(\{i\}) and with compute-matched self-consistency? What explains the Diverse Academic / GPQA-Diamond cell (65.3 vs 32.4)?
  8. For routing methods, are routers retrained for each leave-one-out coalition? How are order-dependent roles handled when a model is removed?
  9. Given KL =0=0, were the KernelBench kernels audited for harness exploits or timing noise?

Suggested experiments

  • At least 3 seeds, multiple rollouts per test item, and paired tests against the strongest baseline.
  • Ablations: training over Shapley-sampled coalitions, a per-sample counterfactual (difference) reward for LLM-Blender and SLM-MUX, and a KL anchor.
  • Transfer to an unseen pool without retraining, including a pool weaker than the policy. Evaluate all checkpoints on all held-out sets, and include prompt-collab in held-out evaluations.
  • A distillation baseline (SFT on the collaborators' solutions), a sample- or compute-matched GRPO baseline, and a compute and cost table.
  • Agreement with the majority before vs after training, to test for learned conformity.
  • Relate C-RL to training-time complementarity objectives (e.g., CoRE's rescue and cross-model diversity rewards), and test whether adding such a term counters conformity.
3 comments

Respectfully, this review is obviously AI-generated top-to-bottom, with more than 60+(!) bullet points, and perhaps the reviewer didn't bother to take a look since many of them are incorrect and don't make sense. We won't dignify this kind of behavior with a point-by-point response. We fix applicable minor things like typos and briefly respond to the high-level claims for the record.

(1) C-RL as GRPO on the system reward with frozen partners. This is correct and not in dispute, and matches the response we have already given reviewer XYWv on the same point: the leave-one-out baseline cancels out of the group-relative advantage because the collaborators are frozen, so training reduces to a standard group-relative update on the raw system reward. The submitted version's related-work passage did read as contradicting this, stating that "the reward is not the system's raw outcome but the policy's own marginal contribution to it"; we have corrected this, matching the same fix given to reviewer XYWv, to state plainly that the per-sample reward is the system's raw outcome itself, with the policy's marginal contribution recovered only at the group-relative advantage, never as a per-sample reward transformation.

Separately, the claim that a Shapley-weighted objective would be cheaper, not more expensive, does not hold: a per-sample Shapley-weighted reward requires sampling coalitions down to and including the one that excludes the policy entirely, i.e. rollouts where the policy acts without any collaborators. That is exactly the no-policy rollout our baseline-cancellation result is built to avoid, so it is not free in the sense our efficiency claim relies on.

(2) The decoupling result is partly mechanical under Shapley's efficiency axiom. We agree that Shapley's efficiency decomposition structurally includes a 1/n-weighted individual-strength term, and this is not in dispute either. What Figure 1 and our LOO/Shapley validation test is whether the remaining, non-mechanical component, the relative ranking of models by compositional strength versus by individual strength, still carries independent information once that structural term is accounted for. Our within-block reanalysis and the direct LOO-vs-exact-Shapley comparison, also given to reviewer XYWv, show substantial divergence between the two rankings even after respecting this structure, which is the basis for the decoupling claim:

PoolnSpearman rhoPearson r
Specialized Expert (existing, Fig. 7)40.4170.661
8-30B Open70.6270.691
Diverse Academic80.6920.713

Both new, larger pools show Spearman agreement with exact Shapley at least as strong as the existing n=4 Specialized Expert check (0.63 and 0.69 vs. 0.42), suggesting that LOO's agreement with exact Shapley improves, rather than degrades, as pool size grows. We now include this comparison in Appendix A.8 of the revised PDF. We agree the paper should state this more precisely rather than imply the two measures are fully independent, and have revised the relevant sentences in Section 2 and Appendix A.8 accordingly.

We have also stress-tested the central Figure 1 correlations themselves (individual vs. compositional strength) with two further tools: Kendall's tau, more robust than Spearman rho at these panel sample sizes, and 95% confidence intervals from a block-level bootstrap that resamples (task, method) pairs with replacement within each panel, rather than relying on a single permutation p-value.

PoolMethod familynrho95% CI for rhotau
Specialized ExpertRouting192+0.186[+0.021, +0.354]*+0.129
Specialized ExpertAgentic192+0.058[-0.087, +0.211]+0.034
Specialized ExpertTraining192+0.286[+0.122, +0.442]*+0.184
8-30B OpenRouting336+0.003[-0.157, +0.166]+0.004
8-30B OpenAgentic315+0.083[-0.064, +0.239]+0.058
8-30B OpenTraining168-0.122[-0.339, +0.122]-0.060
>100B OpenRouting144+0.151[+0.021, +0.294]*+0.103
>100B OpenAgentic132+0.123[-0.094, +0.328]+0.084
>100B OpenTraining95+0.092[-0.100, +0.285]+0.070
Proprietary APIRouting111+0.043[-0.148, +0.244]+0.031
Proprietary APIAgentic129+0.146[-0.030, +0.330]+0.092
Proprietary APITraining63+0.112[-0.119, +0.319]+0.063
Diverse AcademicRouting384+0.009[-0.161, +0.191]+0.014
Diverse AcademicAgentic368-0.131[-0.306, +0.062]-0.087
Diverse AcademicTraining384+0.103[-0.047, +0.244]+0.066

(* bootstrap interval excludes zero.)

Across all 15 panels, tau tracks rho closely in both sign and magnitude (never differing by more than 0.10), and the bootstrap intervals corroborate the original picture: every panel remains consistent with weak association (|rho|, |tau| <= 0.29 throughout).

The bootstrap interval excludes zero in 3 of the 15 panels (Specialized Expert/Routing, Specialized Expert/Training, >100B Open/Routing), which only partly overlaps with the 3 panels flagged by the permutation test (Specialized Expert/Training, Diverse Academic/Agentic, Diverse Academic/Training); the two criteria agree only on Specialized Expert/Training, and every panel where they disagree sits close to one or the other threshold (|rho| <= 0.19 in all cases). This does not change the overall picture: no panel shows anything beyond a weak, borderline association by either criterion. This is now included in the revised PDF, in the same Appendix A.8 paragraph and a new table.

(3) C-RL's individual-mode gains are comparable in size to its collaborative gains. We do not read this as evidence against the collaboration-skill claim; if anything it is a positive result we should state more clearly. A single training signal, system-level reward, improving both compositional and individual strength together is a feature of C-RL, not a confound to explain away: Appendix A.1 already reports this as a by-product rather than the objective, and the central decoupling finding is precisely why this does not happen for free with other methods (e.g. collaborative RAFT is tied with GRPO-indiv individually, .524 vs. .526, yet is the weakest collaborator overall).

Whatever the exact mechanism behind the individual-mode gain, likely some combination of reward shaping and in-context exposure to stronger collaborators during rollouts, it does not substitute for the collaborative gain: Table 6's rescue rate (11-18% of items) shows the system still depends on collaborators substantially after training, so the collaborative improvement is not simply the policy's own answers being forwarded unchanged. We have reframed this explicitly as a positive secondary finding in Section 3.3 rather than leaving it as an implicit aside.

We also fix a handful of typos and minor wording clarifications flagged in the review, in the revised PDF.

We made these edits and uploaded a revised PDF to OpenReview.

Reviewer_hppMOpenReview ↗

Reviewer comment to Authors rebuttal

Thank you for the the detailed responses and revised PDF. The new seeds, the local-reward control, the new baselines and the clearer roles help. I withdraw my comments on the 19/21/14 counts and on extra model families.

Two main issues remain. The paper still does not test whether solo scores predict contribution inside one task and method. It also compares solo scores with exact Shapley for only one pool, though Table 10 shows exact Shapley exists for three more. The text still says routing and large pools show no significant link, while your Table 11 finds one in both. Further, it'd have been great if changes in the revised version have been highlighted.

The gains over baselines are not yet statistically clear. Only C-RL has several seeds, and with 5 seeds no margin passes a standard 95% test. The system's score without the policy is still missing. The abstract still says C-RL "carries over" to frontier models, although section 3.4 retrains.

Considering submitted rebuttal and other reviewers' points, I'll keep my score.

Reviewer_XYWvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a model's standalone performance need not predict its usefulness within a collaborative system. It formalizes this usefulness through a cooperative game over model coalitions, defines compositional strength using the Shapley value, and uses the grand-coalition leave-one-out (LOO) marginal as an inexpensive operational proxy. The evaluation study covers five model pools, twelve collaboration methods, and twelve datasets. Its main finding is that pooled correlations between standalone scores and LOO contributions are small across the examined pool/method-family settings.

The training contribution, C-RL, fine-tunes one policy while its heterogeneous collaborators remain frozen. Each policy generation receives the final system reward, and a group-relative objective updates only tokens sampled by the policy. Because the collaborators-only reward is independent of the policy, subtracting it does not change the objective's gradient and cancels in group-relative advantages; no additional collaborators-only rollout is required. Experiments train Qwen 3.6 27B under LLM-Blender, Multiagent Debate, and SLM-MUX on eight tasks. Table 1 reports average improvements over the untrained collaborative system of 4.8, 4.0, and 5.4 points. Additional experiments examine held-out datasets and protocols, while the frontier-collaborator experiment retrains with a stronger pool rather than demonstrating zero-shot transfer to that pool. The practical results are promising, but the Shapley interpretation, the statistical decoupling claim, and attribution of the training gains require stronger support.

Critical strengths
  1. A useful systems-level evaluation question studied across unusually broad settings. Sections 2.1–2.3 evaluate contribution within a specific pool, task, and protocol rather than assume that a standalone leaderboard determines model selection. The five pools and twelve methods cover several distinct forms of collaboration. Appendix B.1 also addresses a concrete removal confound by fixing the evaluation-time LLM-Blender ranker/fuser rather than allowing removal of a pool member to silently change the fuser. The use of the same test items and rule-based or execution-based scoring across coalitions makes the resulting LOO measurements useful operational evidence, even though their interpretation as Shapley estimates needs qualification.

  2. Promising end-to-end training results with relevant transfer checks. Table 1 shows higher C-RL point estimates than Base in all 24 task/protocol settings, and the protocol-level averages improve by 4.0–5.4 points. Comparisons include both solo-reward RL and collaboration-aware prompting and RAFT. Figures 3–4 test held-out datasets and protocols without further training, and Table 2 tests retraining with frontier collaborators. Appendix B.2 explains how only the policy's own generations are recorded for optimization, avoiding inadvertent teacher forcing on collaborators' outputs. Together, these results support the practical value of training a participant in the environment where it will be used.

Critical weaknesses
  1. The headline decoupling result does not yet establish the claimed relationship with Shapley-defined compositional strength. There are two related inference gaps. First, LOO measures contribution specifically to the full pool, whereas Shapley averages over all coalitions. Equation 4 identifies one term of the Shapley sum but gives no approximation guarantee. The authors appropriately acknowledge this distinction in Appendix A.8, yet their own validation gives only Spearman correlation 0.417 with exact Shapley on one specialized four-model pool (Figure 7, p. 22). Positive Pearson correlation of 0.661 does not establish close agreement in model rankings or show that the main standalone-versus-contribution result survives the substitution. Second, Figure 1 pools different datasets and individual collaboration methods within each panel. The same models recur, solo scores are reused across methods, and observations share coalition evaluations. This pooled correlation is different from asking whether standalone scores rank candidate collaborators usefully within a fixed task and method. The permutation exchangeability unit is not clearly specified, and a non-significant correlation is not evidence that all predictive signal is absent. The authors should report within-task/method ranking agreement and uncertainty that respects the shared observations, and recompute the central correlation using the exact Shapley values already available for the Appendix A.8 subset. Otherwise, the conclusion should be limited to weak pooled associations with this particular full-pool LOO measure.

  2. The training comparison does not isolate the contribution of the system-level reward, and its reliability and efficiency are incompletely established. Appendix B.2 explicitly states that GRPO-indiv and DAPO-indiv use single-model rollouts, while C-RL uses full protocol executions. Thus reward choice changes together with access to collaborator responses, role/turn contexts, policy-call counts, and total sampling cost. Collaboration-aware RAFT is useful but does not control these differences in an on-policy RL comparison. A policy trained inside the same frozen collaboration environment using its own local correctness reward would more directly test the central reward claim; policy-token, collaborator-token, and total compute budgets should be reported alongside it. The Shapley framing also does not create a distinct reward after cancellation: Equations 5–8 optimize expected raw system reward for fixed collaborators. Appendix C's contrary wording overstates the algorithmic distinction from shared-outcome RL. Finally, Appendix B.2 reports one fixed-seed test rollout per question, but independent training-run counts are not clearly specified. Appendix A.2 supplies C-RL confidence intervals, explicitly not hypothesis tests, rather than paired differences against Base or the strongest baseline. These intervals do not establish reliable superiority in 21 cells or quantify training variability; the average margin over the best baseline is only 1.1 points under Multiagent Debate. Matched in-system reward controls, paired difference intervals, and repeated key runs would make the claimed advantage and its source much more convincing.

Questions for authors
  1. Does the main decoupling finding survive a change of estimand and statistical unit? On the existing Appendix A.8 experiments, what is the relationship between standalone scores and exact Shapley values? What are the within-dataset/method model-ranking correlations, and how do uncertainty estimates change when shared models and coalition evaluations are respected rather than treating every plotted point as exchangeable?

  2. Is there a matched in-system reward control? For answer-producing calls with a defined task-local correctness score, what happens if the policy receives the same collaborator messages but is rewarded for its own answer rather than the final system output, with other roles held fixed? What policy-generation, collaborator-generation, and wall-clock/GPU budgets underlie the C-RL and solo-RL comparisons? This would materially affect whether I attribute the gains to the proposed reward or to the broader training environment.

  3. What uncertainty supports the main comparative claims? How many independent training seeds were run, and what are the paired test-item confidence intervals for C-RL minus the strongest baseline under each protocol? Please specify how Table 4's intervals and macro-average intervals were constructed, including for continuous execution/reference-ranking scores. Do the small positive cells and the 1.1-point Multiagent Debate average remain reliable with repeated generation or training runs?

  4. Which policy roles and collaboration components are used during training and evaluation? Section 3.2 describes the policy contributing in diverse LLM-Blender roles; Appendix B.1 fixes the evaluation-study ranker/fuser to Qwen2.5-7B, whereas Appendix B.2 fixes the training ranker to gpt-oss-120b without explicitly identifying the fuser. Please list the ranker, fuser, policy roles, and removal behavior for the training study and its collaborative evaluation. This determines whether the policy is being improved as a candidate contributor, a final decision-maker, or both.

6 comments

We would like to thank the reviewer for their thoughtful comments and feedback.

Does the headline decoupling result survive a change of estimand (exact Shapley vs. LOO) and statistical unit (within-task/method ranking vs. pooled correlation)?

We address this in four steps.

(1) Two more pools. We compute exact Shapley values for the 8-30B Open (n=7, 128 coalitions) and Diverse Academic (n=8, 256 coalitions) pools, following the same protocol as the existing Specialized Expert check (4 datasets x 4 representative methods, one per family: Prompt Routing, Multiagent Debate, LLM-Blender, AggLM):

PoolnSpearman rhoPearson r
Specialized Expert (existing, Fig. 7)40.4170.661
8-30B Open70.6270.691
Diverse Academic80.6920.713

Both new, larger pools show Spearman agreement with exact Shapley at least as strong as the existing n=4 Specialized Expert check (0.63 and 0.69 vs. 0.42). This suggests that LOO's agreement with exact Shapley improves, rather than degrades, as pool size grows, and that the comparatively moderate pooled correlation on the smallest pool was in part an artifact of having only 4 models to rank. We now include this comparison in Appendix A.8 of the revised PDF.

(2) Re-analysis of the existing correlation study. The pooled correlation in Fig. 7/App. A.8 mixes points that are not exchangeable: the same four models recur across datasets and methods, and the pooled Spearman conflates differences in score scale across these settings with genuine rank disagreement. Using the same 64 (model, dataset, method) rows already collected for Fig. 7, we instead computed, separately for each of the 16 (dataset, method) blocks, the within-block Spearman correlation between LOO and exact Shapley (and Banzhaf), and ran a block-respecting permutation test that shuffles the LOO value only within each block (rather than across all 64 rows) before recomputing the pooled statistic.

Within-block agreement is substantially stronger than the pooled number suggests (mean rho = +0.71, median +0.80 against exact Shapley, vs. pooled 0.417; 94% of blocks positive), and the pooled correlation is highly significant once the block structure is respected (p < 1e-4 against exact Shapley, 10,000 permutations). We read this as evidence that LOO is a sound proxy for ranking models within a fixed task and collaboration method, even though the single pooled number across heterogeneous settings understates this.

(3) Updated claim in the paper. We have updated both the main text and Appendix A.8 to report this fuller picture, while still not overclaiming equivalence to exact Shapley:

  • Section 1, "Evaluating LLMs by compositional strength": changed "...costing only n+1 runs, and tracking the exact value closely (Appendix A.8)" to "...costing only n+1 runs; it correlates only moderately with the exact value when pooled across settings, but substantially more closely in rank within a fixed task and collaboration method (Appendix A.8)."
  • Appendix A.8: added a new paragraph reporting the within-block means/medians and the block-respecting permutation p-values from (2), explicitly noting that the pooled-exchangeability assumption is why the headline number understates rank agreement.

(4) Additional robustness check on the main decoupling result itself. Beyond the estimator-agreement check above, we also stress-tested the central Figure 1 correlations (individual vs. compositional strength, not LOO vs. exact Shapley) with two more tools: Kendall's tau, which is more robust than Spearman rho at the panel sample sizes in Figure 1, and 95% confidence intervals from a block-level bootstrap that resamples (task, method) pairs with replacement within each panel, rather than relying on a single permutation p-value.

PoolMethod familynrho95% CI for rhotau
Specialized ExpertRouting192+0.186[+0.021, +0.354]*+0.129
Specialized ExpertAgentic192+0.058[-0.087, +0.211]+0.034
Specialized ExpertTraining192+0.286[+0.122, +0.442]*+0.184
8-30B OpenRouting336+0.003[-0.157, +0.166]+0.004
8-30B OpenAgentic315+0.083[-0.064, +0.239]+0.058
8-30B OpenTraining168-0.122[-0.339, +0.122]-0.060
>100B OpenRouting144+0.151[+0.021, +0.294]*+0.103
>100B OpenAgentic132+0.123[-0.094, +0.328]+0.084
>100B OpenTraining95+0.092[-0.100, +0.285]+0.070
Proprietary APIRouting111+0.043[-0.148, +0.244]+0.031
Proprietary APIAgentic129+0.146[-0.030, +0.330]+0.092
Proprietary APITraining63+0.112[-0.119, +0.319]+0.063
Diverse AcademicRouting384+0.009[-0.161, +0.191]+0.014
Diverse AcademicAgentic368-0.131[-0.306, +0.062]-0.087
Diverse AcademicTraining384+0.103[-0.047, +0.244]+0.066

(* bootstrap interval excludes zero.)

Across all 15 panels, tau tracks rho closely in both sign and magnitude (never differing by more than 0.10), and the bootstrap intervals corroborate the original picture: every panel remains consistent with weak association (|rho|, |tau| <= 0.29 throughout).

The bootstrap interval excludes zero in 3 of the 15 panels (Specialized Expert/Routing, Specialized Expert/Training, >100B Open/Routing), which only partly overlaps with the 3 panels flagged by the permutation test (Specialized Expert/Training, Diverse Academic/Agentic, Diverse Academic/Training); the two criteria agree only on Specialized Expert/Training, and every panel where they disagree sits close to one or the other threshold (|rho| <= 0.19 in all cases). This does not change the overall picture: no panel shows anything beyond a weak, borderline association by either criterion.

We also directly repeated the main decoupling test with exact Shapley values in place of LOO on the >100B Open pool, finding no significant relationship (rho = +0.12, p = 0.287), consistent with the pooled LOO-based result. Because exact Shapley requires 2^n coalition evaluations, we cannot run this direct check on every pool and method combination in Figure 1; given this, and the agreement we do observe wherever exact Shapley is computable, we believe the leave-one-out marginal is a reasonable proxy for the main decoupling claim. This is now included in the revised PDF, in the same Appendix A.8 paragraph and a new table.

The training comparison doesn't isolate the system-level reward's own contribution from rollout/budget differences, and reports CIs rather than paired differences; asks for a matched in-system reward control, compute budgets, and uncertainty quantification.

We agree with all three points and address them in turn.

(1) An in-environment, local-reward ablation, plus compute/token budgets. GRPO-indiv/DAPO-indiv change reward and rollout structure at once, so they cannot by themselves isolate the system-level reward's contribution; collaborative RAFT runs the real protocol but is off-policy supervised fine-tuning, not an on-policy RL control. We ran the requested control: a policy trained with the same frozen collaborators, the same protocol, the same infrastructure, and the same group-relative RL update as C-RL, but with the per-sample reward replaced by the policy's own local correctness on its turn rather than the system outcome. We report it as Local-RL-collab alongside the existing conditions. Budgets for all training conditions, drawn directly from the infrastructure and rollout numbers already in Appendix B.2:

ConditionRollout unitPolicy calls/stepCollab. calls/stepRL stepsWall-clock
GRPO-indiv/DAPO-indivsingle-model64 (fixed)0up to 100~21h
Local-RL-collab (new)full protocolvariable, >=64variable, >0up to 100~30h
C-RLfull protocolvariable, >=64variable, >0up to 100~30h
Collaborative RAFTfull protocol (once, for data collection)variablevariable0 (SFT only)~28h
Prompt nudgefull protocol (eval only)0 (no training)0 (no training)00

Results for Local-RL-collab:

ProtocolBaseGRPO/DAPO-indiv (best)prompt-collabRAFT-collabLocal-RL-collab (new)C-RL
LLM-Blender.572.588.588.577.582.620
Multiagent Debate.574.603.595.589.605.614
SLM-MUX.592.610.606.582.621.646

Local-RL-collab is the strongest non-C-RL condition under Multiagent Debate and SLM-MUX (.605 and .621), but trails GRPO/DAPO-indiv and prompt-collab under LLM-Blender (.582 vs. .588); in every protocol it still falls short of C-RL by a clear margin (-0.038, -0.009, -0.025 respectively).

This suggests that training on-policy inside the real, protocol-matched rollout narrows the gap to C-RL in some settings but does not close it, confirming that C-RL's system-level reward itself, not merely the rollout structure it shares with this control, accounts for its remaining advantage. We now include this ablation, with the wall-clock budgets above, in Appendix B.2 of the revised PDF.

(2) The Shapley framing after cancellation. Equations 5-8 show explicitly that the leave-one-out baseline cancels out of the group-relative advantage, so C-RL trains on expected raw system reward for fixed collaborators; we never claimed otherwise in the derivation itself. The passage the reviewer is responding to is in Appendix C (Related Work), where the submitted version stated that "the reward is not the system's raw outcome but the policy's own marginal contribution to it." We have reworded this to state plainly that the training algorithm is a standard group-relative RL update on the shared system reward: the per-sample reward is the system's raw outcome itself, with the policy's marginal contribution recovered only at the group-relative advantage, never as a per-sample reward transformation.

(3) Paired differences and training variability. Appendix A.2's intervals are per-cell C-RL-vs-Base intervals and each cell comes from a single training run, since running multi-node RL is expensive. Each interval is a 95% bootstrap interval over test items, resampled with replacement at that task's true test-set size: for the binary-scored tasks this resamples the 0/1 correctness of each item, and for the continuous execution/reference-ranking tasks (KernelBench, SMDD-Bench, AssayBench) it resamples each item's continuous score directly, in both cases recomputing the task mean on each resample; the macro-average interval resamples all eight tasks' item sets jointly within each replicate before averaging. To directly quantify training variability, we reran C-RL across 5 independent seeds for each of the three training protocols, and report the mean and standard error below against the existing single-run best baseline:

ProtocolBest baseline (single run)C-RL, 5-seed mean +/- SEMargin (5-seed mean - best baseline)
LLM-Blender.588.622 +/- .019+0.034
Multiagent Debate.603.612 +/- .006+0.009
SLM-MUX.610.651 +/- .015+0.041

This suggests that the margin over the best baseline holds up, and is if anything slightly larger, under repeated-seed training: the 5-seed mean exceeds the best baseline in all three protocols, by more than one standard error in LLM-Blender and SLM-MUX (1.8x and 2.7x SE respectively) and by 1.5x SE in Multiagent Debate, where the single-run margin was already the smallest of the three.

We take the 5-seed standard error, rather than a single-run paired item-level bootstrap, as the right way to establish uncertainty here: it directly measures how much the margin itself moves across independent training runs, the source of variability the reviewer's question is about, whereas a paired bootstrap on one run's test items would only capture test-set sampling noise and say nothing about training stochasticity. We now include this training-variability table in Appendix A.2 of the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.