- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors ask whether a model’s standalone performance reflects its contribution to a collaborative system, and how training can improve that contribution. They define compositional strength using Shapley values, use leave-one-out contributions as a practical measure, and introduce C-RL to train one participant using the final system reward. The evaluation covers five model pools, twelve collaboration methods, and twelve datasets. Training experiments compare baselines across three protocols and eight tasks, with further tests of transfer across datasets and protocols.
Critical strengths
Separating standalone capability from system contribution across diverse pools, tasks, and collaboration methods offers a useful empirical perspective on model selection for collaborative systems.
Critical weaknesses
a. Figure 1 pools tasks and methods without explaining how repeated observations are handled, so it does not establish that standalone scores lack predictive value within a specific collaboration setting (section 2.3). b. Training uses system reward after baseline cancellation, although Appendix C still presents marginal reward as a distinction; without direct comparisons to existing collaborative RL, the methodological advance remains unclear (Eq. 7–9).
Questions for authors
a Does the relationship between standalone performance and leave-one-out contribution remain near zero within fixed datasets and collaboration methods, after accounting for repeated observations? b Can the complete coalition results already available in Appendix A.8 establish whether the main correlation finding also holds for exact Shapley values? c After baseline cancellation, what algorithmic additions does C-RL make over existing collaborative RL, beyond the setting of fixed heterogeneous collaborators? d Which protocol produced the collaborative RAFT training trajectories, and are its protocol matching and data budget comparable to C-RL’s (App A.1 and B.2)?
4 comments
(4) Additional robustness check on the main decoupling result itself. Beyond the estimator-agreement check above, we also stress-tested the central Figure 1 correlations (individual vs. compositional strength, not LOO vs. exact Shapley) with two more tools: Kendall's tau, which is more robust than Spearman rho at the panel sample sizes in Figure 1, and 95% confidence intervals from a block-level bootstrap that resamples (task, method) pairs with replacement within each panel, rather than relying on a single permutation p-value.
| Pool | Method family | n | rho | 95% CI for rho | tau |
|---|---|---|---|---|---|
| Specialized Expert | Routing | 192 | +0.186 | [+0.021, +0.354]* | +0.129 |
| Specialized Expert | Agentic | 192 | +0.058 | [-0.087, +0.211] | +0.034 |
| Specialized Expert | Training | 192 | +0.286 | [+0.122, +0.442]* | +0.184 |
| 8-30B Open | Routing | 336 | +0.003 | [-0.157, +0.166] | +0.004 |
| 8-30B Open | Agentic | 315 | +0.083 | [-0.064, +0.239] | +0.058 |
| 8-30B Open | Training | 168 | -0.122 | [-0.339, +0.122] | -0.060 |
| >100B Open | Routing | 144 | +0.151 | [+0.021, +0.294]* | +0.103 |
| >100B Open | Agentic | 132 | +0.123 | [-0.094, +0.328] | +0.084 |
| >100B Open | Training | 95 | +0.092 | [-0.100, +0.285] | +0.070 |
| Proprietary API | Routing | 111 | +0.043 | [-0.148, +0.244] | +0.031 |
| Proprietary API | Agentic | 129 | +0.146 | [-0.030, +0.330] | +0.092 |
| Proprietary API | Training | 63 | +0.112 | [-0.119, +0.319] | +0.063 |
| Diverse Academic | Routing | 384 | +0.009 | [-0.161, +0.191] | +0.014 |
| Diverse Academic | Agentic | 368 | -0.131 | [-0.306, +0.062] | -0.087 |
| Diverse Academic | Training | 384 | +0.103 | [-0.047, +0.244] | +0.066 |
(* bootstrap interval excludes zero.)
Across all 15 panels, tau tracks rho closely in both sign and magnitude (never differing by more than 0.10), and the bootstrap intervals corroborate the original picture: every panel remains consistent with weak association (|rho|, |tau| <= 0.29 throughout).
The bootstrap interval excludes zero in 3 of the 15 panels (Specialized Expert/Routing, Specialized Expert/Training, >100B Open/Routing), which only partly overlaps with the 3 panels flagged by the permutation test (Specialized Expert/Training, Diverse Academic/Agentic, Diverse Academic/Training); the two criteria agree only on Specialized Expert/Training, and every panel where they disagree sits close to one or the other threshold (|rho| <= 0.19 in all cases). This does not change the overall picture: no panel shows anything beyond a weak, borderline association by either criterion.
We also directly repeated the main decoupling test with exact Shapley values in place of LOO on the >100B Open pool, finding no significant relationship (rho = +0.12, p = 0.287), consistent with the pooled LOO-based result and directly answering this comment's question about whether the complete-coalition results in Appendix A.8 bear on the main correlation finding. Because exact Shapley requires 2^n coalition evaluations, we cannot run this direct check on every pool and method combination in Figure 1; given this, and the agreement we do observe wherever exact Shapley is computable, we believe the leave-one-out marginal is a reasonable proxy for the main decoupling claim. This is now included in the revised PDF, in the same Appendix A.8 paragraph and a new table.
After baseline cancellation C-RL trains on raw system reward, yet Appendix C still frames it as a marginal-contribution reward; without a direct comparison to existing collaborative RL, what does C-RL add algorithmically beyond the fixed-heterogeneous-collaborators setting
This raises three points, which we address in turn.
(1) The Shapley framing after cancellation. We have fixed the wording, matching the same correction already given to reviewers XYWv and hppM on this exact point. Equations 7-9 show explicitly that the leave-one-out baseline cancels out of the group-relative advantage, so C-RL trains on expected raw system reward for fixed collaborators; we never claimed otherwise in the derivation itself. The passage the reviewer is responding to is in Appendix C (Related Work), where the submitted version stated that "the reward is not the system's raw outcome but the policy's own marginal contribution to it." We have reworded this to state plainly that the training algorithm is a standard group-relative RL update on the shared system reward: the per-sample reward is the system's raw outcome itself, with the policy's marginal contribution recovered only at the group-relative advantage, never as a per-sample reward transformation.
(2) What C-RL adds. C-RL makes compositional strength itself the training objective: the per-sample reward is derived from the system's outcome under the leave-one-out/Shapley motivation (Section 2), rather than treating a shared collaborative reward as a given, as prior collaborative RL methods do.
This lets us broaden collaborative RL training to a setting and scale prior work has not covered: a single trainable policy embedded in a fixed pool of genuinely heterogeneous, independently-sourced collaborators (not copies of itself, not jointly initialized), trained once and shown to work across three structurally different collaboration protocols (LLM-Blender, Multiagent Debate, SLM-MUX) and eight tasks, and transferring zero-shot to unseen datasets, unseen protocols, and pools of frontier proprietary models never seen in training. Equation 9's turn-credit rule, broadcasting the single system-level reward to every one of the policy's generations in a rollout, is the mechanism that makes this work when the policy acts multiple times per rollout.
The result is a training recipe that needs no additional rollout cost beyond the collaborative rollouts it already runs (via the cancellation in point (1), which avoids the extra collaborators-only rollout a naive LOO baseline would otherwise require) and that transfers this broadly, which existing collaborative RL approaches, whether they co-train every participant (e.g. MAPoRL (Park et al., 2025), which co-trains all participating LLMs together via multi-turn discussion and reward) or re-derive the advantage grouping for shared prompts (e.g. AT-GRPO), do not establish.
(3) Direct empirical comparison. We agree a direct empirical comparison strengthens the claim, matching the same request from reviewer q9xn (who separately raised AT-GRPO, 2026). We ran two comparisons against this training setting, matched on protocol and system reward where applicable: one isolates the grouping-mechanism question by holding C-RL's frozen pool fixed and swapping only the advantage-grouping rule, and the other isolates the co-training question (MAPoRL and AT-GRPO) by instead reproducing each method in its own native regime:
| Condition | LLM-Blender | Multiagent Debate | SLM-MUX |
|---|---|---|---|
| C-RL (per-question grouping, as in paper) | .620 | .614 | .646 |
| C-RL w/ AT-GRPO-style per-turn grouping (new, policy only) | .587 | .592 | .601 |
| Full AT-GRPO (co-trained pool, role/turn grouping) | .608 | .615 | .623 |
| MAPoRL (co-trained multi-agent RL, all roles trained) | .614 | .582 | .610 |
C-RL outperforms both existing collaborative RL methods under LLM-Blender and SLM-MUX, and is within 0.001 of (co-trained, pool-wide) Full AT-GRPO under Multiagent Debate, its smallest margin of the three protocols.
Swapping only the advantage-grouping rule, from C-RL's per-question grouping to AT-GRPO-style per-turn grouping, while holding the frozen pool, protocol, and system reward fixed, costs 0.033-0.045 points in every protocol; this suggests that per-question grouping, not just the reward definition, is itself well-matched to C-RL's fixed-collaborator setting, and that per-turn grouping does not transfer benefit here even though it is motivated by a different setting (co-trained, role/turn-varying agents). We now include this ablation in Appendix B.2 of the revised PDF.
Which protocol produced the collaborative RAFT training trajectories, and is its protocol matching and data budget comparable to C-RL's?
Collaborative RAFT is trained separately per protocol, matching C-RL exactly on this point: for each of the three protocols (LLM-Blender, Multiagent Debate, SLM-MUX), RAFT's training transcripts are generated by running that same protocol with the untrained policy, and only transcripts from that protocol are used to fine-tune the corresponding RAFT checkpoint. Its data budget also matches C-RL's: up to the same 6,400 rollouts per protocol (N=8 questions x K=8 protocol executions per step, up to 100 steps), with RAFT keeping only the subset of these transcripts whose collaborative outcome scored well for supervised fine-tuning.
We also caught and corrected an inconsistency this question surfaced: Appendix A.1's individual-performance table reports collaborative RAFT's individual-mode score as a single row, with text describing it (incorrectly) as protocol-invariant like the other baselines. Since RAFT is in fact trained separately per protocol, its single reported row is the average of the three protocol-specific checkpoints' individual-mode scores, not a protocol-invariant number. We have corrected the wording in Appendix A.1 and the table caption to state this precisely, and added the protocol-matching and data-budget detail above to Appendix B.2, in the revised PDF.
We made these edits and uploaded a revised PDF to OpenReview. Let us know if you have any further questions.
Thanks for the response. I have updated my evaluation.
We would like to thank the reviewer for their thoughtful comments and feedback.
We address this in four steps.
(1) Two more pools. We compute exact Shapley values for the 8-30B Open (n=7, 128 coalitions) and Diverse Academic (n=8, 256 coalitions) pools, following the same protocol as the existing Specialized Expert check (4 datasets x 4 representative methods, one per family: Prompt Routing, Multiagent Debate, LLM-Blender, AggLM):
Both new, larger pools show Spearman agreement with exact Shapley at least as strong as the existing n=4 Specialized Expert check (0.63 and 0.69 vs. 0.42). This suggests that LOO's agreement with exact Shapley improves, rather than degrades, as pool size grows, and that the comparatively moderate pooled correlation on the smallest pool was in part an artifact of having only 4 models to rank. We now include this comparison in Appendix A.8 of the revised PDF.
(2) Re-analysis of the existing correlation study, respecting repeated observations. The reviewer is correct that Figure 1 pools (model, dataset, method) points that are not exchangeable: the same models recur across datasets and methods within each panel. Using the same 64 (model, dataset, method) rows already collected for the Specialized Expert pool (Fig. 7), we computed, separately for each of the 16 (dataset, method) blocks, the within-block Spearman correlation between LOO and exact Shapley (and Banzhaf), and ran a block-respecting permutation test that shuffles the LOO value only within each block (rather than across all 64 rows) before recomputing the pooled statistic.
Within-block agreement is substantially stronger than the pooled number suggests (mean rho = +0.71, median +0.80 against exact Shapley, vs. pooled 0.417; 94% of blocks positive), and the pooled correlation is highly significant once the block structure is respected (p < 1e-4 against exact Shapley, 10,000 permutations). We read this as evidence that LOO is a sound proxy for ranking models within a fixed task and collaboration method, even though the single pooled number across heterogeneous settings understates this.
(3) Updated claim in the paper. We have updated both the main text and Appendix A.8 to report this fuller picture, while still not overclaiming equivalence to exact Shapley: