acceptodds
Under review as a conference paper at ICLR 2027

Collator: Compositional Multi-Agent Orchestration with Counterfactual Reinforcement Learning

Abstract

Large language models (LLMs) provide a flexible foundation for multi-agent systems, but their effectiveness and computational cost depend critically on orchestration design. Across different tasks, role design, capacity assignment, and dependency construction jointly affect both solution quality and execution efficiency. Existing approaches automate parts of this design process, yet they often optimize these decisions partially or sequentially, and rely on execution-level feedback that provides limited credit assignment for local orchestration decisions. We propose Collator (Compositional Multi-Agent Orchestration with Counterfactual Reinforcement Learning), an LLM-based orchestrator that learns to design efficient multi-agent orchestration. Given a task, Collator designs a unified orchestration specification that composes customized agent duties, capacity levels, and dependency relations. To train the orchestrator, we augment the orchestration-level Group Relative Policy Optimization (GRPO) objective with a localized counterfactual credit signal that edits role, capacity, or dependency fields and applies the resulting reward contrast only to the edited spans. Experiments on six reasoning and coding benchmarks, including MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, show that Collator achieves the best average performance among evaluated multi-agent orchestration methods while improving the accuracy–token trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Multi Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EyojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COLLATOR trains a Qwen2.5-7B orchestrator to generate one YAML spec per query. The spec sets each agent’s duty, its worker size (7B/14B/32B), and its upstream dependencies, and it is validated, compiled to a DAG and executed. Training is SFT, then GRPO on a reward combining (1) correctness, (2) a token saving bonus and (3) a structural penalty. It adds a “localized counterfactual” loss: mutate one field (delete an edge, revert a duty to generic, or downgrade capacity), re-execute, and apply a DPO-style preference on that field’s tokens only, according to the sign of the reward difference. It reports the best average across six benchmarks and a better accuracy–token trade-off.

Critical strengths
  1. The method is intuitive, and MAS orchestration is an important problem. Executable, field-structured spec makes field-level credit cheap. The design lets the authors mutate a single field, re-run only the affected downstream nodes via caching, and attach the reward contrast to exactly those tokens. This is a practical, reusable recipe.
  2. The technique also aims at efficiency, not just accuracy. The paper reports worker tokens alongside accuracy in every ablation. It also discloses the training overhead of the counterfactual loss (+21.6% training tokens). This makes the efficiency claims checkable.
Critical weaknesses
  1. Gains are not clearly attributed and baseline is unclear. In particular, orchestration gains are not separated from model size. COLLATOR can route work to a 32B worker. The Vanilla row (MMLU 74.46, GSM8K 85.59) matches published Qwen2.5-7B scores, which suggests the baselines run on 7B. Table 1 cannot show that orchestration design, rather than bigger models, drives the gains. Can the authors clarify baseline and settings?
  2. The gains are small, and the counterfactual loss may just reward lower cost. The main results (Table 1) come from a single run with no error bars. If decoding is non-greedy, margins of 1–4 test items on HumanEval, MultiArith and SVAMP could be noise. Doubling the counterfactual loss weight (0.05 to 0.10) also drops accuracy by 6–10 points. A control, such as simply raising the token reward, is needed to show it does more than cutting tokens.
  3. Novelty is also somewhat limited: MAS-GPT (Ye et al., ICML 2025) already generates a full query-specific MAS in one pass, and MasRouter (Yue et al., ACL 2025) chooses roles and models together.
Questions for authors
  1. Which worker model(s) does each baseline in Table 1 use? Adding the following baselines would make the work much more solid:14B and 32B workers, a fixed MAS with all-32B workers, and MasRouter baselines under the same worker pool.
  2. Please report mean ±\pm std over at least 3 seeds for Table 1 and Table 2 (if nongreedy sampling + exact match / pass@1 is used).
  3. How does the counterfactual loss compare with simply raising λ\lambda_tok at matched token usage? What fraction of retained counterfactual pairs prefer the factual spec?
  4. Some experiment settings are unclear. Where do the 480 MMLU RL examples and the SFT questions come from? Does cross_dataset_v1 train one orchestrator for all six tasks or one per task?
  5. C(x,y)C(x,y) counts worker tokens only. What is the orchestrator’s own generation cost (up to 2048 tokens) and the end-to-end latency given graph depth?
4 comments

W 1: Vanilla, CoT, SC-CoT, and the other multi-agent baselines in the original paper all use Qwen2.5-14B-Instruct. COLLATOR uses a 7B orchestrator and 7B, 14B, and 32B workers. The table below summarizes the results for Vanilla and the newly added Free baseline across model sizes. Vanilla is instructed to output only the final answer; Free is prompted to answer the question without requiring either a final-answer-only response or step-by-step reasoning. We report accuracy for MMLU and the other non-code benchmarks, and pass@1 for HumanEval, all as percentages.

MethodModel sizeMMLUGSM8KAQuAMultiArithSVAMPHumanEval
Vanilla7B70.6176.6439.7653.6766.3079.03
Vanilla14B74.4685.5946.4676.0080.8080.65
Vanilla32B81.9190.2155.9195.5085.8082.25
Free7B72.3791.5881.8998.8393.8080.65
Free14B80.9394.1681.1098.8394.4081.45
Free32B83.2895.8383.8698.8393.9083.87
COLLATOR7B/14B/32B84.3294.3286.2299.1795.6084.68

The following table further quantifies the contribution of capacity allocation by comparing the ablation with all workers fixed at 14B against dynamic capacity allocation. Both settings retain the task responsibilities and dependency design.

DatasetAccuracy gain from dynamic capacity allocation (percentage points)Reduction in weighted worker tokens
MMLU+1.5636.8%
GSM8K+1.1422.0%
AQuA+1.5750.5%

W 2: In the three additional paired training comparisons on MMLU, the counterfactual term yields accuracy gains of +1.24, +1.77, and +1.37 percentage points, respectively, and reduces execution cost in every comparison. The mean accuracy gain is 1.46 ± 0.28 percentage points. Moreover, increasing the counterfactual loss weight decreases accuracy and increases worker token usage, whereas increasing the token reward coefficient reduces worker token usage. The two adjustments therefore affect cost in opposite directions.

W 3: MAS-GPT trains a model on question–multi-agent-system pairs to generate executable MAS code. MasRouter uses cascaded controllers to sequentially determine the collaboration mode, roles, and model routing. COLLATOR changes only one responsibility, capacity, or dependency field within the same specification and uses the reward difference between the two executions to provide a training signal for that field. The preference loss applies only to the tokens corresponding to the edited field value, allowing execution feedback to directly supervise specific orchestration decisions.

Q 1: All baselines in the original paper use Qwen2.5-14B-Instruct; COLLATOR uses a 7B orchestrator and a worker pool comprising 7B, 14B, and 32B models. The newly added Free baseline is prompted to answer the question without requiring either a final-answer-only response or step-by-step reasoning, and is evaluated at all three model sizes: 7B, 14B, and 32B. The table below compares COLLATOR with a fixed MAS using only 32B workers, MasRouter using the same worker pool, and the other listed methods.

Results are reported for MMLU, GSM8K, and AQuA. Each cell gives accuracy (%) / capacity-weighted worker tokens per question.

MethodMMLUGSM8KAQuA
Free 7B72.37 / 111.8991.58 / 155.3381.89 / 242.36
Free 14B80.93 / 296.4294.16 / 331.0281.10 / 561.90
Free 32B83.28 / 560.6595.83 / 633.6383.86 / 920.53
CoT 32B83.80 / 933.3595.53 / 755.3784.65 / 1,073.40
Fixed MAS with all 32B workers83.93 / 6,945.3494.16 / 5,785.6884.25 / 7,032.53
MasRouter with the same worker pool81.64 / 3,232.7593.63 / 4,953.7883.46 / 5,423.43
COLLATOR84.32 / 2,148.0094.32 / 3,362.0086.22 / 2,056.10

Q 2: The table below summarizes three inference runs of the same models under the same evaluation protocol, quantifying variability due to inference randomness. We compute the metrics from the number of correctly answered questions in each inference run and report means ± sample standard deviations. We report accuracy for MMLU and GSM8K, and pass@1 for HumanEval, all as percentages. Except for Free 7B, Free 32B, and COLLATOR, as indicated, all methods use Qwen2.5-14B-Instruct; COLLATOR uses a 7B orchestrator and a worker pool comprising 7B, 14B, and 32B models. Vanilla is instructed to output only the final answer, Free imposes no constraint on whether to include reasoning, CoT explicitly requests step-by-step reasoning, and SC-CoT aggregates multiple sampled reasoning traces.

MethodMMLUGSM8KHumanEval
Vanilla76.96 ± 2.1786.32 ± 1.7580.91 ± 0.47
Free 7B72.13 ± 0.2191.28 ± 0.2780.11 ± 0.93
Free 14B80.86 ± 0.0793.96 ± 0.2381.45 ± 0.81
Free 32B83.10 ± 0.1695.17 ± 0.5983.60 ± 0.47
CoT75.59 ± 7.8893.71 ± 2.0880.65 ± 0.81
SC-CoT76.92 ± 8.1394.95 ± 1.4784.14 ± 0.47
Chain-MAS79.19 ± 0.6393.93 ± 1.5580.11 ± 0.93
Tree-MAS79.77 ± 0.2394.04 ± 1.3683.33 ± 0.93
Complete-MAS79.97 ± 0.8294.04 ± 1.2281.45 ± 0.81
Random-MAS80.10 ± 0.4293.78 ± 1.4283.33 ± 0.47
AFlow74.96 ± 4.0093.24 ± 1.6476.08 ± 1.68
AgentDropout80.43 ± 0.5994.14 ± 1.1978.23 ± 1.40
AgentPrune80.19 ± 0.6693.56 ± 1.9180.11 ± 0.47
G-Designer79.43 ± 1.0793.71 ± 1.3572.31 ± 0.93
ARG-Designer81.32 ± 3.4593.51 ± 0.3175.81 ± 1.40
OFA-MAS79.40 ± 0.9090.50 ± 4.9480.11 ± 0.47
COLLATOR84.59 ± 0.2694.36 ± 0.2385.48 ± 0.81

MMLU counterfactual-term ablation: results from three independent training runs. The table below compares GRPO alone with COLLATOR, which adds local counterfactual training. It summarizes evaluation results for models obtained from three independent training runs and reports means ± sample standard deviations. This table measures variability due to training randomness, whereas the preceding table measures variability across three inference runs of the same models. The two tables therefore summarize different sources of experimental variability.

MethodMMLU Accuracy (%)Weighted worker tokens per question
GRPO82.90 ± 0.172,548.39 ± 20.17
COLLATOR84.36 ± 0.272,142.09 ± 18.34

Across the three paired comparisons, the mean accuracy gain is 1.46 ± 0.28 percentage points, and the mean reduction in weighted worker tokens per question is 406.30 ± 38.49, corresponding to a mean cost reduction of 15.94%.

Q 3: The two tables below report the results of increasing the token reward coefficient and the win rate of the original specification among retained pairs, respectively. The first comparison evaluates GRPO-only on MMLU with λtok\lambda_{\mathrm{tok}} = 1.5, 2.0, and 3.0, holding all other training and evaluation settings fixed.

DatasetGRPO-only λtok\lambda_{\mathrm{tok}}GRPO-only Accuracy (%)GRPO-only Weighted worker tokens per questionCOLLATOR accuracy / Weighted worker tokens
MMLU1.583.082,543.2184.32 / 2,148.00
MMLU2.082.292462.3284.32 / 2,148.00
MMLU3.082.232434.4684.32 / 2,148.00

Among these three configurations, λtok\lambda_{\mathrm{tok}} = 3.0 yields worker token usage closest to that of COLLATOR, but it remains 13.34% higher, while accuracy is 2.09 percentage points lower.

The following table reports the win rate of the original specification among retained pairs.

Edit typeOriginal specification win rate (%)
Dependency removal64.24
Responsibility fallback75.17
Capacity reduction69.36

Q 4: A separate orchestrator is trained for each of the six benchmarks. The MMLU training examples are sampled from the official auxiliary training split, auxiliary_train.

Q 5: The table below reports the orchestrator's input and output token lengths, mean graph depth, and mean latency. End-to-end latency is defined as the total time from receiving a question to completing its execution.

DatasetOrchestrator input tokensMean orchestrator output tokensMean graph depthMean latency (s)
MMLU1,199.3253.73.954.21
GSM8K1,152.3280.34.646.43
AQuA1,173.7267.53.643.54
MultiArith1,130.3283.43.847.32
SVAMP1,130.4274.43.141.68
HumanEval1,219.2286.74.146.79

Thank you again for your feedback to our paper. We hope the additional experiments in our rebuttal help address the points you raised. If anything would benefit from further clarification, we’d be happy to elaborate.

Reviewer_ZJk3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Collator trains an LLM orchestrator to generate executable multi-agent specifications that jointly define task-specific duties, model capacities, and dependencies. Training combines supervised initialization, orchestration-level GRPO, and localized counterfactual supervision. The latter edits one specification field, compares execution rewards, and applies a preference loss to the edited spans. Experiments on six reasoning and coding benchmarks report improvements in task performance and worker-token efficiency.

Critical strengths

Coherent design and encouraging empirical results. Collator provides a clear, executable specification of task-specific agent duties, model capacities, and dependencies. Results across six benchmarks and component ablations provide encouraging evidence of improved task performance and worker-token efficiency.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with close baselines. Jointly generating subtask instructions, worker assignments, and context dependencies has a close precedent in Conductor. The methodological advance beyond this formulation is not sufficiently established, and a direct comparison with Conductor is missing despite its inclusion in the related work.

  2. Insufficient isolation of localized counterfactual training. Counterfactual training uses approximately 21.6% more recorded training tokens than GRPO alone, but a budget-matched comparison is not reported. Moreover, comparing edited-span supervision with whole-specification preference learning on the same counterfactual pairs would help establish whether localization itself contributes beyond the additional execution feedback.

Questions for authors
  1. Comparison with Conductor. Could the authors clarify the methodological distinction from Conductor and provide a comparison using the same worker pool and a matched training budget?

  2. Isolation of the counterfactual contribution. Could the authors separate the effects of additional execution feedback and span localization using a budget-matched GRPO baseline and whole-specification preference learning on the same counterfactual pairs?

3 comments

W 1: Relative to Conductor, COLLATOR's key methodological advance is to use workflow execution feedback for counterfactual credit assignment to specific orchestration decisions. Conductor jointly generates subtasks, worker assignments, and context-access dependencies, and uses GRPO to train the entire output with a workflow-level advantage. This feedback evaluates the orchestration as a whole, but a single successful execution cannot establish whether a particular large-model worker is necessary, whether a dependency is useful, or whether a responsibility can be simplified.

COLLATOR addresses these questions through controlled execution comparisons: it edits only one responsibility, capacity, or dependency field at a time while holding the rest of the specification fixed, re-executes the affected portion, and converts the reward difference into preference supervision for that field. The local loss applies only to the tokens corresponding to the edited field value, with all other fields serving as conditioning context. For example, if reducing a node's capacity from 32B to 14B preserves correctness and increases the total reward, the local supervision favors the smaller capacity for that node; if the total reward decreases, it favors the original configuration. Training can thus directly assess whether a particular resource allocation is worthwhile within the same orchestration and use the comparison to supervise the corresponding orchestration decision.

Adding local counterfactual training within the same orchestration framework improves accuracy on MMLU, GSM8K, and AQuA by 1.24, 0.92, and 0.79 percentage points, respectively, while reducing weighted worker tokens by 15.5%, 10.2%, and 31.7%. Detailed results are provided in the GRPO–COLLATOR comparison in Q 2.

W 2: We have added two sets of controlled comparisons examining the training budget and the scope of the loss, respectively. Extending GRPO training to bring its training token count close to that of COLLATOR yields accuracies of 83.27%, 93.25%, and 84.65% on MMLU, GSM8K, and AQuA, respectively, which remain below COLLATOR's 84.32%, 94.32%, and 86.22%. On the same set of counterfactual pairs, replacing the preference loss over the full specification with a preference loss over the edited field improves accuracy by 0.85, 0.85, and 1.18 percentage points on the three tasks, respectively. These two comparisons examine the effects of additional training budget and localized supervision, respectively. Detailed results are provided in Q 2.

Q 1: The table below directly compares our reproduction of Conductor with COLLATOR using the same worker pool and reports the actual training budgets. By adding field-level execution comparisons to workflow-level GRPO, COLLATOR achieves higher accuracy and lower worker token usage on all three tasks. Conductor's logged training budget is higher than that of COLLATOR.

DatasetMethodCumulative logged weighted training tokensAccuracy (%)Weighted worker tokens per question
MMLUConductor24.53M79.294,321.64
MMLUCOLLATOR8.90M84.322,148.00
GSM8KConductor24.53M90.295,462.27
GSM8KCOLLATOR10.44M94.323,362.00
AQuAConductor24.53M83.464,064.32
AQuACOLLATOR8.73M86.222,056.10

Q 2: The two tables below report the budget-matched GRPO comparison and the loss comparison on the same set of counterfactual pairs, respectively. In the first comparison, GRPO training is extended to bring the cumulative weighted training token count close to that of COLLATOR. In the second, we hold the sample pairs, initialization, reward labels, and length normalization fixed and vary only the scope of the preference loss, comparing its application to the full specification with its application to the edited field alone.

DatasetMethodUpdate stepsCumulative logged weighted training tokensAccuracy (%)Weighted worker tokens per question
MMLUGRPO1,5007.33M83.082,543.21
MMLUGRPO (budget-matched)19208.89M83.272643.46
MMLUCOLLATOR1,5008.90M84.322,148.00
GSM8KGRPO1,5008.82M93.403,742.59
GSM8KGRPO (budget-matched)199010.45M93.253722.87
GSM8KCOLLATOR1,50010.44M94.323,362.00
AQuAGRPO1,5007.12M85.433,012.54
AQuAGRPO (budget-matched)19508.73M84.652964.32
AQuACOLLATOR1,5008.73M86.222,056.10

Comparison of preference losses on the same set of pairs. Each cell reports accuracy (%) followed by weighted worker tokens per question.

DatasetFull-specification preference lossEdited-field preference loss
MMLU83.47 / 2469.7084.32 / 2,148.00
GSM8K93.47 / 3689.4694.32 / 3,362.00
AQuA85.04 / 2769.3586.22 / 2,056.10

Thank you again for your feedback to our paper. We hope the additional experiments in our rebuttal help address the points you raised. If anything would benefit from further clarification, we’d be happy to elaborate.

Reviewer_CrqgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses orchestration of LLM multi-agent systems, where role design, capacity assignment, and dependency construction are decided jointly rather than in isolation. The paper proposes COLLATOR, in which an LLM orchestrator generates a specification over agent duties, capacity levels, and dependency references. Training combines orchestration-level GRPO with a localized counterfactual term, where one field is edited, the edited specification is re-executed, and the reward contrast is applied only to the edited span. On six reasoning and coding benchmarks, COLLATOR reaches 90.72 average accuracy versus at most 88.53 for baselines.

Critical strengths
  1. The formulation cleanly unifies three previously decoupled design decisions, including roles, capacities, and dependencies of multi-agent workflow. Case studies (fig. 4) show task-specific duties rather than generic personas.
  2. Counterfactual execution is well motivated and affordable. The one-field edit gives a clean attribution target, and node-level caching restricts re-execution to the edited node and its descendants.
  3. The work is reproducible, with prompts, schema, code, etc. very well documented.
Critical weaknesses
  1. The baseline backbone is unclear. The Vanilla row seems to be Qwen2.5-7B-Instruct results. Since COLLATOR can route to 14B and 32B workers, a single 7B baseline would be a weaker comparison, and the gain may be attributed to the stronger worker. A per-baseline backbone table would make the comparison more convincing.
  2. The counterfactual term, though well defined and motivated, contributes a small margin. GRPO alone contributes most of the gains, while the counterfactual term adds marginal accuracy improvements. An analysis of the accuracy effect of the counterfactual term would improve the paper.
  3. There seems to be no out-of-distribution task evaluation. The motivating claim that fixed designs do not transfer across tasks (sec. 1) is therefore not sufficiently tested. An evaluation on held out task types would better support the task-adaptivity claim.
Questions for authors
  1. Which backbone do the baselines in table 1 use?
  2. Are the accuracy gains from the counterfactual term in table 2 consistent across seeds
  3. How does the orchestrator perform on held-out tasks?
5 comments

W 1: Vanilla, CoT, SC-CoT, and the other multi-agent baselines in the original paper all use Qwen2.5-14B-Instruct. COLLATOR uses a 7B orchestrator that assigns 7B, 14B, or 32B workers to individual nodes. The additional single-model baselines evaluate each of these three model sizes.

The table below summarizes the results for Vanilla and the newly added Free baseline across model sizes. Vanilla is instructed to output only the final answer; Free is prompted to answer the question without requiring either a final-answer-only response or step-by-step reasoning. We report accuracy for MMLU and the other non-code benchmarks, and pass@1 for HumanEval, all as percentages.

MethodModel sizeMMLUGSM8KAQuAMultiArithSVAMPHumanEval
Vanilla7B70.6176.6439.7653.6766.3079.03
Vanilla14B74.4685.5946.4676.0080.8080.65
Vanilla32B81.9190.2155.9195.5085.8082.25
Free7B72.3791.5881.8998.8393.8080.65
Free14B80.9394.1681.1098.8394.4081.45
Free32B83.2895.8383.8698.8393.9083.87
COLLATOR7B/14B/32B84.3294.3286.2299.1795.6084.68

W 2: The table below compares accuracy and execution cost with and without the local counterfactual term. Relative to SFT + GRPO alone, adding this term improves accuracy by 0.79–1.24 percentage points across the three tasks and reduces weighted worker tokens by 10.2%–31.7%. The local counterfactual term therefore benefits both accuracy and efficiency.

DatasetAccuracy (%), GRPO → Full methodWeighted worker tokens per question, GRPO → Full method
MMLU83.08 → 84.322,543.21 → 2,148.00 (−15.5%)
GSM8K93.40 → 94.323,742.59 → 3,362.00 (−10.2%)
AQuA85.43 → 86.223,012.54 → 2,056.10 (−31.7%)

W 3: We have added a cross-task evaluation transferring from MMLU to HotpotQA. With the MMLU orchestrator frozen and no HotpotQA training data used, COLLATOR achieves 61.36% EM and 77.04% F1 on all 7,405 questions in the distractor development set. Detailed results are provided in Q 3.

Q 1: In Table 1, Vanilla, CoT, SC-CoT, and the other multi-agent baselines all use Qwen2.5-14B-Instruct for task execution. COLLATOR uses a Qwen2.5-7B-Instruct orchestrator and assigns workers to individual nodes from a pool comprising Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Qwen2.5-32B-Instruct.

Q 2: The table below reports evaluation results for models obtained from three independent training runs on MMLU. Values are means ± sample standard deviations, quantifying variability due to training randomness. In all three paired training comparisons, adding the counterfactual term improves accuracy and reduces execution cost.

MethodMMLU Accuracy (%)Weighted worker tokens per question
GRPO82.90 ± 0.172,548.39 ± 20.17
COLLATOR84.36 ± 0.272,142.09 ± 18.34

Across the three paired comparisons, the mean accuracy gain is 1.46 ± 0.28 percentage points, and the mean reduction in weighted worker tokens per question is 406.30 ± 38.49.

Q 3: The table below presents the complete comparison after freezing the MMLU orchestrator and transferring it to HotpotQA, where COLLATOR achieves 61.36% EM and 77.04% F1. The evaluation covers all 7,405 questions in the HotpotQA distractor development set, with every method receiving the same 10 candidate passages for each question. All comparison baselines use Qwen2.5-14B-Instruct.

MethodAnswer EM (%)Answer F1 (%)Weighted worker tokens per question
Vanilla58.8373.311,460.56
Complete-MAS57.6874.119,021.51
Random-MAS58.8975.098,912.99
AgentPrune58.8874.999,075.38
AgentDropout V155.1472.1116,811.00
ARG-Designer57.1473.659,248.63
G-Designer55.1472.1413,603.62
OFA-MAS54.9071.3410,262.79
AFlow57.1171.388643.35
COLLATOR61.3677.047943.37

The frozen COLLATOR orchestrator achieves 61.36% answer EM and 77.04% answer F1, exceeding all baselines in the table on both metrics while using fewer weighted worker tokens per question than the other multi-agent methods. This result demonstrates that the orchestration policy learned on MMLU transfers to the HotpotQA multi-hop question-answering task.

Thank you again for your feedback to our paper. We hope the additional experiments in our rebuttal help address the points you raised. If anything would benefit from further clarification, we’d be happy to elaborate.

Comments on the paper

3 comments
~Reza_Barati2OpenReview ↗

Does one-sided counterfactual sampling limit credit for beneficial expansions?

Thank you for the interesting formulation of localized credit assignment. I work on sequential decision-making and learned orchestration, and I have a question about the directionality of the counterfactual edits.

In Section 3.3, all three mutation families are subtractive: dependency deletion, role rollback, and capacity downgrade. These comparisons can establish whether an existing field earns its place relative to a simpler alternative. For example, if downgrading capacity hurts performance, the reward contrast reinforces the larger capacity. However, when a sampled specification is already at the lowest capacity or lacks a potentially useful dependency, the localized objective cannot directly evaluate an expansion from that specification. Appendix E establishes a local improvement in the preference margin for a sampled pair, but does not establish whether the counterfactual sampling distribution sufficiently covers beneficial alternatives.

Orchestration-level GRPO can explore richer specifications, but assigns the same advantage to all tokens within a specification. Does relying exclusively on subtractive edits for localized supervision create an asymmetry in discovering beneficial expansions, particularly when the initial policy under-allocates resources?

A related consideration is the interaction with cost. Additional dependencies increase structural cost (Eq. 8), while higher model capacities increase the per-token cost weighting (Eq. 6). Consequently, when resource savings dominate the reward contrast, subtractive comparisons may systematically favour simpler field values. Could this interact with the directionality of localized supervision?

Would a compute-matched comparison incorporating feasible capacity-upgrade and dependency-addition families be informative? Such an experiment could help determine whether one-sided counterfactual sampling is sufficient or whether bidirectional edits improve the learned accuracy–cost trade-off.

Q 1: Using only subtractive edits may miss beneficial capacity upgrades or dependency additions. We adopt subtractive edits because they directly test the necessity of existing capacity and dependencies in a generated specification. Introducing additive edits expands the candidate space: adding a dependency requires selecting a previously unconnected node pair while satisfying the acyclicity constraint, and upgrading capacity increases the cost of counterfactual re-execution. During our exploration, we also tested a setting that included additive edits. Its performance was similar to that of subtractive edits alone, with no clear additional benefit observed; we therefore adopted subtractive edits in the final method.

Q 2: The cost reward affects the direction of preference: when cost savings are sufficient to offset changes in the task reward and structural cost, a lower-cost specification may receive a higher reward. When a specification is valid but its answer is incorrect, the reward function discounts the token reward, so the direction of preference also depends on whether the edit preserves answer correctness. If reducing capacity or removing a dependency decreases the total reward, the local loss reinforces the original capacity or dependency. Under the default counterfactual setting, accuracy on MMLU, GSM8K, and AQuA improves by 1.24, 0.92, and 0.79 percentage points, respectively, while worker tokens decrease by 15.5%, 10.2%, and 31.7%.

Q 3: While developing this work, we tested a setting that added capacity upgrades and dependency additions to the subtractive edits. The available results indicate that including additive edits performs similarly to using subtractive edits alone, with no additional benefit observed. We therefore retain the subtractive-edit scheme, which has a smaller candidate space.

We thank the AC and reviewers for their time and constructive feedback. During the discussion period, we added new experimental evidence to address the four main concerns raised:

  • Fair comparisons: We evaluated single-model baselines across different scales and adapted six multi-agent baselines to use the exact same worker pool. Under this unified setting, COLLATOR achieves the best performance on GSM8K and HotpotQA while consuming the fewest weighted worker tokens across all three benchmarks, whereas ARG-Designer attains the highest accuracy on MMLU.
  • Methodological contribution: We included a local reproduction of Conductor, a training-budget-matched comparison, and an ablation of preference losses on identical counterfactual pairs. Together, these results verify the independent contribution of our localized credit assignment.
  • Robustness and transferability: Across multiple training seeds, the counterfactual term consistently improves accuracy while reducing cost. Furthermore, the frozen orchestrator trained on MMLU transfers effectively to HotpotQA.
  • Experimental clarity: We clarified the training data and model configurations, and additionally reported both orchestrator overhead and end-to-end latency. We hope these additional results provide a clear basis for the AC to assess our paper's contributions.

We hope these additional results provide a clear basis for the AC to assess our paper's contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.