- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
COLLATOR trains a Qwen2.5-7B orchestrator to generate one YAML spec per query. The spec sets each agent’s duty, its worker size (7B/14B/32B), and its upstream dependencies, and it is validated, compiled to a DAG and executed. Training is SFT, then GRPO on a reward combining (1) correctness, (2) a token saving bonus and (3) a structural penalty. It adds a “localized counterfactual” loss: mutate one field (delete an edge, revert a duty to generic, or downgrade capacity), re-execute, and apply a DPO-style preference on that field’s tokens only, according to the sign of the reward difference. It reports the best average across six benchmarks and a better accuracy–token trade-off.
Critical strengths
- The method is intuitive, and MAS orchestration is an important problem. Executable, field-structured spec makes field-level credit cheap. The design lets the authors mutate a single field, re-run only the affected downstream nodes via caching, and attach the reward contrast to exactly those tokens. This is a practical, reusable recipe.
- The technique also aims at efficiency, not just accuracy. The paper reports worker tokens alongside accuracy in every ablation. It also discloses the training overhead of the counterfactual loss (+21.6% training tokens). This makes the efficiency claims checkable.
Critical weaknesses
- Gains are not clearly attributed and baseline is unclear. In particular, orchestration gains are not separated from model size. COLLATOR can route work to a 32B worker. The Vanilla row (MMLU 74.46, GSM8K 85.59) matches published Qwen2.5-7B scores, which suggests the baselines run on 7B. Table 1 cannot show that orchestration design, rather than bigger models, drives the gains. Can the authors clarify baseline and settings?
- The gains are small, and the counterfactual loss may just reward lower cost. The main results (Table 1) come from a single run with no error bars. If decoding is non-greedy, margins of 1–4 test items on HumanEval, MultiArith and SVAMP could be noise. Doubling the counterfactual loss weight (0.05 to 0.10) also drops accuracy by 6–10 points. A control, such as simply raising the token reward, is needed to show it does more than cutting tokens.
- Novelty is also somewhat limited: MAS-GPT (Ye et al., ICML 2025) already generates a full query-specific MAS in one pass, and MasRouter (Yue et al., ACL 2025) chooses roles and models together.
Questions for authors
- Which worker model(s) does each baseline in Table 1 use? Adding the following baselines would make the work much more solid:14B and 32B workers, a fixed MAS with all-32B workers, and MasRouter baselines under the same worker pool.
- Please report mean std over at least 3 seeds for Table 1 and Table 2 (if nongreedy sampling + exact match / pass@1 is used).
- How does the counterfactual loss compare with simply raising _tok at matched token usage? What fraction of retained counterfactual pairs prefer the factual spec?
- Some experiment settings are unclear. Where do the 480 MMLU RL examples and the SFT questions come from? Does cross_dataset_v1 train one orchestrator for all six tasks or one per task?
- counts worker tokens only. What is the orchestrator’s own generation cost (up to 2048 tokens) and the end-to-end latency given graph depth?
4 comments
Q 1: All baselines in the original paper use Qwen2.5-14B-Instruct; COLLATOR uses a 7B orchestrator and a worker pool comprising 7B, 14B, and 32B models. The newly added Free baseline is prompted to answer the question without requiring either a final-answer-only response or step-by-step reasoning, and is evaluated at all three model sizes: 7B, 14B, and 32B. The table below compares COLLATOR with a fixed MAS using only 32B workers, MasRouter using the same worker pool, and the other listed methods.
Results are reported for MMLU, GSM8K, and AQuA. Each cell gives accuracy (%) / capacity-weighted worker tokens per question.
| Method | MMLU | GSM8K | AQuA |
|---|---|---|---|
| Free 7B | 72.37 / 111.89 | 91.58 / 155.33 | 81.89 / 242.36 |
| Free 14B | 80.93 / 296.42 | 94.16 / 331.02 | 81.10 / 561.90 |
| Free 32B | 83.28 / 560.65 | 95.83 / 633.63 | 83.86 / 920.53 |
| CoT 32B | 83.80 / 933.35 | 95.53 / 755.37 | 84.65 / 1,073.40 |
| Fixed MAS with all 32B workers | 83.93 / 6,945.34 | 94.16 / 5,785.68 | 84.25 / 7,032.53 |
| MasRouter with the same worker pool | 81.64 / 3,232.75 | 93.63 / 4,953.78 | 83.46 / 5,423.43 |
| COLLATOR | 84.32 / 2,148.00 | 94.32 / 3,362.00 | 86.22 / 2,056.10 |
Q 2: The table below summarizes three inference runs of the same models under the same evaluation protocol, quantifying variability due to inference randomness. We compute the metrics from the number of correctly answered questions in each inference run and report means ± sample standard deviations. We report accuracy for MMLU and GSM8K, and pass@1 for HumanEval, all as percentages. Except for Free 7B, Free 32B, and COLLATOR, as indicated, all methods use Qwen2.5-14B-Instruct; COLLATOR uses a 7B orchestrator and a worker pool comprising 7B, 14B, and 32B models. Vanilla is instructed to output only the final answer, Free imposes no constraint on whether to include reasoning, CoT explicitly requests step-by-step reasoning, and SC-CoT aggregates multiple sampled reasoning traces.
| Method | MMLU | GSM8K | HumanEval |
|---|---|---|---|
| Vanilla | 76.96 ± 2.17 | 86.32 ± 1.75 | 80.91 ± 0.47 |
| Free 7B | 72.13 ± 0.21 | 91.28 ± 0.27 | 80.11 ± 0.93 |
| Free 14B | 80.86 ± 0.07 | 93.96 ± 0.23 | 81.45 ± 0.81 |
| Free 32B | 83.10 ± 0.16 | 95.17 ± 0.59 | 83.60 ± 0.47 |
| CoT | 75.59 ± 7.88 | 93.71 ± 2.08 | 80.65 ± 0.81 |
| SC-CoT | 76.92 ± 8.13 | 94.95 ± 1.47 | 84.14 ± 0.47 |
| Chain-MAS | 79.19 ± 0.63 | 93.93 ± 1.55 | 80.11 ± 0.93 |
| Tree-MAS | 79.77 ± 0.23 | 94.04 ± 1.36 | 83.33 ± 0.93 |
| Complete-MAS | 79.97 ± 0.82 | 94.04 ± 1.22 | 81.45 ± 0.81 |
| Random-MAS | 80.10 ± 0.42 | 93.78 ± 1.42 | 83.33 ± 0.47 |
| AFlow | 74.96 ± 4.00 | 93.24 ± 1.64 | 76.08 ± 1.68 |
| AgentDropout | 80.43 ± 0.59 | 94.14 ± 1.19 | 78.23 ± 1.40 |
| AgentPrune | 80.19 ± 0.66 | 93.56 ± 1.91 | 80.11 ± 0.47 |
| G-Designer | 79.43 ± 1.07 | 93.71 ± 1.35 | 72.31 ± 0.93 |
| ARG-Designer | 81.32 ± 3.45 | 93.51 ± 0.31 | 75.81 ± 1.40 |
| OFA-MAS | 79.40 ± 0.90 | 90.50 ± 4.94 | 80.11 ± 0.47 |
| COLLATOR | 84.59 ± 0.26 | 94.36 ± 0.23 | 85.48 ± 0.81 |
MMLU counterfactual-term ablation: results from three independent training runs. The table below compares GRPO alone with COLLATOR, which adds local counterfactual training. It summarizes evaluation results for models obtained from three independent training runs and reports means ± sample standard deviations. This table measures variability due to training randomness, whereas the preceding table measures variability across three inference runs of the same models. The two tables therefore summarize different sources of experimental variability.
| Method | MMLU Accuracy (%) | Weighted worker tokens per question |
|---|---|---|
| GRPO | 82.90 ± 0.17 | 2,548.39 ± 20.17 |
| COLLATOR | 84.36 ± 0.27 | 2,142.09 ± 18.34 |
Across the three paired comparisons, the mean accuracy gain is 1.46 ± 0.28 percentage points, and the mean reduction in weighted worker tokens per question is 406.30 ± 38.49, corresponding to a mean cost reduction of 15.94%.
Q 3: The two tables below report the results of increasing the token reward coefficient and the win rate of the original specification among retained pairs, respectively. The first comparison evaluates GRPO-only on MMLU with = 1.5, 2.0, and 3.0, holding all other training and evaluation settings fixed.
| Dataset | GRPO-only | GRPO-only Accuracy (%) | GRPO-only Weighted worker tokens per question | COLLATOR accuracy / Weighted worker tokens |
|---|---|---|---|---|
| MMLU | 1.5 | 83.08 | 2,543.21 | 84.32 / 2,148.00 |
| MMLU | 2.0 | 82.29 | 2462.32 | 84.32 / 2,148.00 |
| MMLU | 3.0 | 82.23 | 2434.46 | 84.32 / 2,148.00 |
Among these three configurations, = 3.0 yields worker token usage closest to that of COLLATOR, but it remains 13.34% higher, while accuracy is 2.09 percentage points lower.
The following table reports the win rate of the original specification among retained pairs.
| Edit type | Original specification win rate (%) |
|---|---|
| Dependency removal | 64.24 |
| Responsibility fallback | 75.17 |
| Capacity reduction | 69.36 |
Q 4: A separate orchestrator is trained for each of the six benchmarks. The MMLU training examples are sampled from the official auxiliary training split, auxiliary_train.
Q 5: The table below reports the orchestrator's input and output token lengths, mean graph depth, and mean latency. End-to-end latency is defined as the total time from receiving a question to completing its execution.
| Dataset | Orchestrator input tokens | Mean orchestrator output tokens | Mean graph depth | Mean latency (s) |
|---|---|---|---|---|
| MMLU | 1,199.3 | 253.7 | 3.9 | 54.21 |
| GSM8K | 1,152.3 | 280.3 | 4.6 | 46.43 |
| AQuA | 1,173.7 | 267.5 | 3.6 | 43.54 |
| MultiArith | 1,130.3 | 283.4 | 3.8 | 47.32 |
| SVAMP | 1,130.4 | 274.4 | 3.1 | 41.68 |
| HumanEval | 1,219.2 | 286.7 | 4.1 | 46.79 |
Thank you again for your feedback to our paper. We hope the additional experiments in our rebuttal help address the points you raised. If anything would benefit from further clarification, we’d be happy to elaborate.
W 1: Vanilla, CoT, SC-CoT, and the other multi-agent baselines in the original paper all use Qwen2.5-14B-Instruct. COLLATOR uses a 7B orchestrator and 7B, 14B, and 32B workers. The table below summarizes the results for Vanilla and the newly added Free baseline across model sizes. Vanilla is instructed to output only the final answer; Free is prompted to answer the question without requiring either a final-answer-only response or step-by-step reasoning. We report accuracy for MMLU and the other non-code benchmarks, and pass@1 for HumanEval, all as percentages.
The following table further quantifies the contribution of capacity allocation by comparing the ablation with all workers fixed at 14B against dynamic capacity allocation. Both settings retain the task responsibilities and dependency design.
W 2: In the three additional paired training comparisons on MMLU, the counterfactual term yields accuracy gains of +1.24, +1.77, and +1.37 percentage points, respectively, and reduces execution cost in every comparison. The mean accuracy gain is 1.46 ± 0.28 percentage points. Moreover, increasing the counterfactual loss weight decreases accuracy and increases worker token usage, whereas increasing the token reward coefficient reduces worker token usage. The two adjustments therefore affect cost in opposite directions.
W 3: MAS-GPT trains a model on question–multi-agent-system pairs to generate executable MAS code. MasRouter uses cascaded controllers to sequentially determine the collaboration mode, roles, and model routing. COLLATOR changes only one responsibility, capacity, or dependency field within the same specification and uses the reward difference between the two executions to provide a training signal for that field. The preference loss applies only to the tokens corresponding to the edited field value, allowing execution feedback to directly supervise specific orchestration decisions.