- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CollabOrchestra, a framework for training language-model orchestrators to coordinate human-agent teams at the workflow level. The orchestrator constructs and revises structured plans specifying subtask goals, worker assignments, and so on. The method consists of two stages. First, the authors profile agent workers through sandboxed execution and train a lightweight feasibility predictor to estimate whether a particular worker can complete a proposed subtask. Second, they train the orchestrator using multi-turn GRPO with simulated users, optimizing a combination of LLM-judged plan quality and predicted execution feasibility.
Critical strengths
- I think the problem formulation is useful. The paper treats human–agent delegation at the workflow level, with the human as an assignable worker rather than a passive recipient. This is a practical abstraction for long-horizon occupational tasks.
- The training recipe is practical. They use an offline, execution-grounded feasibility predictor to replace costly rollouts during RL, and the paper checks that planning gains carry over to execution.
Critical weaknesses
- The bottleneck feasibility in Table 1 is evaluated using the same learned predictor that supplies the training reward. I don't think improvements in this metric are independent evidence of improved execution feasibility. 2.The “team” includes an extra gemini-3-flash agent playing the human. The team therefore gets more compute and more review passes than a single solo agent. I don't think the gain can be credited to orchestration. Although the paper provides ablations analyzing human-assigned roles, it does not sufficiently isolate the contribution of each reward component to downstream delivery rate and task performance.
Questions for authors
- Does the predicted bottleneck feasibility correlate with observed execution success on held-out workflow steps or complete workflows?
- Could the authors report delivery rate and task performance for the ablations without the feasibility reward and without the user-involvement component?
8 comments
The “team” includes an extra gemini-3-flash agent playing the human. The team therefore gets more compute and more review passes than a single solo agent. I don't think the gain can be credited to orchestration. Although the paper provides ablations analyzing human-assigned roles, it does not sufficiently isolate the contribution of each reward component to downstream delivery rate and task performance.
We added a multi-agent baseline and showed that multi-agent systems without hidden task description, cannot achieve desired task performance as if informed users are involved.
Experiment protocol. We implement the multi-agent baseline by using the same workflow plan but giving the "user" steps to an additional GeminiCLI agent. The additional agent receives the original task prompt, the full workflow plan, the current step's goal and deliverables, and the outcomes of previous steps. It does not receive the user profile or the user-only information for underspecified tasks. Each step has a 15-minute timeout instead of 3 minutes, so the baseline gets at least as much compute as the user simulator.
Results. Execution results of the simulated human-agent team and the multi-agent baseline using the same workflow plans from CollabOrchestra. -values are from permutation tests on Task Performance. Each is shown on the row of the higher-scoring setting. n.s. = not significant.
| Orchestrator | Method | GDPval Delivery Rate | GDPval Task Perf. | GDPval-underspecified Delivery Rate | GDPval-underspecified Task Perf. |
|---|---|---|---|---|---|
| Qwen3-4B CollabOrchestra | Simulated Human-Agent Collaboration | 0.970 | 0.741 | 0.952 | 0.645 (p <1e-4) |
| Qwen3-4B CollabOrchestra | Multi-Agent Baseline | 0.964 | 0.742 (n.s.) | 0.980 | 0.472 |
| Qwen3-30B-A3B CollabOrchestra | Simulated Human-Agent Collaboration | 0.957 | 0.597 | 0.952 | 0.474 |
| Qwen3-30B-A3B CollabOrchestra | Multi-Agent Baseline | 1.00 | 0.786 (p = 0.0003) | 0.985 | 0.481 (n.s.) |
| gpt-oss-120b CollabOrchestra | Simulated Human-Agent Collaboration | 0.926 | 0.744 | 0.930 | 0.703 (p <1e-4) |
| gpt-oss-120b CollabOrchestra | Multi-Agent Baseline | 0.974 | 0.753 (n.s.) | 0.985 | 0.495 |
When tasks are fully specified, the multi-agent baseline is about as good or slightly better, largely by delivering more often. For underspecified tasks, however, replacing the informed user with an additional agent substantially reduces task performance for Qwen3-4B (0.645 vs. 0.472, < 1e-4) and gpt-oss-120b (0.703 vs. 0.495, < 1e-4), despite the additional execution budget.
These findings qualify our claims and address the compute confound: additional agent compute can reproduce the benefits of human-agent collaboration when tasks are fully specified, but it cannot reliably substitute for user-specific knowledge and preferences that are absent from the initial task description. Since real-world tasks often depend on such information, effective orchestration must go beyond allocating more agent compute to meaningfully involve informed users when their contributions are necessary.
We execute the workflow plans from two ablations from Section 5.2:
- “w/o Feasibility Reward”, which removes the feasibility term from the training reward;
- “w/o User Involv. Rubric”, which removes the Use Involvement Appropriateness component for the Plan Quality reward
The results are as follows:
| Orchestrator Base Model | Method | GDPval Delivery Rate | GDPval Task Perf. | GDPval-underspecified Delivery Rate | GDPval-underspecified Task Perf. |
|---|---|---|---|---|---|
| gpt-oss-120b | CollabOrchestra | 0.926 | 0.744 | 0.930 | 0.703 |
| gpt-oss-120b | CollabOrchestra w/o feasibility reward | 0.694 | 0.556 | 0.730 | 0.507 |
| gpt-oss-120b | CollabOrchestra w/o user-involvement reward | 0.959 | 0.742 | 0.923 | 0.651 |
Feasibility reward. Removing the feasibility term causes a large drop in Delivery Rate on both benchmarks (GDPval: 0.926 → 0.694; GDPval-underspecified: 0.930 → 0.730). Task Performance drops with it (0.744 → 0.556; 0.703 → 0.507). This confirms that the feasibility reward is critical for producing plans that can actually be executed.
User-involvement rubric. Removing the User Involvement Appropriateness effects Task Performance on GDPval-underspecified where the simulated user holds information the orchestrator needs. In this setup performance drops from 0.703 to 0.651, because without the rubric the orchestrator is less likely to involve the user at the points where that information is needed.
We thank the reviewer for the constructive feedback. We are glad the reviewer finds that the problem formulation is useful and provides a practical abstraction for long-horizon occupational tasks. We also appreciate the reviewer's view that the training recipe is practical: the offline, execution-grounded feasibility predictor replaces costly rollouts during RL, and the paper checks that planning gains carry over to execution. We address each concern below.
We check the correlation between Bottleneck Feasibility and delivery status (0 or 1) for every executed plan, pairing each orchestrator plan with its end-to-end execution. The point-biserial correlation r reveals that a delivered plan is scored significantly higher than an undelivered one.
We agree that the Bottleneck Feasibility numbers in Table 1 are not independent evidence of execution feasibility, since the policy was optimized against the same predictor. We use Table 1's Bottleneck Feasibility as a diagnostic of what the policy learned. The independent evidence comes from two other sources: