- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PORCHESTRA lets a sub-agent pause mid-task, ask the orchestrator to change its goal, tools or budget, and carry on with its context intact. OPOD then uses these requests as hindsight hints to train a small orchestrator. Across GAIA, GAIA2 and a 100-issue SWE-bench subset, with three executor models, it baets the best baseline by 7 to 11 points on GAIA2, by less on GAIA, and by one issue on SWE-bench, while costing less.
Critical strengths
On GAIA2, the one benchmark where things change mid-task, the gain is large and consistent across all three executors. A same-model comparison shows it comes from the communication mechanism itself. It also cuts cost per task from 1.21 to 0.72 relative to AOrchestra.
Critical weaknesses
OPOD isn't established. Its edge over a simpler baseline is a few tasks, its settings were tuned on the test set, and it closely follows self-distillation work it barely cites. Outside GAIA2 the gains are within noise. Inside GAIA2, the prompt tells sub-agents to report every notification, which effectively acts as a trigger, and this is never controlled for.
Questions for authors
How many runs underlie each number? How did the baselines receive GAIA2 notifications, and what would a simple notification trigger score? For future work, OPOD should be compared with GRPO at matched compute.
We thank the authors for the extensive additional experiments; however, the novelty concern remains, since the harness revision is still delivered as an observation appended to the sub-agent's trajectory and is functionally close to mechanisms already available in mainstream harnesses; the contribution of OPOD's process supervision over its ablation is not shown to be significant; and most new comparisons (SWE-Bench, AutomationBench, DeepSWE) fall within run-to-run variance, with Claude Code missing, harness configurations unspecified, and no harness comparison on GAIA/GAIA2, so I maintain my original score.