- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BRIGADE maintains beliefs about intermediate output correctness and uses two-step lookahead to select execution, verification, or submission actions within a task DAG. Calibration data support action and observation models. Simulations examine the benefit of feedback and lookahead, while AppWorld experiments report favorable completion/cost trade-offs at lower budgets.
Critical strengths
- Extending Bayesian agent control to dependent subtasks is useful. Tracking output versions and assessing dependent work addresses a practical difficulty that a controller over a single candidate does not encounter.
- The simulations separate feedback-based updating from lookahead and include an informative failure case, where adaptive controllers fail at the smallest budget. This helps find out when additional planning is useful.
Critical weaknesses
- The AppWorld evidence does not isolate the controller's contribution. Appendix E reports one run on 57 development tasks, uncontrolled code revisions across batches, and costs excluding planning, calibration, and attempts outside terminal records. These limitations affect the completion/cost comparison.
- The benefit of two-step lookahead is demonstrated primarily in simulation. AppWorld lacks a corresponding one-step controller and an ablation of the additional repetition penalty. The initial correctness priors use only 29 labelled outputs, without held-out validation or sensitivity evidence. Consequently, the respective contributions of Bayesian updating, lookahead, and implementation heuristics remain uncertain.
Questions for authors
- Can you provide comparisons with more detailed cost accounting, separating reusable calibration costs from per-task costs?
- Under the same AppWorld implementation and repetition penalty, how does one-step planning compare with two-step planning? How sensitive are the results to the estimated calibration tables?
8 comments
Q2: one-step against two-step planning under the same implementation, and sensitivity to the calibration tables
Answer. (a) On AppWorld, two-step lookahead gives no measurable gain over the identical one-step controller; we will limit that claim to the simulation. (b) The results are insensitive to errors in verifier reliability and repair success and to over-estimated correctness priors, and sensitive to under-estimated priors when the budget is close to the minimum cost of the task.
Setup (a). BRIGADE-Q1 is BRIGADE with the lookahead depth set to one; everything else is identical. 50 held-out test tasks; three runs at K=16, one at K=8 and K=32. Differences are paired, with 95% bootstrap intervals over task families. The table gives TGC on AppWorld and the success rate in the simulation.
| Setting | Two-step (%) | One-step (%) | Two-step − one-step, points [95%] |
|---|---|---|---|
| AppWorld, K=8 | 38 | 44 | −6 [−20, +8] |
| AppWorld, K=16 | 51.3 | 52.0 | −0.7 [−8.7, +7.3] |
| AppWorld, K=32 | 48 | 50 | −2 [−16, +10] |
| AppWorld, averaged over budgets | −2.9 [−9.8, +3.8] | ||
| Simulation, pooled over budgets | 25.4 | 20.0 | +5.4 [+3.9, +6.9] |
Setup (b). We replay the 2,400 episodes of the paper's simulation. The environment keeps the true parameters, and the controller plans with one family of its estimates shifted by a constant bias δ. The table gives the change in the success rate of BRIGADE in points (unbiased: BRIGADE 25.4%, Fixed 16.9%).
| Biased estimate | δ = −0.15 | −0.05 | +0.05 | +0.15 |
|---|---|---|---|---|
| Correctness prior | −6.3 | −6.3 | +0.1 | −1.2 |
| Verifier reliability | −0.5 | −0.2 | +0.5 | +1.8 |
| Repair success | −2.6 | −0.8 | +0.2 | +0.5 |
Analysis.
- Lookahead depth. No difference is significant on AppWorld at any budget, and the second step needs 6 to 12 times more controller computation. In the simulation the gain is significant. A likely reason is self-correction: on AppWorld a repair after a FAIL verdict fixes an incorrect output about half of the time and breaks a correct one about a quarter of the time, while in the simulation repairs are far more reliable (61–95% and 7–11%). This suggests that the second step becomes worthwhile as models get better at self-correction (details in our response to Reviewer 8jdC, Q1).
- Robust to calibration error. Errors in verifier reliability and repair success, and optimistic priors, change the success rate by a few points at most.
- One sensitive case. With pessimistic priors the whole loss comes from the budget just above the minimum cost of the task, where the controller keeps verifying and repairing the first nodes and exhausts the budget.
Conclusion. Under one implementation, lookahead depth does not explain the AppWorld gains; they come from adaptive allocation from calibrated estimates. The calibration does not need to be exact, with one stated failure mode. The revision will report BRIGADE-Q1 next to BRIGADE, limit the two-step claim to the simulation, add the sensitivity experiment to the appendix, and state the failure mode in the Limitations.
We thank the reviewer again. If the cost accounting, the one-step ablation and the sensitivity analysis address your concerns, we would be grateful if you would consider raising your score.
Thank you for the additional experiments and the appropriately adjusted statements. The revised cost accounting and one-step comparison address these parts of my review. My remaining concern is the explanation of the AppWorld improvements. The new results do not report the frozen posterior baseline or an ablation of the repetition penalty under the same held-out setup. Also, the calibration sensitivity analysis is confined to simulation, so robustness of the sparse AppWorld calibration estimates remains uncertain. Could you clarify what evidence supports explaining the held-out improvements as a result of Bayesian adaptive allocation?
We thank the reviewer for the constructive review and for the two concrete requests. We ran both and answer them in two comments.
Q1: cost accounting that separates the reusable calibration cost from the per-task cost
Answer. We now report the per-task cost with every component, and the one-time calibration cost separately. Per task, BRIGADE completes more tasks than ReAct at a lower cost when the budget is restricted. The calibration is recovered after roughly one to two thousand tasks. Above about ¥0.9 per task ReAct is the better choice.
Setup. 50 held-out test tasks, with the configuration frozen on the development set; K=16 is the mean of three runs, K=8 and K=32 are single runs; realized spending in CNY. The per-task cost is execution + retried attempts + planning (the one-off generation of the task DAG). Calibration is paid once per model pair and reused across tasks and splits. ReAct is one saved run, scored at its first K steps (K = 8, 16, 24, 32); its cost is its realized spending.
BRIGADE's TGC at K=16 and K=32 (51.3% against 48%) differs by one to two tasks out of 50, which is within run-to-run variation.
How to read this, at K=16: BRIGADE costs ¥0.60 per task and completes 51% of the tasks. To complete the same share, ReAct needs about ¥0.92 per task (it completes 36% at ¥0.75 and 70% at ¥1.13). So BRIGADE saves about ¥0.32 per task, and the ¥337 is paid back after 337 / 0.32 ≈ 1,050 tasks.
Analysis.
Conclusion. The efficiency gain holds per task under a restricted budget, and in total only after the calibration is amortized over on the order of a thousand tasks. We will report this accounting in the paper and replace the "same K" comparison with the cost–completion comparison above.