- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MASOpt, a framework that changes prompts during a task within a fixed multi-agent workflow. An observer diagnoses visible failures, a repair module edits an agent's prompt or interface instruction, and a feedback gate decides whether to keep the edit after replaying the affected part of the workflow.
Cross-Actuation Relative Advantage (CARA) compares candidate interventions from the same execution prefix during offline calibration. The resulting repair memory stays fixed during deployment. Prompt edits apply only to the current task.
Across six non-interactive benchmarks, MASOpt reports an average score of 83.40, versus 80.58 for MASPOB. Its average success across ALFWorld seen, ALFWorld unseen, and TextCraft is 64.85, versus 39.23 for AFlow. Additional experiments remove experience components and test different models and workflows.
Critical strengths
-
The repair procedure is documented in enough detail to inspect its decisions. Appendices B.2 and B.5 to B.7 specify which invocation receives an edit, record the original and edited prompts, and rerun affected downstream calls over the same comparison horizon. Appendix C.3 groups experience from each independent task. These records allow the effects of an edit to be traced and the repair process to be reproduced.
-
The interactive-task results show substantial performance gains. Table 2 reports 89.55% success on ALFWorld unseen tasks, versus 54.48% for the strongest baseline. MASOpt also solves 24/80 TextCraft tasks, versus MASPOB's 11/80. The controller keeps online observations separate from offline reward labels, and additional experiments test other models and workflows.
Critical weaknesses
-
The claimed guarantee is broader than the acceptance test supports. Property 1 says measured error cannot increase across checkpoints. Yet a new observation can reveal a new violation even when the prompts stay unchanged. Appendix B.7 only compares edits at the same saved state and horizon, and B.6 notes that replay results can vary. Property 1 should state these conditions. Appendix A.4 also says reducing measured error cannot sacrifice correctness. However, agents can agree on a wrong answer, and a program can pass public tests while failing hidden ones. Appendix F.2 acknowledges this limitation. The paper should measure how often accepted edits worsen final task performance.
-
The experiments do not show how much CARA adds beyond extra computation. Appendix A.6 reports 888 TextCraft calls for MASOpt versus 684 for MASPOB, about 29.8% more. Using the same model and environment-step limit still allows MASOpt to make more calls. Token and latency comparisons are also incomplete across benchmarks. Figure 3 removes experience components, but does not compare CARA with raw utility ranking or alternative repair locations using the same candidates and replay budget. The acceptance gate is not separately tested under these controls. Without these comparisons, the gains cannot be clearly attributed to CARA and the controller rather than extra computation.
Questions for authors
-
When does Property 1 hold, and how often do accepted edits worsen final task performance?
-
How does MASOpt compare with retries or feedback-based repair using the same inference budget and visible observations?
-
How is environment state restored for interactive replay, and do the 35/20-step limits include all replayed actions?
-
How much do CARA normalization, repair location, and the acceptance gate each add when candidates and replay budgets are matched?
-
Which task sets, initial prompts, and configurations explain the MBPP scores of 73.314 in Figure 3 and 84.16 in Table 1?
11 comments
Part 2/5: Q2, Q3
Q2 (Weakness 2). Extra computation, incomplete cost reporting, and missing component controls prevent attribution to CARA and the controller.
A2: Thank you for asking us to separate resource use from component contributions. Following your suggestions, we report the shared-budget comparison and evaluate the component ablations separately. The original TextCraft result is 30.00% versus 13.75% success with 888 versus 684 calls (Appendix A.6, Table 4).
Following your suggestion, we compare MASPOB, fixed-prompt retry, feedback-based output repair and MASOpt using the same initial workflow, model, task inputs and visible feedback. Each benchmark uses common per-task call and total-token ceilings; diagnosis, generation, verification and replay are included in the budget. All scores are on a 0–100 scale, using the task metrics in Appendix A.1, Table 3.
The registered per-task ceilings (calls / total tokens) are HumanEval 27 / 35,097; MBPP 27 / 24,677; GSM8K 50 / 87,540; MATH 21 / 85,029; HotpotQA 30 / 55,776; DROP 29 / 18,795; both ALFWorld splits 644 / 2,086,652; and TextCraft 83 / 205,110.
Average is the unweighted mean of the nine displayed benchmark/split scores, with ALFWorld seen and unseen counted separately.
Table B2. Controlled task scores under shared inference ceilings.
| Benchmark | MASPOB | Retry | Output repair | MASOpt |
|---|---|---|---|---|
| HumanEval | 91.22 | 91.37 | 90.84 | 91.60 |
| MBPP | 71.99 | 71.55 | 71.85 | 71.85 |
| GSM8K | 93.65 | 91.03 | 93.22 | 93.84 |
| MATH | 54.42 | 52.45 | 54.63 | 54.84 |
| HotpotQA | 73.82 | 74.04 | 60.87 | 71.85 |
| DROP | 81.29 | 80.93 | 58.57 | 70.50 |
| ALFWorld seen | 45.86 | 44.14 | 45.14 | 71.25 |
| ALFWorld unseen | 48.75 | 46.58 | 49.30 | 76.83 |
| TextCraft | 15.00 | 12.50 | 11.25 | 18.50 |
| Average | 64.00 | 62.73 | 59.52 | 69.01 |
The results support a performance advantage under shared inference ceilings, with the clearest gains on interactive tasks. MASOpt achieves the highest average score even when the baselines receive feedback and correction opportunities. The advantage is not uniform: MASPOB remains stronger on MBPP and DROP, and retry leads on HotpotQA. This addresses the computation concern by showing where instruction adaptation adds value under the controlled budget, without claiming equal actual consumption or superiority on every benchmark.
To clarify actual resource consumption, Table B3 reports measured inference costs. All controller and replay calls are included in the token accounting.
Table B3. Measured inference cost for completed non-ALFWorld conditions.
| Benchmark | Retry tokens/task | Output repair tokens/task | MASOpt tokens/task | MASOpt median latency (s) |
|---|---|---|---|---|
| HumanEval | 13,040 | 12,902 | 13,535 | 169.40 |
| MBPP | 9,672 | 10,097 | 9,752 | 120.39 |
| GSM8K | 31,047 | 28,800 | 32,172 | 132.33 |
| MATH | 46,212 | 41,758 | 44,611 | 111.87 |
| HotpotQA | 22,944 | 24,618 | 22,740 | 64.72 |
| DROP | 8,396 | 9,350 | 9,640 | 141.87 |
| TextCraft | 90,716 | 41,383 | 90,035 | 272.57 |
The cost results support a performance–cost trade-off rather than uniform savings. On TextCraft, MASOpt improves success with slightly fewer tokens than retry, but consumes more tokens than output repair. Shared ceilings therefore control the available budget without forcing equal actual expenditure. Together with Table B2, this addresses both performance and resource use; A6 (Part 4) separately examines the component contributions.
Q3 (Question 1). When does Property 1 hold, and how often do accepted edits worsen final performance?
A3: Thank you for requesting the precise condition and frequency. Property 1 applies to the recorded reference/candidate comparison at the same saved state, horizon and public checks, with both continuations completed and the gate conditions satisfied. Rejected candidates leave the incumbent unchanged. It does not extend that ordering to a new state or a newly sampled stochastic replay.
Table B1 (Part 1) in A1 reports the requested frequencies. The task-averaged decline rates are 32.15% on HotpotQA and 25.33% on DROP. No decline was observed in the evaluated HumanEval, MBPP, GSM8K, MATH and TextCraft pairs, or in the verified ALFWorld subset. These are conditional measurements of accepted edits, not whole-system error rates. They motivate reporting final correctness independently of the local residual guarantee.
Part 3/5: Q4, Q5
Q4 (Question 2). How does MASOpt compare with retries or feedback-based repair under the same inference budget and visible observations?
A4: Thank you for asking whether the gains persist with equal access to feedback and inference opportunities. Yes, MASOpt retains an aggregate advantage over both runtime baselines. In Table B2 (Part 2), the average gains over retry and output repair are 6.27 and 9.49 points, respectively, calculated before rounding.
The clearest benefits occur on the interactive tasks, while the comparison also identifies benchmarks where a simpler runtime baseline remains competitive. Because the baselines share the observations and resource ceilings and can already retry or revise outputs, their lack of correction opportunities cannot explain the overall advantage. The conclusion is an empirical benefit from instruction adaptation under these controls, with task-dependent gains and potentially different actual token consumption.
Q5 (Question 3). How is interactive state restored, and do the 35/20-step limits include every replayed action?
A5: Thank you for asking us to separate trajectory limits from total interaction costs. The environment is reconstructed by resetting the same native task and seed and executing the saved action prefix. No, the 35/20-step limits apply to the retained trajectory, not to all replayed actions. Before branching, the runner verifies the reconstructed public state, step count and state fingerprint. All six checked prefixes matched the saved checkpoints. Appendices B.2 and B.6 specify the checkpoint and replay scope.
The measured distinction between the retained trajectory and all executed actions is:
Table B4. MASOpt environment actions per task in the main controlled comparison.
| Benchmark | Retained trajectory | All executed actions |
|---|---|---|
| ALFWorld seen | 34.86 | 1,203.33 |
| ALFWorld unseen | 34.76 | 1,209.43 |
| TextCraft | 16.27 | 51.92 |
Retained-trajectory length substantially understates total environment interaction. Prefix reconstruction and alternative continuations account for the gap, so they must be included when evaluating interaction cost. Additional diagnostic audits are accounted for separately. Following your suggestion, we impose a common ceiling on all TextCraft actions, including reconstruction and replay:
Table B5. TextCraft under a shared ceiling on all environment actions.
| Method | Success (%) |
|---|---|
| MASPOB | 13.12 |
| Retry | 12.50 |
| Output repair | 11.25 |
| MASOpt | 15.88 |
MASOpt retains its TextCraft advantage when reconstruction and replay actions are charged. It outperforms all three baselines under the total-action ceiling, so the gain in this setting is not explained solely by allowing extra environment interaction. Table B4 reports actual action use under the main inference-budget protocol, while Table B5 controls total actions directly. The corresponding ALFWorld comparison remains incomplete. We will make these accounting definitions explicit in Appendix A.3 and the cost analysis.
Response to Reviewer bV8t: Guarantees, Costs, and Component Analyses
Part 1/5: Q1
Thank you for your detailed review and for recognizing the traceability of the repair procedure. The added controls support a higher average score under shared inference ceilings and a TextCraft advantage when all environment actions are charged. We narrow the guarantee to matched-checkpoint retention, quantify when accepted edits reduce final scores, and explain the MBPP configuration discrepancy. Your seven points are answered individually below.
Q1 (Weakness 1). The stated guarantee exceeds the acceptance test, and accepted edits may worsen final correctness.
A1: Thank you for identifying where the theoretical statement needs more precise conditions. We will restrict Property 1 to verified retention at a matched checkpoint: the controller keeps an edit only after a completed comparison demonstrates a lower public residual under the same state, horizon and checks. It also requires non-decreasing progress and valid constraint/budget checks. Otherwise, it retains the reference, as specified in Appendix B.7.
This design uses calibration on downstream development-task outcomes to guide local repair choices, but the acceptance rule itself establishes a local observable property. A later observation can reveal a new violation, and public-test agreement need not imply hidden-test correctness. We will state the matched-state conditions directly in Property 1 and align Appendix A.4 with the replay and scoring distinctions in Appendices B.6–B.7 and F.2.
To answer the empirical part, we replay accepted edits from the same checkpoint both with and without the edit and compare their final scores:
Table B1. Final-score decline after an accepted edit.
Accepted edits can satisfy the public acceptance test while reducing final correctness. The declines on HotpotQA and DROP demonstrate why Property 1 must be stated as a matched-checkpoint guarantee. The zero entries do not establish universal safety: the HumanEval, TextCraft and ALFWorld audits contain only four, two and six evaluable pairs, respectively. We will report the local property and final-task outcomes separately.