- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to choose repairs for failing LLM multi-agent systems when a total repair budget is shared across task instances, using intervention responses that have already been collected.
-
Benchmark: It introduces EvidenceGraph, a synthetic benchmark of 492 matched blocks drawn from three strata. In each block, one task is instantiated with six planted failure mechanisms in a four-role pipeline that ends in a deterministic ACCEPT/REJECT reducer. Every system is replayed under nine ordered repair sequences with fixed unit costs, giving 26,568 outcomes.
-
Theory: The paper defines best-fixed, observable-adaptive, and full-response values (Thm. 1). It bounds the regret of an observable shared-budget policy against a full-information oracle by four terms: coverage, information, twice the prediction error, and allocation error (Thm. 2).
-
Method: OAE computes 11 features from baseline, edge-mask, and conditional source-swap replays. It fits nine logistic heads that predict whether each sequence's replay will match its expected answer, then allocates greedily by predicted gain per unit cost.
-
Evaluation: Development, Primary, and Fresh cohorts are disjoint, with a prespecified comparator (PARE), matched-block bootstrap intervals, and max-T adjustment for multiple comparisons. On Fresh, OAE-Greedy reaches 42.44% success versus 34.26% for PARE at 2,592 units, an 8.18-point gain with 95% interval [6.02, 10.34].
Critical strengths
-
Careful, preregistered evaluation on a matched design. The cohorts are disjoint, and scoring keys stay hidden until policies are locked. Resampling is done at the block level, with max-T adjustment applied separately to Primary and Fresh (Sec. 3.1, 5.1). Matched six-mechanism blocks with clean-fork replays (Fig. 2) let repair outcomes be compared from a common snapshot. This makes the Fresh contrast a genuine held-out test, which is more rigorous than most work on multi-agent failure analysis.
-
The response landscape shows that adaptive selection matters. Outcomes depend strongly on the failure mechanism (Table 2, Fig. 3). At least one repair sequence succeeds for 99.77% of Fresh systems, versus 66.90% for the best single sequence (Sec. 5.2). This motivates the problem well.
Critical weaknesses
- The headline gain is measured against the authors' own variant, and a trivial equal-spend baseline is missing. The 8.18-point result in the abstract is relative to PARE, which is OAE plus residual corrections, not an external method.
- The only fixed baseline spends half the budget: "Best fixed" spends 1,296 units (Table 4: "best fixed 1,296, others 2,592"). Yet applying the cost-2 sequence MS (edge mask, then source swap) to every system spends exactly 2,592 units. From Table 2, MS averages about 37.4% on Fresh. That would beat every equal-spend baseline in Table 3, including PARE, and leave OAE only about 5 points ahead rather than 8. It also beats the Exact, Lagrangian, and LocalSearch allocators (33.9–36.4%, Table 5).
- Probe cost is excluded: The budget "covers terminal repairs, excluding replay-bank acquisition" (Sec. 4.3). OAE's observations are themselves edge-mask and source-swap replays on every system, costing about 1–2 units each, which is comparable to the whole terminal budget. Fixed and random baselines don't need these probes.
- Binary answers make the probes informative: Answers are ACCEPT/REJECT and the baseline is wrong 94% of the time. Seeing that a probe flips the answer therefore nearly reveals success, and only OAE gets this signal. Together these mean it is not yet established that observable responses guide budgeted repair better than simple alternatives. Report fixed MS at 2,592 units, with block-bootstrap intervals for OAE minus MS. Then either charge probe cost to the budget, or compare against a simple rule that uses the same probes.
- The learned predictor is near chance, so it is unclear what OAE learns, and the setting is narrow relative to the framing.
- The predictor is weak. On Fresh, its AUROC is 0.533 and its calibration error is 0.428 (Sec. 5.5).
- Better optimization of its scores hurts. Allocating exactly on the predicted gains loses 8.56 points to the greedy rule (Table 5). The win may come from the greedy rule's structure rather than from the predictions. -The gain is concentrated. It comes from two mechanisms: terminal corruption (+25.5 points) and shared retriever (+31.9). It is partly offset on source mismatch (−6.4), where OAE gets 26.9% while fixed MS gets 94.0% (Fig. 6a). The ranking also reverses at budgets of 324 and 1,296 units (Sec. 5.4), which the abstract does not mention.
- The theory is never instantiated. None of the Thm. 2 terms are estimated. With a calibration error near 0.43, the prediction-error term alone is likely near 0.86, which makes the bound vacuous here.
- The benchmark is narrow. It is fully synthetic, and the same model family generated and reviewed all records with no human check. Replays use one LLM. The "multi-agent system" ends in a deterministic binary reducer, and mechanisms map almost one-to-one to repairs.
Questions for authors
- Please report fixed MS on all systems at 2,592 units on Primary and Fresh, with matched-block intervals for OAE minus MS. Why was "Best fixed" limited to a cost-1 sequence? If OAE's advantage stays clearly positive after max-T adjustment, I would weigh Weakness 1 much less.
- What is the average unit cost of acquiring the observations per system? How do the results change if it is charged against the budget? How does OAE compare with a simple rule that uses the same probes: choose the cheapest probed sequence whose answer differs from baseline; otherwise, a default sequence?
- Is the 0.533 AUROC measured against task success or against replay agreement? What does OAE-Greedy achieve with permuted or random scores under the same allocator? Why does exact maximization of the predicted gains lose 8.56 points?
- What are the full-information budgeted value and the Thm. 2 terms on Fresh at 2,592 units? Knowing where OAE sits relative to them would show whether the theory says anything about the method actually run.
- Are replays deterministic, and at what temperature? If observing a probe's answer equals its final outcome, the "hidden outcome" setup is weaker than described. Do the conclusions hold with a second behavior model?
- The cost-matched random baseline's intervention rate equals OAE's on Fresh (61.96%) but PARE's on Primary (87.58%). Which policy is it matched to?