- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes the Cross-Surface Constrained Recovery Model, a framework for evaluating recovery strategies in failed multi-agent workflows. The authors evaluate the framework primarily on CRPA-R1, a synthetic benchmark consisting of 144 template-generated traces, and investigate how constraints on intervention locations, action spaces, and budgets affect recovery performance. The results suggest that allowing interventions beyond the diagnosed root cause can improve recovery rates.
Critical strengths
None
Critical weaknesses
The paper appears insufficiently developed in its current form. While the research question is potentially interesting, the limited experimental validation, unclear methodological novelty, and presentation issues significantly reduce the contribution of this work.
-
Limited alignment between motivation and evaluation. The paper motivates recovery from failures in multi-agent systems, yet its primary experiments rely on synthetic scenarios with predefined root causes and intervention outcomes. It remains unclear whether the findings generalize to realistic multi-agent failures.
-
Insufficient experimental validation. CRPA-R1 contains only 144 template-generated traces, with limited discussion of the underlying agent design, failure-generation process, and representativeness. The AgentRx evaluation covers only 27 calibration cases and compares continuation rules rather than directly validating the proposed cross-surface recovery framework.
-
Limited methodological novelty. Feasibility-Gated Joint Replay Search primarily enumerates feasible intervention combinations and selects the one maximizing recovery. The methodological contribution beyond expanding the intervention space and applying constrained exhaustive search is unclear.
-
Limited practical significance of oracle-based comparisons. The primary results compare an offline oracle against restricted intervention sets. Since the full candidate set subsumes the restricted sets, its optimal recovery value is guaranteed to be no worse. These comparisons quantify the cost of restricting intervention locations but do not demonstrate improvements achievable by a practical recovery method under comparable computational budgets.
-
Unclear experimental presentation. The results introduce multiple settings (e.g., Full, Strict-Root, RCF-H, all traces, and root-feasible) without clearly explaining their distinctions. For example, Table 1 reports comparisons across subsets of 144, 96, and 48 traces, making the results difficult to interpret without consulting other sections.
Minor weakness
Overly complex conceptual figure. Figure 2 combines the framework, algorithm, theoretical analysis, and experimental findings into an overly dense illustration. Several components are difficult to identify; for example, the downstream check discussed in Lines 161–189 is not clearly indicated. A simplified figure focusing on the core workflow would improve clarity.
Questions for authors
See weakness