- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies runtime error healing, in which an LLM agent modifies the state of a paused program to recover from an exception and continue execution without changing repository source code. It introduces HealBench, containing 265 runtime errors from 18 Python repositories, assembled from SWE-Bench, SWE-Bench Pro, and R2E-Gym. Each instance includes a target test and a patched reference execution, while a shared framework supports repository exploration, live-state inspection, and healing submissions. The authors evaluate Healer and three general coding agents across three backbone models using execution completion, target-test success, and trace similarity. The strongest reported configuration achieves 38.11% completion and 28.68% target-test success. The paper also proposes HealGuard, which treats healing-induced state changes as taint sources and developer-protected operation inputs as sinks. Its design combines HealCore, a restricted Python subset, with static analysis and dynamic checks for unresolved dependencies. Evaluation includes offline analysis of saved executions and 684 templated detection configurations derived from recorded healing points. The evaluated checking sequence detects all 342 configurations labeled unsafe but also blocks 234 of 342 labeled safe, yielding a 68.42% false-positive rate. Case studies illustrate how healing can introduce file deletion, subprocess execution, or unnecessary output even when addressing the original error.
Critical strengths
HealBench transforms existing repository-repair tasks into a setting where agents must repair a paused execution without modifying source files or restarting the process.
Critical weaknesses
-
The practical trade-off between safety and successful recovery is insufficiently justified. HealGuard achieves 100% recall on the controlled evaluation but incorrectly blocks 68.42% of safe cases, and the main paper does not adequately justify why this level of rejection is acceptable for the intended applications (§5.2). Given that the purpose of runtime healing is to preserve useful execution, stronger evidence is needed that the proposed protection retains sufficient recovery capability to provide a practical benefit.
-
The safety evaluation provides limited evidence of effectiveness on naturally occurring healing failures. The headline detection result comes from constructed source–sink injection cases, while flagging potentially unsafe executions and presenting illustrative examples does not establish detection accuracy across representative repository executions (§5). Evaluating independently labeled, naturally occurring healing attempts—and comparing against simpler conservative protection mechanisms—would better establish both real-world effectiveness and the added value of the proposed guard.
Questions for authors
- When is live-state healing preferable to restarting or propagating the exception, and what application-level correctness and safety conditions should determine that choice?
- Could you evaluate the complete guard in matched online runs using the specified rule for invoking dynamic checks, compare it with static-only and dynamic-only variants, and report independently verified violations prevented, correct recoveries over all benchmark instances, and total runtime overhead?
Response to Reviewer MQUs.
Thank you for your constructive review. We promise to revise our manuscript to address all the points raised.
Q1: When to use live-state healing
Live-state healing is preferable when restarting is costly and terminating execution would lose completed work.
Long-running computations may have intermediate results, such as model training progress that would be lost after the last checkpoint. An unhandled error can also terminate a service process and interrupt other requests, as in vLLM CVE-2026-105757. Healing repairs the state at the crash point and resumes execution, avoiding repeated computation and interruption of other requests.
Q2: Matched runs, variant comparisons, violations prevented, correct recoveries, and runtime overhead
(1) Matched online runs and variant comparisons. We run the application and HealGuard online on the same instances and initial environments, replaying recorded LLM outputs without new LLM calls to fix healing actions and avoid output variation. The complete HealGuard follows Section 4.3, enabling dynamic checking when static analysis returns unknown. Table 5 compares static-only, dynamic-only, and + HealGuard.
(2) Verified violations prevented. Human Validation reports manual review of 265 mini-SWE-agent cases. The review confirms that HealGuard prevents violations in all 17 unsafe cases before the protected operations execute, achieving 100% recall with 49 false positives.
(3) Correct recoveries. We add CR-All (%) using all 265 benchmark instances as the denominator, as shown below.
(4) Runtime overhead. We analyze the time spent handling crashes with mini-SWE-agent. Mean heal time is 58.3782s, including agent processing and healing code execution. Static analysis averages 0.0032s and dynamic checking 0.0054s, adding 0.0147% relative to heal time. Code graphs and related preparation are completed offline and reused across crashes in the same repository.
W1: Rejection acceptability and recovery capability
(1) Rejection acceptability. A high false positive rate may be acceptable when the cost of allowing dangerous behavior exceeds the task interruption cost of wrongly rejecting safe healing. Developers set protection scope and strictness and assess application consequences. Acceptability therefore depends on the applications and developers' requirements.
(2) Sufficient recovery capability. HealGuard aims to prevent unsafe execution while limiting lost recovery opportunities. It rejects 41 executions without successful recovery and 25 with successful recovery across 265 mini-SWE-agent tasks with GPT-5.6-Terra. Rejections are therefore more concentrated among executions without successful recovery, showing that HealGuard retains sufficient recovery capability.
W2: Labeling and baseline comparison
(1) Independent labeling. Human Validation in the appendix compares manual review labels with guard decisions for 265 mini-SWE-agent cases. HealGuard detects all 17 unsafe cases and incorrectly rejects 49 safe cases, achieving 25.76% precision and 100% recall. These results complement the controlled source–sink injection cases.
(2) Simpler conservative protection. We add a code blacklist that rejects direct calls to developer-declared protected operations. The blacklist detects no unsafe cases and produces 4 false positives, with 0.00% precision, 0.00% recall, and 1.61% FPR. HealGuard achieves 25.76% precision, 100.00% recall, and 19.76% FPR. It also shows HealGuard's effectiveness.