acceptodds
Under review as a conference paper at ICLR 2027

Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrails

Abstract

Runtime error healing lets a crashed program continue by generating code that repairs its live runtime state. Recent work shows that LLMs can generate such healing code, but it is evaluated only on small competition programs, and executing LLM-generated code inside a live process raises safety concerns that remain unaddressed. In this paper, we take LLM-based runtime healing toward practical use in real-world repositories. We first build HealBench, a benchmark of 265 runtime errors from 18 real-world repositories, each paired with a reference execution on the patched version. HealBench also provides a unified framework that lets LLM agents heal with cross-file context and live runtime state. We then design HealGuard, which requires healing code to be written in HealCore, an analyzable subset of Python, and uses static and dynamic taint analysis to check whether state changed by healing reaches operations protected by developers. We evaluate a dedicated healing method and three general coding agents with three backbone LLMs. The best setting resumes execution in 38.11% of instances and passes the target test in 28.68%, showing that existing agents can already heal a meaningful share of real repository-level crashes. However, among executions that pass, HealGuard flags 17.4% whose healing-changed state may reach a protected operation. On 684 controlled cases, HealGuard detects all unsafe cases, at the cost of a 68.42% false positive rate.

Primary area:
datasets and benchmarks
Keywords:
Runtime Error Healing, Self-Healing Software, LLM Agents, Runtime Recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MQUsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies runtime error healing, in which an LLM agent modifies the state of a paused program to recover from an exception and continue execution without changing repository source code. It introduces HealBench, containing 265 runtime errors from 18 Python repositories, assembled from SWE-Bench, SWE-Bench Pro, and R2E-Gym. Each instance includes a target test and a patched reference execution, while a shared framework supports repository exploration, live-state inspection, and healing submissions. The authors evaluate Healer and three general coding agents across three backbone models using execution completion, target-test success, and trace similarity. The strongest reported configuration achieves 38.11% completion and 28.68% target-test success. The paper also proposes HealGuard, which treats healing-induced state changes as taint sources and developer-protected operation inputs as sinks. Its design combines HealCore, a restricted Python subset, with static analysis and dynamic checks for unresolved dependencies. Evaluation includes offline analysis of saved executions and 684 templated detection configurations derived from recorded healing points. The evaluated checking sequence detects all 342 configurations labeled unsafe but also blocks 234 of 342 labeled safe, yielding a 68.42% false-positive rate. Case studies illustrate how healing can introduce file deletion, subprocess execution, or unnecessary output even when addressing the original error.

Critical strengths

HealBench transforms existing repository-repair tasks into a setting where agents must repair a paused execution without modifying source files or restarting the process.

Critical weaknesses
  • The practical trade-off between safety and successful recovery is insufficiently justified. HealGuard achieves 100% recall on the controlled evaluation but incorrectly blocks 68.42% of safe cases, and the main paper does not adequately justify why this level of rejection is acceptable for the intended applications (§5.2). Given that the purpose of runtime healing is to preserve useful execution, stronger evidence is needed that the proposed protection retains sufficient recovery capability to provide a practical benefit.

  • The safety evaluation provides limited evidence of effectiveness on naturally occurring healing failures. The headline detection result comes from constructed source–sink injection cases, while flagging potentially unsafe executions and presenting illustrative examples does not establish detection accuracy across representative repository executions (§5). Evaluating independently labeled, naturally occurring healing attempts—and comparing against simpler conservative protection mechanisms—would better establish both real-world effectiveness and the added value of the proposed guard.

Questions for authors
  • When is live-state healing preferable to restarting or propagating the exception, and what application-level correctness and safety conditions should determine that choice?
  • Could you evaluate the complete guard in matched online runs using the specified rule for invoking dynamic checks, compare it with static-only and dynamic-only variants, and report independently verified violations prevented, correct recoveries over all benchmark instances, and total runtime overhead?
1 comment
AuthorsOpenReview ↗

Response to Reviewer MQUs.

Thank you for your constructive review. We promise to revise our manuscript to address all the points raised.

Q1: When to use live-state healing

Live-state healing is preferable when restarting is costly and terminating execution would lose completed work.

Long-running computations may have intermediate results, such as model training progress that would be lost after the last checkpoint. An unhandled error can also terminate a service process and interrupt other requests, as in vLLM CVE-2026-105757. Healing repairs the state at the crash point and resumes execution, avoiding repeated computation and interruption of other requests.

  • Correctness. (1) Normal execution. The updated state should meet later code's type, value, and state requirements, and continued execution should produce the expected result. (2) Exception handling. Expected exceptions should propagate to their intended handlers without being suppressed to bypass required checks.
  • Safety. (1) Direct effects. Healing code operations and state changes should respect application constraints. Object changes and external side effects left by failed attempts should stay within acceptable limits. (2) Later effects. Healing-generated values and state changes should not affect the inputs of developer-declared protected operations through data flow.

Q2: Matched runs, variant comparisons, violations prevented, correct recoveries, and runtime overhead

(1) Matched online runs and variant comparisons. We run the application and HealGuard online on the same instances and initial environments, replaying recorded LLM outputs without new LLM calls to fix healing actions and avoid output variation. The complete HealGuard follows Section 4.3, enabling dynamic checking when static analysis returns unknown. Table 5 compares static-only, dynamic-only, and + HealGuard.

(2) Verified violations prevented. Human Validation reports manual review of 265 mini-SWE-agent cases. The review confirms that HealGuard prevents violations in all 17 unsafe cases before the protected operations execute, achieving 100% recall with 49 false positives.

(3) Correct recoveries. We add CR-All (%) using all 265 benchmark instances as the denominator, as shown below.

(4) Runtime overhead. We analyze the time spent handling crashes with mini-SWE-agent. Mean heal time is 58.3782s, including agent processing and healing code execution. Static analysis averages 0.0032s and dynamic checking 0.0054s, adding 0.0147% relative to heal time. Code graphs and related preparation are completed offline and reused across crashes in the same repository.

MethodSettingGPT-5.6-Terra CR-AllDeepSeek-V4-Flash CR-AllGLM-5.2 CR-All
HealerVanilla16.9810.944.91
Healer+ HealGuard9.815.283.40
mini-SWE-agentVanilla24.5316.2325.28
mini-SWE-agent+ HealGuard15.0912.4512.45
OpenHandsVanilla21.518.3012.08
OpenHands+ HealGuard15.476.429.43
CodexVanilla28.6815.4720.75
Codex+ HealGuard23.4012.4516.98

W1: Rejection acceptability and recovery capability

(1) Rejection acceptability. A high false positive rate may be acceptable when the cost of allowing dangerous behavior exceeds the task interruption cost of wrongly rejecting safe healing. Developers set protection scope and strictness and assess application consequences. Acceptability therefore depends on the applications and developers' requirements.

(2) Sufficient recovery capability. HealGuard aims to prevent unsafe execution while limiting lost recovery opportunities. It rejects 41 executions without successful recovery and 25 with successful recovery across 265 mini-SWE-agent tasks with GPT-5.6-Terra. Rejections are therefore more concentrated among executions without successful recovery, showing that HealGuard retains sufficient recovery capability.

W2: Labeling and baseline comparison

(1) Independent labeling. Human Validation in the appendix compares manual review labels with guard decisions for 265 mini-SWE-agent cases. HealGuard detects all 17 unsafe cases and incorrectly rejects 49 safe cases, achieving 25.76% precision and 100% recall. These results complement the controlled source–sink injection cases.

(2) Simpler conservative protection. We add a code blacklist that rejects direct calls to developer-declared protected operations. The blacklist detects no unsafe cases and produces 4 false positives, with 0.00% precision, 0.00% recall, and 1.61% FPR. HealGuard achieves 25.76% precision, 100.00% recall, and 19.76% FPR. It also shows HealGuard's effectiveness.

Reviewer_7nQoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a benchmark on evaluating agents' fixing runtime errors. Together with that dataset, this work also proposes a problem, which monitors whether a fix will violate some pre-defined action constraint.

Critical strengths

1, The work targets an interesting problem: reliably fixing runtime errors.

2, The resulting benchmark serves to distinguish the performances of different coding agents.

3, The manuscript is well written and easy to follow.

Critical weaknesses

1, The runtime error is just a subset of swe-bench like tasks. The significance of selecting a subset and claiming a new benchmark is not clear to me.

2, While the trustworthy error/bug fixing makes sense to me, formulating the problem as avoiding operations in a set seems not to be common practice. I am open to change my view if reliable references could be provided.

Questions for authors

Listed above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7nQo.

Thank you for your constructive review. We promise to revise our manuscript to address all the points raised.

Q1/W1: Benchmark contribution

We adapt existing SWE-bench-like tasks for runtime healing rather than simply selecting a subset. Using their repositories, tests, and reference patches, we run buggy versions to record exception types, messages, tracebacks, and crash points. We then retain errors related to the target bug where the crash frame remains accessible and execution can continue in the current process. We also run the same tests on patched versions to pair each retained error with a target test and reference execution results and paths. This construction provides the crash points and recovery evaluation references needed for runtime healing. The differences are as follows.

(1) Task goal. Existing benchmarks require fixing repository bugs so that the modified program passes tests. We require recovering the current interrupted execution so that the program continues and passes the target test.

(2) Repair target. Existing benchmarks modify repository source code. We use healing code to update the runtime state at the crash point.

(3) Execution process. Existing benchmarks rerun tests from the start after modifying source code. We pause the program when an exception occurs, execute healing code, and resume the current process until the target test ends.

Q1/W2: Safety policy and references

We apologize for the unclear description. We do not prohibit all operations in the set. We check whether healing affects the key inputs of protected operations, following the source–sink taint analysis framework.

(1) Existing practice. LATTE [1], EvoTaint [2], and CodeQL [3] define sources of untrusted data and sinks at sensitive inputs. Sources identify where untrusted data comes from, and sinks identify the inputs to protect. They track whether data flows from sources to sinks, supporting this way of defining the safety policy.

(2) Application to healing. Healing code is generated by an LLM at runtime, so we treat its values and state changes as untrusted. Developers specify protected operations and their key inputs. We treat values and state changes introduced by healing code as sources and these key inputs as sinks. We check whether healing affects the protected inputs. When the inputs are independent of healing, the policy allows the application to perform these operations.

References

[1] Liu et al. LLM-Powered Static Binary Taint Analysis. ACM Transactions on Software Engineering and Methodology, 2025.

[2] Guo and Cai. EvoTaint: Incremental Static Taint Analysis of Evolving Android Apps. ACM Transactions on Software Engineering and Methodology, 2026.

[3] https://codeql.github.com/docs/codeql-language-guides/analyzing-data-flow-in-python

Reviewer_4neaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors builds HealBench, a benchmark of 265 runtime-error instances from 18 repositories, together with a framework that allows agents to inspect and modify live runtime state. They evaluate four healing methods with three backbone LLMs, achieving up to 38.11% of proceed rate and 28.68% of target-test correct rate.

They also propose HealGuard, which combines a restricted Python subset with static dependency analysis to detect and block unsafe healing. They gives offline analysis of recorded healing executions and 684 controlled configurations, on which the evaluated checking procedure achieves 100% recall with a 68.42% false-positive rate.

Critical strengths

The authors propose a practical benchmark and framework for repository-level runtime healing. HealBench connects runtime failures in real-world repositories with target tests and patched reference executions. And the framework supports repository exploration, live-state inspection, and healing submissions. Furthermore, they address runtime error healing as well as safety policy, while conventional test-based recovery methods may miss additional effects introduced by healing.

Critical weaknesses

1.In Section 4.3, the authors specify a sequential runtime procedure in which dynamic checking is enabled only when static analysis returns an unknown verdict. However, in Appendix C.4, they state that dynamic checking was applied to all the 495 configurations not blocked by static analysis, and that none of these configurations received an unknown verdict. Therefore, the reported detection result does not directly evaluate the unknown-triggered fallback specified in the runtime procedure. The scope of the safety claim need to be clarified and limited accordingly.

2.The evaluation of HealGuard does not establish that protected operations are actually blocked during application execution.

3.The reported results are based on controlled configurations, simulated cases, and saved decisions. They provide evidence about detection decisions, but do not directly provide the evidence of runtime prevention. Therefore, the claims about blocking unsafe healing need to be stated more narrowly unless application-level evidence is provided.

4.Table 2 does not provide consistent basis for comparing Vanilla with + HealGuard. The two settings differ in the handling of retained executions and in the way TS is calculated. The authors need to clarify the evaluation basis and ensure that the reported metrics are calculated and presented consistently.

Questions for authors

The authors builds HealBench, a benchmark of 265 runtime-error instances from 18 repositories, together with a framework that allows agents to inspect and modify live runtime state. They evaluate four healing methods with three backbone LLMs, achieving up to 38.11% of proceed rate and 28.68% of target-test correct rate. They also propose HealGuard, which combines a restricted Python subset with static dependency analysis to detect and block unsafe healing. They gives offline analysis of recorded healing executions and 684 controlled configurations, on which the evaluated checking procedure achieves 100% recall with a 68.42% false-positive rate.

However, the following issues still need further clarification and revision:

1.In Section 4.3, it is specified a sequential runtime procedure in which dynamic checking is enabled only when static analysis returns an unknown verdict. While in Appendix C.4, it is stated that dynamic checking was applied to all the 495 configurations not blocked by static analysis, and that none of these configurations received an unknown verdict. Please clarify that the reported 100% recall evaluates the supplied checking experiment rather than the unknown-triggered fallback specified in Section 4.3. If the fallback procedure is not evaluated on cases that produce an unknown verdict, please limit the corresponding claim accordingly.

  1. The controlled configurations and saved decisions do not establish that HealGuard prevented protected operations during application execution. Please provide application-level evidence of runtime prevention, or limit the claims about blocking unsafe healing to the controlled and offline evaluation settings.

  2. In Table 2, the authors uses different evaluation bases for Vanilla and + HealGuard, and there are differences in retained executions and TS calculation. Please clarify the denominators and evaluation sets used for the reported metrics, and use a consistent TS calculation for both settings.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4nea.

Thank you for your constructive review. We promise to revise our manuscript to address all the points raised.

Q1/W1: Unknown-triggered fallback and recall

The reported 100% recall is the result of the sequential checking procedure across all 684 configurations. Static analysis detects 177 unsafe cases. Dynamic checking detects the remaining 165, covering all 342 unsafe cases.

The statement "none of these configurations received an unknown verdict" in Appendix C.4 is an oversight in the description. It should be "dynamic checking was applied to the 495 configurations that received an unknown verdict from static analysis."

Q2/W2/W3: Application-level runtime prevention

We run application tests by replaying recorded agent actions, rather than analyzing saved decisions. At each crash point, we replay the recorded state checks and healing code in their original order, then resume the application. HealGuard provides runtime prevention by checking protected operations before they execute and blocking policy violations. And the appendix Healing Safety Cases shows application-level operations that need protection, including cache deletion, subprocess calls, and extra cookie output. We replay the recorded healing actions without calling the LLM again. This avoids variation in LLM outputs and keeps the healing actions and their call order reproducible.

In addition to replay, we run mini-SWE-agent with GPT-5.6-Terra on 20 instances, generating healing code and running HealGuard during application execution. HealGuard blocks 3 unsafe healing attempts, preventing subsequent protected operations including file renaming, directory deletion, and output calls. Manual inspection confirms that the corresponding side effects do not occur, consistent with runtime prevention observed during replay.

Q3/W4: Evaluation sets, denominators, and TS

Vanilla uses all 265 instances. + HealGuard uses N executions allowed to continue by runtime safety checks. Blocked executions are excluded from its evaluation set. Vanilla measures overall recovery performance, while + HealGuard measures recovery among the retained executions. The denominator is 265 for Vanilla and N for + HealGuard.

PR measures the proportion of executions that complete the target call. CR measures the proportion that pass the target test. TS is the average across instances of the matched location visit count between healed and gold patch executions divided by the total gold patch visit count.

Table 2 is expanded below. PR, CR, and TS show /265 (/N) percentages.

MethodSettingGPT-5.6-TerraDeepSeek-V4-FlashGLM-5.2
NPRCRTSNPRCRTSNPRCRTS
HealerVanilla26529.43 (29.43)16.98 (16.98)32.68 (32.68)26524.91 (24.91)10.94 (10.94)29.98 (29.98)2657.92 (7.92)4.91 (4.91)5.98 (5.98)
Healer+ HealGuard16716.98 (26.95)9.81 (15.57)18.65 (29.59)10711.70 (28.97)5.28 (13.08)5.84 (14.45)2544.91 (5.12)3.40 (3.54)4.55 (4.75)
mini-SWE-agentVanilla26536.60 (36.60)24.53 (24.53)27.71 (27.71)26520.38 (20.38)16.23 (16.23)6.62 (6.62)26532.83 (32.83)25.28 (25.28)16.04 (16.04)
mini-SWE-agent+ HealGuard19924.15 (32.16)15.09 (20.10)15.01 (19.99)24816.60 (17.74)12.45 (13.31)5.25 (5.61)19717.36 (23.35)12.45 (16.75)8.00 (10.76)
OpenHandsVanilla26529.81 (29.81)21.51 (21.51)18.28 (18.28)2659.43 (9.43)8.30 (8.30)2.34 (2.34)26515.47 (15.47)12.08 (12.08)7.43 (7.43)
OpenHands+ HealGuard21822.26 (27.06)15.47 (18.81)9.40 (11.43)2577.17 (7.39)6.42 (6.61)1.84 (1.89)16412.45 (20.12)9.43 (15.24)5.51 (8.90)
CodexVanilla26538.11 (38.11)28.68 (28.68)17.02 (17.02)26520.00 (20.00)15.47 (15.47)9.25 (9.25)26526.42 (26.42)20.75 (20.75)13.95 (13.95)
Codex+ HealGuard23232.08 (36.64)23.40 (26.72)9.91 (11.31)24816.98 (18.15)12.45 (13.31)6.57 (7.02)23221.13 (24.14)16.98 (19.40)8.67 (9.90)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Reminder (Submission 29114)

Dear Area Chair,

Thank you for the time you have devoted to coordinating the reviews and facilitating the discussion. We recognize that the conference review process could not run smoothly without the work of ACs and SACs.

We have posted comprehensive, point-by-point responses addressing all the concerns raised by Reviewers 4nea, 7nQo, and MQUs, supported by substantial additional experimental results:

  1. Clarified how HealBench adapts existing repository repair tasks to recover the current execution by updating the live runtime state, and provided references supporting the use of sources and sinks to define the safety policy (addressing Reviewer 7nQo).
  2. Clarified that dynamic checking runs when static analysis returns an unknown verdict, and provided application-level evidence that HealGuard prevented protected operations from executing in replay runs and in runs with newly generated healing code (addressing Reviewers 4nea and MQUs).
  3. Compared HealGuard with static-only and dynamic-only variants and a code blacklist, reported manually verified violations prevented and runtime overhead during crash handling, and reported recovery metrics for Vanilla and + HealGuard using the same set of benchmark instances (addressing Reviewers 4nea and MQUs).

We are currently finalizing the revised manuscript incorporating these updates and will upload it shortly before the deadline.

As the discussion period is progressing and we have not yet heard back from any of the reviewers, we would be very grateful if you could kindly send a brief reminder encouraging them to check our rebuttal responses. We would be glad to engage further and answer any remaining questions.

Thank you again for your time and help.

Best regards,
Authors of Submission 29114

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.