acceptodds
Under review as a conference paper at ICLR 2027

SysRepair-Bench: Measuring Whether Agents Can Fix a Vulnerability Without Breaking the Service

Abstract

Vulnerability remediation is the operator's last step: change a running system so a disclosed weakness is no longer exploitable while the protected service keeps serving. No benchmark measures this. Offensive benchmarks reward exploits with no availability obligation, and repair benchmarks edit source and re-validate a rebuilt target. We present SysRepair-Bench, remediation scenarios, each a live vulnerable service an agent must fix through a shell; the binary scenarios are graded by a two-component verdict: the exploit is blocked and the service stays healthy. Every scenario in the main grid runs in two conditions, one supplying the vulnerability report and one requiring the agent to find the fault itself. Our central result is the gap between them. Given the report, the median model and suite cell reach , and the strongest models approach the ceiling; required to locate the fault first, the same models on the same hosts with the same budget reach a median of . Every model loses ground, by a median of points. The report-informed condition is an anchor: it measures execution once a fault is named, and the gap is the cost of withholding the report. Collateral damage follows the same axis. The Collateral-Damage Rate, the share of exploit-blocking successes achieved by breaking the service, reframes remediation as specification gaming; without the report it is higher for seven of the eight models. We release the benchmark, the two-component verifiers, the scripted validity agents, and the evaluation harness.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agentic evaluation, cybersecurity benchmark, vulnerability remediation, defensive security, service availability, specification gaming, negative side effects, collateral damage, vulnerability discovery, execution-based verification, safety of action

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4zStOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Current benchmarks measuring how able LLM agents are at fixing are vulnerable to reward hacking through incorrectly saying a patch is "correct" even when it breaks the system it's meant to be patching. This paper proposes a new benchmark which fixes that issue through verifying that the patch works and the system still works after the patch is applied.

Critical strengths
  1. The examples of reward hacking like the one shown with Qwen3.5-4B on page 13 show why this benchmark is useful, as models can appear to be patching vulnerabilities on the surface when, in reality, they are just breaking the system.
Critical weaknesses
  1. The authors' work may not be as novel as they claim. A claim in the paper's abstract is that no existing benchmark measures vulnerability patching while also measuring that the server continues to work. However, one of the cited papers, Zhang et al. 2025a, does measure the ability of LLMs to patch vulnerabilities and then runs test suites on those patched repos to ensure functionality is intact, which is a very similar idea.
Questions for authors

For 'discovery-required,' what guidance is the agent given? And, how do you ensure they have identified the vulnerability you were targeting vs. another one? Are you assuming there is exactly one vulnerability in the project?

1 comment

Thank you for the review and for the note on the reward-hacking example.

Novelty. BountyBench is our closest neighbour, and we agree it checks runtime health: it runs code and runtime invariants (server and database checks) before the exploit check. We credit this partially in Table 3 (Appendix). The distinction is the target and the fix. BountyBench edits source and re-instantiates the runtimes from the patched code. SysRepair-Bench grades the running system after operator-level changes (configuration, service, permission, end-of-life packages, vulnerabilities with no upstream patch), which a source patch cannot express. We will reword the abstract and introduction to "no existing benchmark measures operator-level remediation of a live system under an availability constraint", and name BountyBench explicitly as the nearest work.

Discovery-required. The agent receives only the running system and the availability requirement, with no disclosure text (§5).

Vulnerability. Each scenario is built around a single target vulnerability, and its container carries no other planted flaw. We grade only with that scenario's PoC, which replays the target vulnerability, and the security component passes only when it fails. We do not check whether the agent finds or fixes anything else. A scenario may bundle a few related settings that make up one finding, such as SSH X11 forwarding with no MaxAuthTries. We will state this grading unit explicitly in §5.

Reviewer_RhC6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SysRepair-Bench evaluates agents that remediate vulnerabilities by changing a running system's configuration, permissions, packages, or services. A successful repair must block the exploit and pass service regression checks. The benchmark contains 297 binary-scored scenarios and 16 free-roam hosts with partial-credit scoring. Eight models are evaluated with and without a vulnerability report, using three fresh episodes per scenario and up to five submissions per episode. The paper analyzes repair success, collateral damage, and the additional cost of discovering the fault.

Critical strengths
  1. The benchmark captures a practical failure mode that an exploit-only score would miss: disabling a vulnerable service can prevent the attack while making the system unusable. Operator-level changes on live containers and VMs are a useful evaluation target. The scripted reference repairs and negative controls strengthen the benchmark: the tested no-op, service-killing, and exact-PoC-blocking strategies receive zero joint passes. These checks give the joint objective concrete meaning.
  2. The paired report-informed and discovery-required conditions produce an informative result across a substantial task set. On the same hosts and budgets, joint pass@5 drops by 12.6–47.5 percentage points across the eight models, with a median drop of 37.4 points (Table 8). This shows how much performance depends on being told what is wrong. The free-roam track and reported execution costs provide useful complementary evidence about the difficulty of autonomous remediation.
Critical weaknesses
  • The claimed tamper resistance is stronger than the demonstrated protection of the grader. Section 7 describes an in-sandbox attack that exits successfully without a verdict, and fixes it by rejecting that outcome. This does not establish the integrity of a valid-looking verdict or of the executable dependencies producing it. Because agents modify the system being graded, this boundary matters directly to benchmark reliability. The limitation is acknowledged, but an independent regrade or a precise account of protected grader state is still needed.
  • The collateral-damage estimator is not sufficiently specified across attempts and episode pools. Equation 1 defines episode-level security and joint-pass means, whereas Table 10 analyzes failed episodes with both verdict components. It is unclear which attempt determines each component and how episodes without a complete verdict enter CDR. For example, Qwen3.5-4B's report-informed 72.2% joint pass over 891 episodes implies approximately 643 successes. Table 10 implies approximately 58 collateral-only failures. Combining those counts gives about 8.3% CDR, while Figure 2 reports 7.2%. A different aggregation could explain this, but the exact rule is needed to interpret the claimed collateral-damage trends.
Questions for authors
  1. What prevents the agent from modifying the verifier, its dependencies, or the output consumed by the scorer? Have final model-produced system states been regraded by a trusted evaluator outside the modified environment?
  2. Please specify the attempt-to-episode reduction for security and regression outcomes, and provide the joint-pass, collateral-only, and missing-verdict counts used for CDR. How do these counts reconcile Figure 2 with Tables 8 and 10? Do the report-informed versus discovery-required trends hold on a common episode pool?
  3. What information does the vulnerability report provide beyond identifying the affected component? If it includes mitigation guidance or exploit details, the comparison measures the effect of withholding that report under the fixed budget; interpreting it as a clean separation of perception and execution would need an additional control.
2 comments

We thank the reviewer for a careful and constructive review. We answer each point below, and list the resulting changes to the paper at the end.

Grader integrity (W1, Q1)

This is a fair point, and §7 understated our protections. What the agent can reach: the verifier is written to a random path only during grading and deleted immediately afterwards, so the agent never sees it. The agent receives only pass/fail. The scorer reads the two components from the verifier's structured summary on the host, not from its exit code alone. Verifier dependencies: the container is frozen before and after each verifier run. A host-side docker diff, which the agent cannot forge, lists every changed grading-critical binary, library and loader file. Each one is hashed from the host and accepted only if it matches the original image or a signature-verified package. Any file that changes during the verifier run is rejected. Agent processes: at boot the harness records every process and listening socket. Before each grading it suspends every process not in that record, then resumes them afterwards. Protected services, including ones the agent restarted, are not suspended, so the regression checks can still reach them.

Regrading outside the environment. Grading entirely outside a container is not possible, because many checks (processes, PAM and sudo, kernel parameters, privileges) observe live state that exists only inside the system. On a hypervisor, almost all grading could move outside the agent's environment. Disks would be read from the host, live state recovered by memory introspection, and only behaviour checks run, on a clone with a host-supplied kernel and tools. Our ad-vm track already grades from the host this way, and porting the container suites is future work. We will describe all of this in §7 and soften "tamper-resistant" to state exactly what is protected.

CDR estimator (W2, Q2)

We thank the reviewer. This question exposed an error in Figure 2, and we apologize for it. The reviewer's figure of 8.3% for Qwen3.5-4B RI is correct.

Reduction rule (added to §5). An episode ends at its first passing attempt, or after attempt k=5. Its s and r are those of the grading record that closes it. Since the grader is non-fail-fast, every submission reports both components. CDR = collateral-only (s=1, r=0) / security passes (s=1), pooled over the 891 episodes (297 × 3) of each cell. Episodes with no submit before the budget expired count as pass@k failures and cannot contribute to CDR. Episodes that errored before any verdict are excluded from both.

ModelCond.JointColl.-onlyn (Table 10)No-submitCDR (%)
DeepSeek-V4-FlashRI / DR850 / 6505 / 1427 / 21214 / 290.6 / 2.1
MiniMax-M3RI / DR801 / 52116 / 1251 / 14439 / 2262.0 / 2.3
MiniMax-M2.7RI / DR740 / 4218 / 1726 / 118125 / 3521.1 / 3.9
Qwen3.5-4BRI / DR643 / 22058 / 38192 / 48756 / 1848.3 / 14.7
Qwen3.5-9BRI / DR632 / 26622 / 24157 / 456102 / 1693.4 / 8.3
Qwen3.5-27BRI / DR715 / 36932 / 29108 / 37368 / 1494.3 / 7.3
Qwen3.5-35BRI / DR720 / 60826 / 599 / 12672 / 1573.5 / 0.8
Qwen3.5-122BRI / DR782 / 40526 / 1078 / 16331 / 3233.2 / 2.4

Reconciliation. These counts reproduce Tables 8 and 10. Figure 2 does not: it came from an earlier extraction that did not apply the closing-record rule, and it overstates CDR in most cells (e.g., MiniMax-M2.7 DR: 12.7% vs. 3.9%). We will regenerate Figure 2 and Table 4. This corrects one claim: discovery-required CDR exceeds report-informed CDR for six of eight models, not seven (Qwen3.5-35B and Qwen3.5-122B go the other way).

Common pool. pass@k and the discovery gap (median 37.4 points) use the same 891 episodes in both conditions. CDR does not depend on the no-submit episodes, because they contain no security passes. Table 10 covers failed-with-verdict episodes only. We will add a no-submit column (12–83% of failures), which reinforces that discovery-required failures are mostly non-remediation. We will also report CDR restricted to scenarios with a security pass in both conditions.

Vulnerability report (Q3)

The report supplies perception only: the CVE or CWE, the weakness, the affected service and its paths, and any scenario constraint. It does not name the vulnerable configuration, and it contains no fix and no exploit code. In both conditions the agent must choose and carry out the remediation itself. The discovery-required prompt gives only a generic instruction not to break the affected services. We will add an example report to the appendix so readers can verify this.

Changes to the paper: a full account of the grader protections in §7, with "tamper-resistant" softened; the reduction rule in §5; a regenerated Figure 2 and Table 4; "six of eight" in the abstract, §1 and §6.2–6.4; a no-submit column in Table 10 and a common-scenario CDR; the report contents and an example report in the appendix.

Reviewer_JGfMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SysRepair-Bench, a benchmark with 313 vulnerability-remediation scenarios adapted from existing security resources and public vulnerabilities. The evaluated agents modify live systems, and the authors built scripts to verify whether exploits are blocked and whether the service continues healthily. There are two difficulty levels: the agent can see the vulnerability report vs can't.

Critical strengths

A benchmark that evaluates workaround (not patching) of vulnerabilities seems novel and useful.

Critical weaknesses

The assumption that agent can't patch vulnerability seems off. Today's agents are capable, and can patch even vulnerabilities at binary level.

The eval scripts need to be very carefully written. One vulnerability can be exploited in many different ways. Blocking one POC doesn't mean the other exploits fail. Similarly, a service may run partially and few successful requests many not reveal degraded or compromised services. The fact that a no-op agent has 0.3% poc-blocking pass rate really concerns me.

Questions for authors

How did you validate your security and availability evaluators?

1 comment

Thank you for the review and for calling non-patch remediation novel and useful. Patching. We do not assume agents cannot patch. The agent has a shell on the live host, and any change that blocks the PoC and keeps the service healthy passes. The benchmark targets vulnerabilities whose remedy is an operator action rather than a code diff: misconfigurations, unsafe permissions, vulnerable package versions and end-of-life software. We will clarify this scope in the introduction.

One PoC versus other exploits. We test this directly (§7). An adversary that firewalls only the exact PoC signature passes security on 1 of 210 scenarios and joint passes none, so the checks do not rest on a single probe signature. We state the remaining scope in Limitation (5): a pass certifies that the disclosed weakness is closed and the probed service survives.

Partially running services. Negative probes are paired with a positive control, and a probe that cannot run counts as a failure. A down or half-started service therefore cannot pass. An adversary that disables the service passes security-only on 10% of scenarios and joint-passes none.

No-op security pass. The no-op joint-passes 0%. Its single security pass is rejected by the regression component, which is the case the two-component verdict exists to catch.

Validation. We ran these audits through the same grading path models use (§7):

  1. A reference solver joint-passes every scenario that ships a remediation.
  2. A no-op, a service-killer and a probe-blocker all score 0% joint.
  3. A grader-tampering adversary scores 0% joint against the fail-closed scorer.

We will summaries these results in §5 next to the verdict definition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.