acceptodds
Under review as a conference paper at ICLR 2027

NetMend: Can LLM Agents Repair Faults in Live Networks?

Abstract

As LLM agents move into infrastructure operations, rigorous evaluation of their network-troubleshooting ability becomes critical. However, most existing benchmarks evaluate diagnosis rather than repair. Others evaluate repair, but in simplified settings far removed from operational networks. We present NetMend, a benchmark that evaluates whether LLM agents can repair injected faults in live containerized networks. The agent starts from a symptom report with the injected fault hidden, and must localize and repair it using tools that run real operator commands. Correctness is judged from the state of the running network, rather than from the agent's own report. NetMend also scores whether the agent found the true root cause and whether its fix damaged unrelated parts of the network. Underlying NetMend is a task generator rather than a fixed dataset. It synthesizes networks across blueprints, routing protocols, and network operating systems. Each task is emitted as a replayable specification with a computed difficulty and a minimal reference solution. Beyond synthetic networks, NetMend also rebuilds two production topologies from real vendor configurations. NetMend currently comprises 29 fault classes and their compound combinations, instantiated on networks of configurable scale. Across five open-weight models, repair success ranges from 25.3% to 85.1%, and even successful agents take four to eight times as many tool calls as the minimal reference solution. Nearly one in five correctly diagnosed faults is still not repaired, while no repair succeeds without the correct root cause. Code is available at https://anonymous.4open.science/r/NetMend-0517.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, benchmark, network fault repair, network operations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F8SROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces NetMend, a benchmark for diagnosing and repairing injected faults in running containerized networks. A generator constructs network configurations, an injector creates replayable tasks, and a verifier evaluates repair using reachability, device state, and routing-table predicates. The agent interacts through a closed set of typed inspection and repair actions. The main evaluation contains 87 tasks covering 29 fault classes on three generated instances per class, with seven open-weight models and 609 episodes. Additional experiments vary the harness, budget, network size, and vendor, and examine production-derived topologies, compound faults, and symptom wording. Reported repair success ranges from 27.6% to 94.3%. The paper also reports localization, post-repair reachability, and tool-use efficiency relative to an idealized reference for six classes.

Critical strengths
  1. The combination of executable faults, replayable configurations, and fault-specific verification predicates is a substantive benchmark contribution. In particular, Section 3.3 recognizes that successful reachability alone can conceal an unrepaired routing adjacency, and therefore uses device-state and routing predicates where appropriate. This provides a stronger assessment of actual repair than judging an agent's final explanation alone.

  2. The fixed-model harness comparison isolates a substantial systems effect: GLM-4.7 ranges from 63.0% to 96.3% success across harnesses on the same task subset. The symptom-wording experiment, class-level uncertainty analysis, production-topology comparison, and compound-fault study further characterize what the benchmark measures. The paper also explicitly limits the reference ratio to six classes and describes the reference as idealized, rather than interpreting every excess call as avoidable waste.

Critical weaknesses
  1. The RCA metric does not establish correct diagnosis independently of repair. Section 3.3 grants a hit when the agent declares any equivalent root-cause node OR issues any repair action on such a node. The predicate does not require the correct fault type, interface, or rationale. Consequently, exploratory or incorrect modifications can count as successful localization, and successful repair can itself generate the RCA credit. Appendix F reports 33 purportedly localized failures with an incorrect action, interface, or value on the root-cause device. This does not invalidate the repair-success metric, but it weakens the headline interpretation that about one in nine correctly diagnosed faults cannot be repaired, and that only one in fifty repairs succeeds without correct diagnosis.

  2. The collateral-damage metric conflates residual injected damage with damage introduced by the agent. Appendix D explicitly describes prefix_list_missing repairs that restore the reported prefix but leave other prefixes blocked by the injected filter; these fail the collateral check. That demonstrates incomplete restoration relative to the healthy baseline, but does not by itself show that the repair made previously reachable destinations unreachable. Since Section 4.2 identifies filtering classes as the main source of collateral damage, this distinction affects a central safety conclusion. Moreover, the metric is conditional on successful repairs, so it is not an unconditional measure of agent-induced harm. The benchmark should distinguish full restoration, residual fault effects, and newly introduced regressions.

Questions for authors
  1. Can you separately report declaration-based diagnosis, requiring the correct fault type and sufficiently specific location, and action-based contact with a root-cause node? Under the declaration-based definition, how do the localization-versus-repair contingency counts change? This would determine whether the diagnostic conclusions survive the current metric's dependence on repair actions.

  2. For each flagged pair, can you distinguish reachability in the healthy baseline, immediately after injection, and after the agent acts? Please report how many failures are newly introduced versus inherited from the injected fault, including unsuccessful repair episodes where this comparison is possible. In particular, how many of the 16 flagged prefix_list_missing episodes represent new damage rather than incomplete restoration?

3 comments
AuthorsOpenReview ↗

Response to Reviewer F8SR(1/2)

Thank you for the careful review, and for your positive assessment of the executable faults with fault-specific verification predicates and of the supporting experiments. Your two weaknesses shaped the revision more than any other comment, and both of your questions could be answered with new measurements. We checked each point against our code and data and confirmed the issues you raised. Corrections that affect results across the paper are summarized in our general response. We respond to your weaknesses and questions in order below.

Weakness 1

Thank you, we agree. The predicate requires neither the fault type nor the interface. An exploratory write can therefore count as localization, and a successful repair can earn its own RCA credit. The revision uses separate measures on the 583 main-evaluation episodes that have a usable transcript. A declaration-based diagnosis requires every declared device to lie in the equivalence class. The strict version follows your definition and also requires the declared fault type and, where the ground truth names one, the interface. Action-based contact is reported on its own.

Under these measures, the two statements you single out do not hold as written.

Statement in the submissionSubmitted metricDeclaration-basedStrict (device, type, interface)
About one in nine localized faults is not repaired54/518 (10.4%)14/410 (3.4%)1/334 (0.3%)
About one in fifty repairs succeeds without localization0/46468/464 (14.7%)*131/464 (28.2%)

*57 of these 68 episodes end without any declaration.

We also rechecked the 33 failures in Appendix F.

What the agent did on the root-cause deviceEpisodes
Never used the repair action of the injected class (misdiagnosis)23
Right action, wrong interface or value6
Repaired the fault, then broke it again3
Task unsolvable because of a defect, since fixed1

Only 2 of the 33 declared the correct fault type. The revision removes the one-in-nine and one-in-fifty statements and reports the contingency table under Question 1 in their place.

Weakness 2

Thank you, we agree, and prefix_list_missing is the right example. With the revised benchmark, prefix_list_missing is repaired in 17 of 18 episodes, and all 17 pass the collateral check.

You also point out that the metric is conditional on repair. We replayed the recorded actions of all 583 main-evaluation episodes. We probed every baseline router pair in the healthy state, after injection, and after the agent's actions. A pair is restored if the agent fixed what the injection broke, residual if it stays broken, and newly damaged if it was reachable after injection but broken after the agent.

551 episodes whose replay matched the recordEpisodesWith residual pairsWith newly damaged pairs
Repaired4493 (110 pairs)0
Not repaired10242 (1,126 pairs)9 (149 pairs)

All nine episodes with new damage come from the two weakest models. The controls behaved as expected. Re-probing the healthy network lost no pair, reference repairs restored every pair, injection alone damaged no pair outside the fault, and deliberately damaging writes were flagged in all 12 cases. Configuration differences on modified devices are reported separately, because they do not change reachability.

The revision calls the metric a reachability-regression check and no longer uses it as evidence of safety. We revised Section 4.2 and the conclusion accordingly.

AuthorsOpenReview ↗

Response to Reviewer F8SR(2/2)

Question 1

Yes. On the 583 episodes, the declaration-based definition gives the following contingency table.

RepairedNot repaired
Diagnosis correct39614
Diagnosis not correct68 (57 without any declaration, 11 with only some declared devices in the class)105

Action-based contact, reported separately, occurs in 518 episodes, including all 464 repaired ones. The declared fault type is correct in 363 episodes. Under the strict definition, 334 episodes are correct on device, type, and interface, and 333 of them are repaired.

Agents usually declare the root cause after acting, and the prompt lists the 29 class names. A correct declaration may therefore partly describe the repair. We read these counts as descriptions and do not take them as evidence that diagnosis causes repair. The conclusion that survives is narrower than our submitted reading. A correct diagnosis is almost always followed by a repair. A sizeable share of repairs, however, comes without a correct declaration, mostly from models that stop without reporting.

Question 2

Yes. The three-state comparison under Weakness 2 covers every episode, including unsuccessful ones. No repaired episode introduces a new failure. Newly introduced failures (149 pairs in 9 episodes) occur only in unrepaired episodes. Failures inherited from the injection are far more common (1,236 pairs in 45 episodes).

For the 16 flagged prefix_list_missing episodes, we restored the submitted injection and the submitted prefix-list action. We replayed each episode's recorded writes in order and probed all baseline pairs after the injection and after the writes. In all 21 episodes of this class, the replayed injection, write results, and verdict matched the original record.

NetworkFlagged episodesFlagged pairs per episodeAlready broken after injectionNewly broken by the agentNot broken in the replay
campus-s15525200
campus-s26242400
campus-s34242400
campus-s3 (one more episode)11001
Total16501 pairs50001

In 15 of the 16 episodes, the agent re-added only the prefix named in the symptom, while the injected filter still blocked most other prefixes of that domain. The five episodes that were not flagged had re-attached the redistribution instead. The 16 flags therefore reflect incomplete restoration after an overly wide injection, not damage introduced by the agents.

Thank you again for the precise questions. They led directly to the three-state analysis and to the separate diagnosis measures, which are now central to the revised paper. We would be glad to discuss any further questions.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer F8SR,

Thank you again for your careful review. As the discussion period is coming to an end, we would like to check whether our response addresses your concerns. In brief:

  • Declaration-based diagnosis is now reported separately from action-based contact with the root-cause device, with a strict variant that also requires the correct fault type. A correct diagnosis is followed by a repair in 401 of 415 episodes (338 of 339 under the strict variant), and 73 of the 474 repairs come without an explicit diagnosis.
  • A three-state comparison probes every baseline router pair in the healthy state, after injection, and after the agent's actions. No repaired episode contains a newly damaged pair, residual pairs left by the injection appear in 4 repaired and 45 unrepaired episodes, and deliberately damaging writes are flagged in 12 of 12 control cases.

If any point remains open, we would be glad to provide further details or results while the discussion is open. If the response resolves your concerns, we would be grateful if you could take it into account in your final assessment.

Best regards, The Authors

Reviewer_pr8ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NetMend evaluates whether an LLM agent can repair a running containerized network. Networks are generated programmatically, and devices run network software such as FRR, VyOS, or SR Linux. A fault injector creates 29 single-fault classes and a small set of compound faults.

The agent receives network documentation and a symptom report, but not the fault location or fault type. It interacts through predefined inspection and repair tools, such as interface inspection, route-table inspection, neighbor-state inspection, and configuration repair. After the agent finishes, a verifier judges the actual network state instead of relying on the agent’s textual report.

The main evaluation contains 29 fault classes, three generated networks per class, and seven models, for 609 episodes. Repair success ranges from 27.6% to 94.3%. The paper also studies harnesses, tool budgets, network scale, device types, production-derived topologies, and compound faults.

Critical strengths
  1. The research question is practically relevant. The benchmark evaluates whether an agent can make changes to a running network.

  2. The evaluation pipeline is well structured. Network generation, fault injection, agent interaction, and state verification form a reproducible task workflow.

  3. The evaluation is multidimensional. The paper reports repair success, root-cause evidence, collateral effects, tool calls, and execution time.

Critical weaknesses

1 The paper counts an episode as a root-cause hit when either the agent names a device in the root-cause equivalence class or one of its repair actions touches such a device. The second condition prevents the benchmark from missing episodes in which the agent repaired the network but did not submit a final diagnosis. However, touching the correct device does not necessarily show that the agent understood the fault mechanism. The paper should report explicit-diagnosis hits, repair-action hits, and their overlap separately. This would distinguish identifying the cause from reaching the correct device during search.

2 The no-damage metric is evaluated on repaired episodes by re-probing paths associated with devices modified by the agent. This is useful for detecting incomplete recovery and some effects on other paths. However, a failed episode may reflect either an incompletely removed injected fault or new damage introduced by the repair. The paper should compare the healthy baseline, the injected state, and the repaired state to separate residual fault impact, newly introduced collateral damage, and configuration changes that the current probes cannot detect. The current results support target-specific recovery and reachability checks, but they do not directly establish the broader claim that the agent is safe to operate on a production network.

3 All 87 tasks in the main evaluation use FRR. The paper includes a controlled comparison across FRR, VyOS, and SR Linux, but that comparison covers only five link-layer fault classes and does not evaluate routing-layer differences across network operating systems. The production-derived transfer study covers 11 shared fault classes, five models, and 70 episodes. The paper should clearly distinguish what the generator supports, what the released task bank contains, and what was actually evaluated.

4 Each task is evaluated with one temperature-zero trajectory. The authors report variation across generated network instances and bootstrap intervals over fault classes, so the paper does include statistical analysis. However, it does not measure the stability of repeated runs on the same task. Model and harness differences should therefore be presented as descriptive results for the current task set and execution protocol.

Questions for authors
  1. Can you report explicit-diagnosis hits, repair-action hits, and their overlap separately?
  2. Can you use healthy-baseline, injected-state, and repaired-state comparisons to distinguish residual fault impact from new collateral damage?
  3. Can you provide the executed protocol by device system by fault class by topology by seed matrix, and identify which entries are generator capabilities only?
  4. Without additional repeated runs, can the model and harness comparisons be explicitly limited to descriptive results under the current protocol?
  5. Do the released artifacts contain container and image versions, seed manifests, task specifications, aggregate logs, model and harness versions, and authorization and de-identification information for the real configurations?
6 comments
AuthorsOpenReview ↗

Response to Reviewer pr8Z (1/5): Diagnosis and Repair Measured Separately

Thank you for the careful review and for the concrete questions. We checked every point you raised against our code and data, and confirmed each of the issues you identified. For several of these issues, the cause lay in our code and could not be seen from the paper. Investigating them led to the main changes described below. We now measure diagnosis and repair separately, distinguish residual from newly introduced damage across three network states, repeat runs on identical tasks, and compare routing-layer faults across operating systems. These changes have substantially improved the paper. We are revising the paper accordingly, and several of its numbers change. Your summary is accurate for the submitted version. We will respond to your weaknesses and questions in order below.

Weakness 1

Thank you for this point. We have verified it and agree with your assessment. The submitted metric gave the same credit to an agent that named the faulty device and to an agent that only wrote to it. We now report the two separately on the 583 main-evaluation episodes that have a usable transcript. An explicit-diagnosis hit requires every device the agent declares to lie in the equivalence class. A repair-action hit requires a successful write on such a device. We also report their overlap and a strict variant that additionally requires the correct fault type and interface.

The table below crosses the two measures with the repair outcome. Explicit-diagnosis hits occur in 410 episodes and repair-action hits in 518 (410 + 108). Every explicit hit also has an action hit, so both measures hit in the same 410 episodes, and no episode has an explicit hit alone. The 108 action-only episodes reach the root-cause device through actions alone. Most of them come from models that stop without a final report, and GPT-OSS-120B alone accounts for 48.

583 episodesEpisodesRepairedNot repaired
Both explicit diagnosis and repair action41039614
Repair action only1086840
Explicit diagnosis only000
Neither65065
Total583464119

The separation removes two statements from the submitted paper. The submitted metric counted the first three rows as localized. Under that metric, 54 of 518 localized episodes were not repaired (10.4%). The same metric credited every repair with localization, so no repair appeared to lack it. When localization instead requires an explicit-diagnosis hit, only 14 of 410 localized episodes were not repaired (3.4%), and 68 of the 464 repairs (14.7%) lacked one. Of these 68 repairs, 57 ended without any declaration. Under the strict variant, 334 episodes are correct on device, type, and interface. Only 1 of them was not repaired (0.3%), and 131 of the 464 repairs (28.2%) lack a strictly correct diagnosis.

Agents usually declare the root cause after acting, and the prompt lists the 29 class names, so a correct declaration alone does not show that diagnosis preceded repair. The transcripts give two direct measures of order. In 454 of the 462 repairs, the first write already lands on the root-cause device, and 379 repairs use a single write. In 333 of the 338 repairs whose transcripts contain reasoning text, the agent’s reasoning mentions the root-cause device before its first write. Repairs therefore usually follow localization rather than trial and error.

AuthorsOpenReview ↗

Response to Reviewer pr8Z (2/5): Residual Versus Newly Introduced Damage

Weakness 2

Thank you. This was the most valuable point for us, and our verification confirmed it. The submitted metric compared the post-repair state only with the healthy baseline. A pair that the injected fault broke and the agent left broken was therefore charged to the agent. Two defects made the filter-class result worse. The prefix-list repair action overwrote the previous entry instead of appending one, and the injection filtered far more prefixes than the symptom described. We fixed both.

To separate the effects, we replayed the recorded actions of all 583 main-evaluation episodes. We probed every baseline router pair in the healthy state, after injection, and after the agent's actions, for unrepaired episodes as well. A pair is restored if the agent fixed what the injection broke, residual if it stays broken, and newly damaged if it was reachable after injection but broken after the agent.

551 episodes whose replay matched the recordEpisodesWith residual pairsWith newly damaged pairs
Repaired4493 (110 pairs)0
Not repaired10242 (1,126 pairs)9 (149 pairs)

All nine episodes with new damage come from the two weakest models. The controls behaved as expected.

ControlResult
Re-probing the healthy networkno pair lost
Reference repairsevery pair restored
Injection aloneno damage outside the fault
Deliberately damaging writesflagged in 12 of 12 cases

Configuration changes that leave every probed pair reachable remain on the modified devices in 82% of episodes. We report them separately and do not count them as damage. We also list apart the 32 episodes whose replay differed from the original record, mostly because a write returned a different result. With the defects fixed, the filter-class effect disappears. prefix_list_missing is now repaired in 17 of 18 episodes, all without collateral damage. The probes cover router loopback pairs only, so the comparison carries no information on layer-2 and host-side classes. The revision states this limit.

We also agree with your point about the broader claim. These results concern targeted recovery and reachability in a lab network without production traffic or change control. They do not show that an agent is safe to operate a production network. We removed the sentence about trusting an agent with an operational network, and the revision calls the metric a reachability-regression check.

AuthorsOpenReview ↗

Response to Reviewer pr8Z (3/4): Coverage, Repeated Runs, and Harness Comparison

Weakness 3

Thank you for this suggestion. The revision follows it in two ways. First, we extended the routing-layer evaluation beyond FRR to VyOS and SR Linux, and the setup and results follow below. Second, the revision distinguishes the generator, the released task bank, and the evaluation throughout. The abstract, the contribution list, the coverage paragraph, and the conclusion now state which of the three each claim concerns. The matrix under Question 3 documents every combination.

For the routing-layer comparison, we first deployed 17 combinations of protocol and operating system, including RIP on FRR and VyOS and SRv6 on FRR. Thirteen of them reached full reachability between all router loopbacks, and the three RIP and SRv6 combinations are among them. The VyOS and SR Linux networks keep FRR in the core, so this step tests interoperation with FRR. We then implemented five routing-layer classes natively on VyOS and SR Linux. The classes are static_route_wrong, l3_loop, prefix_list_missing, route_policy_misconfig, and blackhole_route. Injection, inspection, and repair all go through each system's own configuration interface. Each episode is paired with an all-FRR twin, a network that is identical except that every router runs FRR (3 models, 2 seeds, 120 episodes).

NetworkPass@1Tool calls per episodeSeconds per episode
VyOS network (root cause on VyOS, native configuration)28/3019.7151
All-FRR twin of the VyOS network29/3017.4113
SR Linux network (root cause on SR Linux)29/3025.2237
All-FRR twin of the SR Linux network30/307.176

Outcomes barely differ. Only 2 pairs disagree, and both were solved only on the all-FRR twin. Cost differs substantially on SR Linux. VyOS uses FRR as its routing daemon, so the VyOS arm tests a different configuration interface rather than a different routing implementation. We also fixed a generator defect that left the host subnets missing behind VyOS leaves, and we reran the link-layer vendor comparison.

Link-layer classes, 3 modelsFRR segmentVyOS segmentSR Linux segment
Submitted24/3028/3022/28
After the generator fix29/3028/3022/30

The production-derived transfer study keeps its scope of 11 shared classes. The two production topologies are also part of the public release.

Reviewer_iCYVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper expose the fact that existing benchmarks for LLM agent in networking operations evaluate diagnosis (an agent report as suspected root cause, graded against a label) rather than repair. The few that do grade repair do so environments without real rounting software (NetArena/mininet) or without a running network at all (Cornetto, which verifies static configs formally). Netmend fills that gap. It is not a fixed dataset, but a task generator. A network generator synthesizes networks across blueprints, routing protocols, and network operating systems, and a fault injector turns each network into a task. Each task is emitted as a replayable specification that records how to rebuild the network and re-inject the fault. A computed difficulty accompanies each specification, and six fault classes carry a reference solution. Single faults can be composed into compound tasks with multiple independent root causes. The current release covers 29 fault classes, with network scale controlled by the generator. The agent receives design documentation and as a symptom report with the fault hidden, and acts through two typed tools ( 20 read actions, 23 write actions) that render into real operator commands. The verdict is read from the running network, reachability probes, device state, routing tables entries, plus a root cause equivalence check and a collateral damage reprobe of every baseline pair touching a modified device. Efficiency is scored against a per task reference solution for six of the 29 classes. Across seven open-weight models, repair success ranges from 27.6% to 94.3%, and even successful agents take three to five times as many tool calls as the reference solution. Secondary arms vary harness, tool budget, network scale (27-180 nodes), and vendor.

Critical strengths
  1. To the best of my knowledge the collateral damage check find something no report-based or probe-only evaluation can find, and the paper diagnoses it properly. Prefix-list_missing scores 100% pass@1 and 23.8% no damage. The mechanism worked out in Appendix D, is that the injected filter admits anly a few prefixes. An agent that re-adds the one prefix named in the symptom satisfies the target probe while leaving every other filtered destination unreachable, and only re-probing the pairs that traverse the repaired devices observe it. 15 of 21 passing episodes took that route.

  2. The dissociation between reporting and remediation in GPT-OSS-120B serves as a clear demonstration of the paper’s central methodological claim. The model issues a finalization statement in 8 out to 87 episodes. Consequently, based on an evaluation criterion derived from report, it shows a root-cause identification rate of approximately 9%, while the network status indicates a repair rate of 66.7%. Appendix F provides further clarification: when tasked solely with performing a diagnosis, the model fails to provide a response in 7 out of 9 instances, yet it offers a correct answer on the two occasions it does respond. A communication failure that an evaluation based exclusively on reports cannot distinguish from a failure in the diagnostic process itself. The approach adopted in the article is demonstrated using a specific model, with a particular measurement error of approximately 58 points.

  3. The validity engineering is careful and unusually transparent including a clean-injection gate;replayable task specifications;audited symptom wording, with the leakage effect measured; honest reporting of limitations, such as the RTT threshold being too tight and the difficulty score predicting outcomes only weakly. The harness ablation is also an important result for the scientific community.

Critical weaknesses
  1. The benchmark is close to saturated at the frontier, and the stated remedy is barely instantiated. 5 of 7 open- weight models exceed 85% and the paper repot that their boostrap intervals overlap. Consequently, on it main principal task, Netmend cannot separate the models it most needs to separate. The hard subset is offered as the answer, but it is 18 taks, and claude Opus 5.5 passes 17 of 18 with root cause named in all 18. The authors say so themselves: the frontier nearly saturates even this subset. The forwardlooking mitigation is compound tasks, the sample use for that is appropriately hedged, but it is thin ground on which to rest the claim that headroom exists.

  2. The closed 23-action repair interface make repair substantially narrower than the title and abstract imply, and the paper’s own numbers suggest the benchmark mostly measures localization. Table 4 maps the 23 write actions to classes nearly one-to-one. Once the agent has localized the fault, choosing the repair is often a lookup in a 23 item menu. Section 4.4 compounds this, Pass@1 is flat from 27 to 178 nodes. But, the authors clearly state that the execution of open ended command is not addreses, and closed action space is justified design choice for an initial version . My objection concern the discrepancy between this choice and the way the paper is framed. The paper would be more honest and in my view no les interesting if section 1 stated upfront that the candidate set is bounded by construction and that repair composition is deferred.

  3. The evaluation is small relative to the generality the paper claims, and the benchmark is close to saturation.

  4. the heterogeneity across vendors and protocols is presented as a core contribution, yet the main evaluation runs FRR only. The multi-vendor arm covers only five link-layer classes, and routing-layer faults on VyOS and SR Linux are explicitly untested.

  5. The results come from a single greedy sample per task. Bootstrap intervals span 10 to 28 points, and the five leading models are not separable; and most ablations use 27 tasks or 9 episodes per arm, and compound tasks are limited to two classes and 30 episodes.

Questions for authors
  1. Variance across repeated executions. All results correspond to pass@1 at temperature 0, with n=1. You report a variance between the three generated networks but this represent instance variance, not execution variance. Did you run a model twice on an identical task specification? What percentage of tasks yield different outcomes upon repetition? Without this information , it is impossible for me to determine whether the difference between DeepSeek-V4F (94.3%) and GLM-5.3(92.0%) is statistically meaningful. Furthermore, the fact that pairs of models diverge on 7% to 67% of individual tasks suggest that the noise level per task is significant.

  2. In the appendix H episode, the agent explicitly notes at R6 that the ping tool offers no packet-size option, so it never confirm the stated symptom as large packets failing. The standard ping passes and the episode is scored a pass. Does P for this class include a large-packet probe? If not, the device-state check is carrying the verdict alone, and the task instruction to confirm the symptom is gone by re-probing is not satisfiable with the tools provided. How many other classes have symptoms their probe set cannot express?

  3. How much of the task is repair? For how many of the 29 classes is the appropriate write action determined solely by the class itself, once device and interface are known? Table 4 suggest that this is the case for the majority of them. Providing this percentage would allow readers to directly assess the balance between localization and repair, an element that in my point of view would strengthen the paper rather than weaken it.

  4. The first paragraph in section 4.1 states that framework runs FRR on each device due to memory constraints, and that the vendor component covers only 5 link layer classes. Given that the heterogeneity of the generation is cited as a contribution, which results from the main evaluation might change if the networks were, in fact, multi-vendor? The 17.1 benchmark ration for GLM-4.7 in the VyOS segment (71 calls versus 16 in FRR, with 10 to 25 errors per episode) suggests that the vendor’s impact on cost can be significant, even when its impact on the outcome is nil. This warrants more than just a single paragraph in Appendix E.

  5. Regarding the remaining room for maneuver: in concrete terms, how many problems in the entire problem set remain unsolved by Claude Opus 5.5 when using claude code? If the number is small, it would be advisable to state this in paragraph 5, as it has a direct bearing on the ongoing relevance of this benchmark

  6. How much does the typed action menu drive performance? A free-form command arm on even a subset of classes, or an analysis of how often the action name alone identifies the fault class, would help separate reasoning ability from menu selection.

  7. In the Appendix H episode, the symptom says small packets get through. Yet the agent's first default-size ping from c1 to a1h1 shows 100% loss, presumably because the MTU change breaks the IS-IS adjacency rather than only dropping large frames. Is the symptom report accurate for this class? Since the ping tool cannot set packet size, can the agent ever directly verify the reported symptom? How many classes have a similar mismatch between symptom wording and actual network behavior?

  8. Can the author report routing-layer classes on VyOS and SR Linux for at least a subset, given that vendor heterogeneity is a headline contribution?

6 comments
AuthorsOpenReview ↗

Response to Reviewer iCYV (1/5):

Thank you for a very thorough review, and for your positive assessment of the collateral-damage analysis, the gap between reporting and repair, and the care taken in the validity engineering and the harness ablation. We checked each of your points against our code and data and confirmed the issues you raised. Several of your questions led to new experiments, which we describe below. Corrections that affect results across the paper are summarized in our general response. We respond to your weaknesses and questions in order below.

Weakness 1

Thank you, we agree that the main evaluation is close to saturation. After our fixes, the five leading open-weight models lie between 89.5% and 97.7% on the corrected main evaluation, and Claude Opus 5.5 solves all 87 tasks. To restore headroom, we built a harder task set with the same generator. It contains 39 tasks on eight networks not used elsewhere, including remote route hijacks, hard single-fault classes, and compound faults whose second component the symptom does not mention. Symptoms are user reports, no network inventory is given, and an episode succeeds only if the whole network returns to its designed state.

ModelMain evaluationHarder task setCompound faults in the harder set (15 tasks)
Claude Opus 5.587/87 (100%)31/39 (79.5%)7/15 (46.7%)
DeepSeek-V4-Pro78/86 (90.7%)26/39 (66.7%)10/15 (66.7%)
DeepSeek-V4F85/87 (97.7%)21/39 (53.8%)3/15 (20.0%)
GPT-6-Luna73/87 (83.9%)17/39 (43.6%)2/15 (13.3%)
Qwen3.5-397B77/86 (89.5%)14/38 (36.8%)1/15 (6.7%)
GLM-5.383/87 (95.4%)8/39 (20.5%)2/15 (13.3%)

The harder task set separates the models that the main evaluation cannot, with Pass@1 from 20.5% to 79.5%. Its 15 compound faults form the generator's top difficulty band, which is assigned before any run. On this subset, Claude Opus 5.5 solves 7 tasks, and no model solves more than 10. The revision rests the headroom claim on the harder task set and reports it as a separate result.

Weakness 2

Thank you, this objection is fair, and we changed the framing as you suggest. Section 1 now states at the outset that the action space is closed, that the candidate set is bounded by construction, and that repair composition is left to future work. We also measured how much the closed space matters. In a free-form command arm, one tool that runs vendor commands on a device replaces the typed write actions, on six classes, three networks, and three models.

ModelFree-form commands: Pass@1Typed actions: Pass@1Mean tool calls (free / typed)
DeepSeek-V4F18/1816/1832.9 / 56.9
GLM-4.717/1818/1830.5 / 25.6
Qwen3.5-397B16/1817/1823.4 / 19.7
Total51/5451/54

Removing the menu did not lower Pass@1. The agents issued 385 commands, 63 of which changed device state, and they wrote correct vendor commands. The scale arm uses six classes, none of which our fixes affected, so the flat Pass@1 from 27 to 178 nodes still holds as measured. We now read this result as you do. For these classes, the difficulty lies mainly in localization, which grows little with network size in our generator.

AuthorsOpenReview ↗

Response to Reviewer iCYV (2/5):

Weakness 3

Thank you, we agree, and the revision addresses both parts. First, it adds more than 1,800 agent episodes. The main additions are summarized below, and each is reported in full under the item listed.

Added experimentScaleMain resultDetails
Harder task set39 tasks, 6 modelsPass@1 from 20.5% to 79.5%Weakness 1
Free-form command arm6 classes, 3 models, 54 episodes51/54, as with typed actionsWeakness 2
Routing-layer faults on VyOS and SR Linux5 classes, 3 models, 120 episodes57/60, against 59/60 on all-FRR twinsWeakness 4
Harness comparison under equal conditions and two shell policies27 tasks, 4 harnessesorder changes between passesWeakness 5
Repeated runs5 models, up to 87 tasks4.6% to 18.6% of outcomes flipQuestion 1
Corrected MTU arm and symptom rerun18 and 153 episodes17/18; 45 to 50 of 51Question 2
Claude Opus 5.5 on all tasks87 tasks, native agent and Claude Code87/87 with bothQuestion 5

Second, the revision limits each claim to what was executed. An appendix matrix lists every executed combination of protocol, operating system, fault class, topology, and seed, and each result is reported with its sample size. Saturation is addressed under Weakness 1.

Weakness4

Thank you, we agree, and we added the evaluation you point to. We implemented five routing-layer classes natively on VyOS and SR Linux (static_route_wrong, l3_loop, prefix_list_missing, route_policy_misconfig, blackhole_route). Injection, inspection, and repair all go through each system's own configuration interface. Each episode is paired with an all-FRR twin, a network that is identical except that every router runs FRR (3 models, 2 seeds, 120 episodes).

NetworkPass@1Tool calls per episodeSeconds per episode
VyOS network (native configuration)28/3019.7151
All-FRR twin of the VyOS network29/3017.4113
SR Linux network29/3025.2237
All-FRR twin of the SR Linux network30/307.176

Outcomes barely differ, while cost rises substantially on SR Linux. We also fixed a generator defect that left the host subnets missing behind VyOS leaves, and we reran the link-layer vendor comparison.

Link-layer classes, 3 modelsFRR segmentVyOS segmentSR Linux segment
Submitted24/3028/3022/28
After the generator fix29/3028/3022/30

The revision reports these results as the evaluated part of the heterogeneity and separates them from what the generator implements.

Weakness 5

Thank you, we agree, and we measured this directly. We ran the tasks a second time with five models and counted the tasks whose outcome flipped between the two runs.

ModelTasksOutcomes that flipped
GLM-5.38716.1%
Qwen3-30B8718.4%
DeepSeek-V4-Pro2010%
DeepSeek-V4F874.6%
Qwen3.5-397B8618.6%

For GLM-5.3, every flip comes from a step in which the model spent its whole 16k output budget on reasoning and returned no action. On the 66 tasks where neither run hit this limit, the two runs agree on every task. Paired McNemar tests and bootstrap intervals cannot separate the leading models (Question 1). The revision therefore reports model comparisons as descriptive, describes arms with 27 tasks or 9 episodes as exploratory, and limits all comparisons to the current task set and protocol. The compound tasks grew from two to eight combinations. They remain a small sample, and we interpret them with care.

AuthorsOpenReview ↗

Response to Reviewer iCYV (3/5):

Question 1

Thank you for this question, which we now answer with direct measurements. We ran the tasks a second time with five models under identical task specifications. Between the two runs, 4.6% of outcomes flip for DeepSeek-V4F and 16.1% for GLM-5.3, and 4.6% to 18.6% across the five models (table under Weakness 5).

For your specific comparison, we added paired significance tests. DeepSeek-V4F (85/87) and GLM-5.3 (83/87) differ on 3 against 1 discordant tasks, with an exact McNemar p = 0.63 and a paired bootstrap interval of the difference from -2.3 to +6.9 points. Their difference therefore lies within run-to-run variation, and no pair of the five leading models differs after correction for multiple comparisons. The harder task set under Weakness 1 is where these models separate.

Question 2

No. The verdict for this class read only the interface MTU and included no large-packet probe. Our ping tool also had no size option, so the agent could not follow the instruction to confirm that the symptom was gone. We added a size option to the ping tool and a 1,472-byte probe to the verdict. We also moved the fault to the access port that faces the victim host, because on the uplink it broke the routing adjacency (see Question 7). In this arm, six models pass 17 of 18 episodes. All 18 episodes used sized pings, and all 17 successful ones re-sent a large ping after the repair.

For the other classes, we injected each of the 87 main tasks without an agent and measured the symptom from the device it names. In 11 of the 29 classes, at least one statement of the symptom fails on at least one network (Question 7 lists them). We also found that the agent's tools did not work on host nodes, because the host device type had no adapter. This defect affected 240 of the 584 main-evaluation episodes, mostly in layer-2 classes whose symptom is observed from a host. We fixed both problems and reran the 17 affected classes with three models.

Rerun with measured symptoms and working host tools (17 classes × 3 networks)Pass@1
DeepSeek-V4F50/51
Qwen3.5-397B45/51
GLM-5.345/51

On the 10 classes whose symptom changed, the rerun raises DeepSeek-V4F from 28 to 29 of 30 paired tasks, Qwen3.5-397B from 25 to 27 of 29, and GLM-5.3 from 21 to 26 of 30. The main-evaluation numbers are left unchanged, and the rerun is reported beside them.

Question 3

Thank you for this question, which the revision now answers directly. By our count, 17 of the 29 classes (59%) map one-to-one to a write action, and the action name nearly names the fault. The other 12 classes share six actions and differ only in their arguments. Of the 29 classes, 28 need a single action type, and static_route_wrong may need two. The prompt also lists the closed vocabulary of 29 class names. What remains is choosing the right device, interface, and value. We rechecked the 33 failures in Appendix F to see which part fails in practice.

What the agent did on the root-cause deviceEpisodes
Never used the repair action of the injected class (misdiagnosis)23
Right action, wrong interface or value6
Repaired the fault, then broke it again3
Task unsolvable because of a defect, since fixed1

Most failures are therefore misdiagnoses rather than wrong arguments. The balance therefore leans toward localization, as you suspected, and the revision says so.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.