- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces NetMend, a benchmark for diagnosing and repairing injected faults in running containerized networks. A generator constructs network configurations, an injector creates replayable tasks, and a verifier evaluates repair using reachability, device state, and routing-table predicates. The agent interacts through a closed set of typed inspection and repair actions. The main evaluation contains 87 tasks covering 29 fault classes on three generated instances per class, with seven open-weight models and 609 episodes. Additional experiments vary the harness, budget, network size, and vendor, and examine production-derived topologies, compound faults, and symptom wording. Reported repair success ranges from 27.6% to 94.3%. The paper also reports localization, post-repair reachability, and tool-use efficiency relative to an idealized reference for six classes.
Critical strengths
-
The combination of executable faults, replayable configurations, and fault-specific verification predicates is a substantive benchmark contribution. In particular, Section 3.3 recognizes that successful reachability alone can conceal an unrepaired routing adjacency, and therefore uses device-state and routing predicates where appropriate. This provides a stronger assessment of actual repair than judging an agent's final explanation alone.
-
The fixed-model harness comparison isolates a substantial systems effect: GLM-4.7 ranges from 63.0% to 96.3% success across harnesses on the same task subset. The symptom-wording experiment, class-level uncertainty analysis, production-topology comparison, and compound-fault study further characterize what the benchmark measures. The paper also explicitly limits the reference ratio to six classes and describes the reference as idealized, rather than interpreting every excess call as avoidable waste.
Critical weaknesses
-
The RCA metric does not establish correct diagnosis independently of repair. Section 3.3 grants a hit when the agent declares any equivalent root-cause node OR issues any repair action on such a node. The predicate does not require the correct fault type, interface, or rationale. Consequently, exploratory or incorrect modifications can count as successful localization, and successful repair can itself generate the RCA credit. Appendix F reports 33 purportedly localized failures with an incorrect action, interface, or value on the root-cause device. This does not invalidate the repair-success metric, but it weakens the headline interpretation that about one in nine correctly diagnosed faults cannot be repaired, and that only one in fifty repairs succeeds without correct diagnosis.
-
The collateral-damage metric conflates residual injected damage with damage introduced by the agent. Appendix D explicitly describes prefix_list_missing repairs that restore the reported prefix but leave other prefixes blocked by the injected filter; these fail the collateral check. That demonstrates incomplete restoration relative to the healthy baseline, but does not by itself show that the repair made previously reachable destinations unreachable. Since Section 4.2 identifies filtering classes as the main source of collateral damage, this distinction affects a central safety conclusion. Moreover, the metric is conditional on successful repairs, so it is not an unconditional measure of agent-induced harm. The benchmark should distinguish full restoration, residual fault effects, and newly introduced regressions.
Questions for authors
-
Can you separately report declaration-based diagnosis, requiring the correct fault type and sufficiently specific location, and action-based contact with a root-cause node? Under the declaration-based definition, how do the localization-versus-repair contingency counts change? This would determine whether the diagnostic conclusions survive the current metric's dependence on repair actions.
-
For each flagged pair, can you distinguish reachability in the healthy baseline, immediately after injection, and after the agent acts? Please report how many failures are newly introduced versus inherited from the injected fault, including unsuccessful repair episodes where this comparison is possible. In particular, how many of the 16 flagged prefix_list_missing episodes represent new damage rather than incomplete restoration?
3 comments
Response to Reviewer F8SR(2/2)
Question 1
Yes. On the 583 episodes, the declaration-based definition gives the following contingency table.
| Repaired | Not repaired | |
|---|---|---|
| Diagnosis correct | 396 | 14 |
| Diagnosis not correct | 68 (57 without any declaration, 11 with only some declared devices in the class) | 105 |
Action-based contact, reported separately, occurs in 518 episodes, including all 464 repaired ones. The declared fault type is correct in 363 episodes. Under the strict definition, 334 episodes are correct on device, type, and interface, and 333 of them are repaired.
Agents usually declare the root cause after acting, and the prompt lists the 29 class names. A correct declaration may therefore partly describe the repair. We read these counts as descriptions and do not take them as evidence that diagnosis causes repair. The conclusion that survives is narrower than our submitted reading. A correct diagnosis is almost always followed by a repair. A sizeable share of repairs, however, comes without a correct declaration, mostly from models that stop without reporting.
Question 2
Yes. The three-state comparison under Weakness 2 covers every episode, including unsuccessful ones. No repaired episode introduces a new failure. Newly introduced failures (149 pairs in 9 episodes) occur only in unrepaired episodes. Failures inherited from the injection are far more common (1,236 pairs in 45 episodes).
For the 16 flagged prefix_list_missing episodes, we restored the submitted injection and the submitted prefix-list action. We replayed each episode's recorded writes in order and probed all baseline pairs after the injection and after the writes. In all 21 episodes of this class, the replayed injection, write results, and verdict matched the original record.
| Network | Flagged episodes | Flagged pairs per episode | Already broken after injection | Newly broken by the agent | Not broken in the replay |
|---|---|---|---|---|---|
| campus-s1 | 5 | 52 | 52 | 0 | 0 |
| campus-s2 | 6 | 24 | 24 | 0 | 0 |
| campus-s3 | 4 | 24 | 24 | 0 | 0 |
| campus-s3 (one more episode) | 1 | 1 | 0 | 0 | 1 |
| Total | 16 | 501 pairs | 500 | 0 | 1 |
In 15 of the 16 episodes, the agent re-added only the prefix named in the symptom, while the injected filter still blocked most other prefixes of that domain. The five episodes that were not flagged had re-attached the redistribution instead. The 16 flags therefore reflect incomplete restoration after an overly wide injection, not damage introduced by the agents.
Thank you again for the precise questions. They led directly to the three-state analysis and to the separate diagnosis measures, which are now central to the revised paper. We would be glad to discuss any further questions.
Follow-up on our response
Dear Reviewer F8SR,
Thank you again for your careful review. As the discussion period is coming to an end, we would like to check whether our response addresses your concerns. In brief:
- Declaration-based diagnosis is now reported separately from action-based contact with the root-cause device, with a strict variant that also requires the correct fault type. A correct diagnosis is followed by a repair in 401 of 415 episodes (338 of 339 under the strict variant), and 73 of the 474 repairs come without an explicit diagnosis.
- A three-state comparison probes every baseline router pair in the healthy state, after injection, and after the agent's actions. No repaired episode contains a newly damaged pair, residual pairs left by the injection appear in 4 repaired and 45 unrepaired episodes, and deliberately damaging writes are flagged in 12 of 12 control cases.
If any point remains open, we would be glad to provide further details or results while the discussion is open. If the response resolves your concerns, we would be grateful if you could take it into account in your final assessment.
Best regards, The Authors
Response to Reviewer F8SR(1/2)
Thank you for the careful review, and for your positive assessment of the executable faults with fault-specific verification predicates and of the supporting experiments. Your two weaknesses shaped the revision more than any other comment, and both of your questions could be answered with new measurements. We checked each point against our code and data and confirmed the issues you raised. Corrections that affect results across the paper are summarized in our general response. We respond to your weaknesses and questions in order below.
Weakness 1
Thank you, we agree. The predicate requires neither the fault type nor the interface. An exploratory write can therefore count as localization, and a successful repair can earn its own RCA credit. The revision uses separate measures on the 583 main-evaluation episodes that have a usable transcript. A declaration-based diagnosis requires every declared device to lie in the equivalence class. The strict version follows your definition and also requires the declared fault type and, where the ground truth names one, the interface. Action-based contact is reported on its own.
Under these measures, the two statements you single out do not hold as written.
*57 of these 68 episodes end without any declaration.
We also rechecked the 33 failures in Appendix F.
Only 2 of the 33 declared the correct fault type. The revision removes the one-in-nine and one-in-fifty statements and reports the contingency table under Question 1 in their place.
Weakness 2
Thank you, we agree, and prefix_list_missing is the right example. With the revised benchmark, prefix_list_missing is repaired in 17 of 18 episodes, and all 17 pass the collateral check.
You also point out that the metric is conditional on repair. We replayed the recorded actions of all 583 main-evaluation episodes. We probed every baseline router pair in the healthy state, after injection, and after the agent's actions. A pair is restored if the agent fixed what the injection broke, residual if it stays broken, and newly damaged if it was reachable after injection but broken after the agent.
All nine episodes with new damage come from the two weakest models. The controls behaved as expected. Re-probing the healthy network lost no pair, reference repairs restored every pair, injection alone damaged no pair outside the fault, and deliberately damaging writes were flagged in all 12 cases. Configuration differences on modified devices are reported separately, because they do not change reachability.
The revision calls the metric a reachability-regression check and no longer uses it as evidence of safety. We revised Section 4.2 and the conclusion accordingly.