- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses a relevant problem and reports promising numerical improvements. However, I have substantial concerns about its novelty, understanding of prior work in network automation, benchmark realism, and whether the evaluation demonstrates actual network task automation.
Critical strengths
- The paper recognizes that multiple workflows may satisfy the same intent, making exact-match evaluation insufficient.
- The authors compare multiple prompting, SFT, and RL baselines, provide component ablations, test several model backbones, and report confidence intervals.
Critical weaknesses
-
The related-work section overlooks important advances in LLM-driven network automation. Confucius (SIGCOMM 2025) already develops a production multi-agent framework for intent-driven network management, using DAG workflows, domain-specific tools, and validation. It reports over 60 deployed application. CEGS (NSDI 2025) uses LLMs for intent-driven network configuration synthesis and evaluates correctness on real networks. NetLLM (SIGCOMM 2024) examines domain adaptation and efficient fine-tuning for networking tasks. NetArena (ICLR 2026) provides dynamic, execution-based benchmarks for network automation agents, challenging the paper's broad benchmark novelty narrative. These omissions are particularly concerning because the paper claims to bring multi-agent intelligence into production networking without adequately distinguishing itself from established research.
-
The central methodological motivation is reinforcement learning from execution feedback. However, the reported success metric only checks capability selection and ordering constraints. Appendix H explicitly acknowledges that no live network execution is performed. Thus, the paper demonstrates agreement with a predefined validity model, rather than actual workflow executability or operational correctness. This is a substantial gap between the claimed contribution and the evidence.
-
The paper identifies capability ordering as the principal bottleneck. However, the ordering requirements are represented by a predefined precedence relation. Given a selected capability set, finding a valid order is a standard topological sorting problem. Why train an additional LLM-based workflower rather than use a deterministic graph algorithm? Without this baseline, the claimed benefit of multi-agent RL for ordering is unconvincing.
-
Multi-path keys are generated through LLM annotation and majority voting. The paper provides insufficient evidence that these keys correctly capture all valid alternatives or exclude invalid ones. Moreover, the purportedly independent LLM judge is given the reference workflow in its prompt. Human expert validation, annotator agreement, and analysis of judge errors are needed before treating the benchmark scores as reliable (Appendix D.3 and G).
-
All 7,901 intents use the same 12-capability catalog from one operator, with workflows containing at most seven capabilities. This is a highly constrained combinatorial problem compared with actual network management. Although other backbones and TaskBench are evaluated, neither of these demonstrates transfer to an unseen networking capability catalog, new dependencies, or different operational environments.
Questions for authors
-
How does MaNet differ technically from prior work from SIGCOMM, NSDI, ICLR as mentioned above?
-
Does any reported experiment execute the generated workflow against real services or a network emulator? If not, on what basis is the workflow considered executable?
-
How were the 7,901 intents created? How many originated from real network operators? Were the train/test splits constructed to prevent overlap in intent templates?
-
How does the core/optional/invalid labeling represent workflows that require one of several alternative capability combinations? How to make sure the correctness of these labels?
-
What happens when the capability catalog changes, previously unseen capabilities are introduced, or the dependency graph becomes substantially larger?
3 comments
Response to Reviewer hNFS (2/3)
W3. The learned workflower and the topological sort
A topological sort needs the precedence relation, and no system receives that relation at inference time. TaskBench follows the same official protocol, supplying the tool list with its type signatures while withholding the dependency graph. The relation reaches MaNet through the training reward alone, which is also the reward single-agent GRPO is trained against. Table R1 scores each method twice under these conditions, once in the order it produces and once with its selection reordered by a topological sort of the relation.
Table R1. Long-intent Succ with deterministic ordering.
| Method | Own order | Selection + topological sort |
|---|---|---|
| GPT-5.5 | 0.312 | 0.563 |
| Single-agent GRPO | 0.388 | 0.472 |
| MaNet | 0.650 | 0.672 |
The sort resolves the ordering for every method, and MaNet reaches about 97% of its score under oracle ordering without reading any rule, having taken the dependencies from the verifier during training. The sort cannot supply the selection it has to order, and MaNet still leads GPT-5.5 by 0.109 once all three are given oracle ordering.
A deterministic graph algorithm suffices for the ordering whenever the precedence relation is available as explicit rules, and the modular interface admits one in place of the workflower without retraining the other roles. The workflower answers the case the paper addresses, where the relation reaches the system only through training and where the ordering is optimized jointly with the selections it has to order.
Q3. Construction of intents and splits
Every intent originates from requests collected with the operator, across 48 operational scenarios of its service portfolio that span transport hubs, industrial sites, crowded venues and low-altitude inspection. A stratified sample of 500 was rated by domain experts, of which 91.2% were judged representative of the requests an operator receives.
A lexical gate removes every intent that names a capability or a catalog verb, which would otherwise reduce the task to copying, and leaves the capabilities to be inferred from the scene. Since natural operator requests rarely involve more than four capabilities, long intents draw on several interacting services within one scene, either by extending a reference chain or by combining the requirements of services that sit side by side.
The splits are separated at the level of templates rather than of individual intents. Intents sharing a scenario and a phrasing pattern go to the same split, so a pattern seen in training does not return in the test set carrying different capabilities. The overlap that remains is lexical:
Table R2. Overlap between test and training intents.
| Short test | Long test | |
|---|---|---|
| Test intents duplicated in training | 0 | 0 |
| Word overlap with the nearest training intent (Jaccard, mean / max) | 0.395 / 0.679 | 0.311 / 0.460 |
| Capability set unseen in training | 4.8% | 25.4% |
| Reference chain unseen in training | 6.6% | 41.4% |
| MaNet Succ on intents with an unseen capability set | 0.667 | 0.622 |
MaNet retains most of its performance on intents with a capability set absent from training, where a model leaning on recurring templates would lose it.
Response to Reviewer hNFS (3/3)
W4 & Q4. Multi-path answer key and judge
Representation of alternatives. The answer key is set-valued at the capability level, labelling each capability core, optional or invalid. A selection earns full credit once it covers the core set and stays within core ∪ optional. This form fits the alternatives NetBench actually contains, which arise mainly from auxiliary steps such as data collection, inference and rendering that an intent can take or leave without ceasing to be fulfilled. Intents requiring one of several combinations account for 3.1% of NetBench, and re-scoring them against the enumerated admissible combinations moves MaNet's long-intent Succ by 0.004 with all rankings intact.
Label correctness. Three frontier LLMs label each capability independently, and their labels are aggregated by majority vote. A capability outside the reference chain is admitted as optional only when all three annotators accept it. Agreement among the annotators is κ = 0.71. Domain experts then went over a stratified sample of 200 test intents independently of the key. Their labels agree with it on 94.3% of capabilities, and they uphold 88.4% of the predictions that earn full credit without matching the reference.
Judge. No LLM enters the primary metrics, since Sel and Succ follow from the key and the precedence relation alone, so the judge can be put to the test separately. We compare its verdicts with expert verdicts system by system, and add a judge denied the reference workflow as well as one from a model family not used in training.
Table R3. Judge verdicts compared with expert verdicts (long intents).
| Judge | Agreement (MaNet) | Agreement (GPT-5.5) | Agreement (single-agent GRPO) | False accepts / false rejects |
|---|---|---|---|---|
| Judge of the paper, with reference | 85.2% | 87.6% | 85.5% | 6.4% / 7.5% |
| Same judge, reference withheld | 83.0% | 84.1% | 81.0% | 9.3% / 8.0% |
| Judge from a different model family | 79.6% | 82.2% | 81.2% | 9.8% / 9.2% |
The judge's errors are balanced between false accepts and false rejects and are similar across systems, and all three judges yield the same ranking of the methods.
W5 & Q5. Catalog scope, catalog change and scale
The catalog is part of MaNet's input. A change to it calls for the new capability descriptions and precedence pairs, followed by an update of the affected adapters, and the full 280-step schedule takes about 1.3 hours on 8 GPUs.
Renaming the capabilities, paraphrasing their descriptions and adding distractors each cost MaNet at most 0.039 Succ on long intents. It stays ahead of the baselines even on intents that require a capability withheld during training, and replacing most of the precedence relation at test time leaves it with 94% of its score.
To evaluate a substantially larger dependency structure, we construct an extended catalog in which each capability is split into finer-grained variants. The catalog doubles to 24 capabilities, the precedence relation triples in size while remaining acyclic, and reference workflows extend to 11 capabilities.
Table R4. Succ on the extended catalog.
| Method | Original catalog (long intents) | Extended catalog (all intents) | Extended catalog (8–11 capabilities) |
|---|---|---|---|
| GPT-5.5 | 0.312 | 0.256 | 0.142 |
| Single-agent GRPO | 0.388 | 0.316 | 0.160 |
| MaNet | 0.650 | 0.597 | 0.490 |
On the extended catalog, MaNet's margin over both baselines widens on the longest workflows, consistent with the finding in Fig. 5 that decomposition becomes more valuable as the compositional depth of the intent grows. Even twelve capabilities under fixed dependencies place an ordered workflow among several million candidates, a space in which frontier models succeed on fewer than a third of long intents. We will grow the benchmark by assembling intents and catalogs across operators in future work.
We sincerely promise to incorporate the above experiments and discussions into our final version. We hope that these responses have addressed your concerns, and would greatly appreciate it if you could kindly raise the scores accordingly. Your support really means a great deal to us.
Response to Reviewer hNFS (1/3)
We thank the reviewer for the perspective of networking practice and for the references to related systems. We address each concern point by point with the supporting results.
W1 & Q1. Relation to Confucius, CEGS, NetLLM and NetArena
Confucius orchestrates management workflows over an operator's internal tools and has served over sixty applications in production. CEGS synthesizes device configurations by generalizing examples drawn from device documentation. NetLLM adapts a frozen pretrained model to individual networking tasks, and NetArena evaluates agents by executing their commands in emulators. In each case the workflow or the task is specified in advance, and the agent is measured on how well it carries that specification out.
MaNet addresses the step before this one, namely the choice of which capabilities a service catalog should contribute to an intent and in which order. This choice cannot be imitated from a single recorded workflow, because an intent is satisfied by several of them and their validity turns on dependencies that pretraining does not cover.
Correctness therefore has to be defined as a set of admissible workflows rather than as one target state, which is what NetBench provides and what MaNet is trained against. Within such a set an alternative receives the same credit as the recorded workflow, and each role can be rewarded on the part it produces.
Confucius is in this sense the premise of our work, and our results indicate that workflows now composed by prompting can be learned once correctness admits alternatives. CEGS and NetArena point the other way, towards devices and emulated infrastructure, where an execution layer of that kind would let a selected workflow be carried out rather than only verified. We will cite these works in Sec. 2 and position NetBench with respect to them.
W2 & Q2. Grounds for executability
Executability is defined by two requirements that come directly from the operator's catalog, and the verifier checks both of them explicitly.
Where the requirements come from. Neither the capability catalog nor the reference workflows were written for this benchmark. Both are taken from a service catalog in production use, together with the precedence relation that records how data moves through it, as when analysis consumes what sensing produces (App. D.1). A workflow is accepted only if it covers the capabilities the intent needs and places each capability after those producing its inputs.
How the metrics follow. Sel measures the first requirement and Succ adds the second, so a workflow scoring Succ = 1 satisfies both conditions under which the catalog would accept it. Neither metric depends on the state of the network at any particular moment, which is what keeps the evaluation reproducible for readers without access to an operator's infrastructure, and it is the level at which TaskBench scores tool graphs as well.
In future work we will extend the training loop to the digital twin that the catalog provides, which would carry the signal from acceptance through to execution.