acceptodds
Under review as a conference paper at ICLR 2027

MaNet: Towards Network Task Automation with Multi-Agent Intelligence

Abstract

Large language agents have demonstrated remarkable versatility in personal computing scenarios and are increasingly expected to operate critical infrastructure. Network Task Automation (NTA), the focus of this paper, aims to compile a network operator's natural-language intent into an executable workflow over a proprietary capability catalog, which confronts a small on-premise model with heterogeneous decisions beyond the coverage of pretraining. Existing approaches typically rely on hand-crafted workflows or prompts over frozen models and thus struggle to incorporate execution feedback, while the lack of relevant public data also hinders further advancement. In response to these challenges, we present NetBench, the first NTA benchmark that pairs thousands of realistic operator intents with corresponding reference workflows over the network capability catalog. Beyond this dedicated dataset, we further introduce a multi-agent framework (MaNet) for NTA, which assigns the heterogeneous network decisions to specialized agent roles and jointly optimizes these roles via reinforcement learning from execution feedback. Empirically, MaNet exhibits consistent superiority on NetBench, as well as on the open-source benchmark TaskBench for general task automation. The analysis reveals that specialized agents improve steadily on complex intents where a shared policy plateaus, showcasing the effectiveness of multi-agent intelligence in NTA.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Large Language Agents, Multi-Agent Systems, Network Task Automation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hNFSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses a relevant problem and reports promising numerical improvements. However, I have substantial concerns about its novelty, understanding of prior work in network automation, benchmark realism, and whether the evaluation demonstrates actual network task automation.

Critical strengths
  1. The paper recognizes that multiple workflows may satisfy the same intent, making exact-match evaluation insufficient.
  2. The authors compare multiple prompting, SFT, and RL baselines, provide component ablations, test several model backbones, and report confidence intervals.
Critical weaknesses
  1. The related-work section overlooks important advances in LLM-driven network automation. Confucius (SIGCOMM 2025) already develops a production multi-agent framework for intent-driven network management, using DAG workflows, domain-specific tools, and validation. It reports over 60 deployed application. CEGS (NSDI 2025) uses LLMs for intent-driven network configuration synthesis and evaluates correctness on real networks. NetLLM (SIGCOMM 2024) examines domain adaptation and efficient fine-tuning for networking tasks. NetArena (ICLR 2026) provides dynamic, execution-based benchmarks for network automation agents, challenging the paper's broad benchmark novelty narrative. These omissions are particularly concerning because the paper claims to bring multi-agent intelligence into production networking without adequately distinguishing itself from established research.

  2. The central methodological motivation is reinforcement learning from execution feedback. However, the reported success metric only checks capability selection and ordering constraints. Appendix H explicitly acknowledges that no live network execution is performed. Thus, the paper demonstrates agreement with a predefined validity model, rather than actual workflow executability or operational correctness. This is a substantial gap between the claimed contribution and the evidence.

  3. The paper identifies capability ordering as the principal bottleneck. However, the ordering requirements are represented by a predefined precedence relation. Given a selected capability set, finding a valid order is a standard topological sorting problem. Why train an additional LLM-based workflower rather than use a deterministic graph algorithm? Without this baseline, the claimed benefit of multi-agent RL for ordering is unconvincing.

  4. Multi-path keys are generated through LLM annotation and majority voting. The paper provides insufficient evidence that these keys correctly capture all valid alternatives or exclude invalid ones. Moreover, the purportedly independent LLM judge is given the reference workflow in its prompt. Human expert validation, annotator agreement, and analysis of judge errors are needed before treating the benchmark scores as reliable (Appendix D.3 and G).

  5. All 7,901 intents use the same 12-capability catalog from one operator, with workflows containing at most seven capabilities. This is a highly constrained combinatorial problem compared with actual network management. Although other backbones and TaskBench are evaluated, neither of these demonstrates transfer to an unseen networking capability catalog, new dependencies, or different operational environments.

Questions for authors
  1. How does MaNet differ technically from prior work from SIGCOMM, NSDI, ICLR as mentioned above?

  2. Does any reported experiment execute the generated workflow against real services or a network emulator? If not, on what basis is the workflow considered executable?

  3. How were the 7,901 intents created? How many originated from real network operators? Were the train/test splits constructed to prevent overlap in intent templates?

  4. How does the core/optional/invalid labeling represent workflows that require one of several alternative capability combinations? How to make sure the correctness of these labels?

  5. What happens when the capability catalog changes, previously unseen capabilities are introduced, or the dependency graph becomes substantially larger?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hNFS (1/3)

We thank the reviewer for the perspective of networking practice and for the references to related systems. We address each concern point by point with the supporting results.

W1 & Q1. Relation to Confucius, CEGS, NetLLM and NetArena

Confucius orchestrates management workflows over an operator's internal tools and has served over sixty applications in production. CEGS synthesizes device configurations by generalizing examples drawn from device documentation. NetLLM adapts a frozen pretrained model to individual networking tasks, and NetArena evaluates agents by executing their commands in emulators. In each case the workflow or the task is specified in advance, and the agent is measured on how well it carries that specification out.

MaNet addresses the step before this one, namely the choice of which capabilities a service catalog should contribute to an intent and in which order. This choice cannot be imitated from a single recorded workflow, because an intent is satisfied by several of them and their validity turns on dependencies that pretraining does not cover.

Correctness therefore has to be defined as a set of admissible workflows rather than as one target state, which is what NetBench provides and what MaNet is trained against. Within such a set an alternative receives the same credit as the recorded workflow, and each role can be rewarded on the part it produces.

Confucius is in this sense the premise of our work, and our results indicate that workflows now composed by prompting can be learned once correctness admits alternatives. CEGS and NetArena point the other way, towards devices and emulated infrastructure, where an execution layer of that kind would let a selected workflow be carried out rather than only verified. We will cite these works in Sec. 2 and position NetBench with respect to them.

W2 & Q2. Grounds for executability

Executability is defined by two requirements that come directly from the operator's catalog, and the verifier checks both of them explicitly.

Where the requirements come from. Neither the capability catalog nor the reference workflows were written for this benchmark. Both are taken from a service catalog in production use, together with the precedence relation that records how data moves through it, as when analysis consumes what sensing produces (App. D.1). A workflow is accepted only if it covers the capabilities the intent needs and places each capability after those producing its inputs.

How the metrics follow. Sel measures the first requirement and Succ adds the second, so a workflow scoring Succ = 1 satisfies both conditions under which the catalog would accept it. Neither metric depends on the state of the network at any particular moment, which is what keeps the evaluation reproducible for readers without access to an operator's infrastructure, and it is the level at which TaskBench scores tool graphs as well.

In future work we will extend the training loop to the digital twin that the catalog provides, which would carry the signal from acceptance through to execution.

AuthorsOpenReview ↗

Response to Reviewer hNFS (2/3)

W3. The learned workflower and the topological sort

A topological sort needs the precedence relation, and no system receives that relation at inference time. TaskBench follows the same official protocol, supplying the tool list with its type signatures while withholding the dependency graph. The relation reaches MaNet through the training reward alone, which is also the reward single-agent GRPO is trained against. Table R1 scores each method twice under these conditions, once in the order it produces and once with its selection reordered by a topological sort of the relation.

Table R1. Long-intent Succ with deterministic ordering.

MethodOwn orderSelection + topological sort
GPT-5.50.3120.563
Single-agent GRPO0.3880.472
MaNet0.6500.672

The sort resolves the ordering for every method, and MaNet reaches about 97% of its score under oracle ordering without reading any rule, having taken the dependencies from the verifier during training. The sort cannot supply the selection it has to order, and MaNet still leads GPT-5.5 by 0.109 once all three are given oracle ordering.

A deterministic graph algorithm suffices for the ordering whenever the precedence relation is available as explicit rules, and the modular interface admits one in place of the workflower without retraining the other roles. The workflower answers the case the paper addresses, where the relation reaches the system only through training and where the ordering is optimized jointly with the selections it has to order.

Q3. Construction of intents and splits

Every intent originates from requests collected with the operator, across 48 operational scenarios of its service portfolio that span transport hubs, industrial sites, crowded venues and low-altitude inspection. A stratified sample of 500 was rated by domain experts, of which 91.2% were judged representative of the requests an operator receives.

A lexical gate removes every intent that names a capability or a catalog verb, which would otherwise reduce the task to copying, and leaves the capabilities to be inferred from the scene. Since natural operator requests rarely involve more than four capabilities, long intents draw on several interacting services within one scene, either by extending a reference chain or by combining the requirements of services that sit side by side.

The splits are separated at the level of templates rather than of individual intents. Intents sharing a scenario and a phrasing pattern go to the same split, so a pattern seen in training does not return in the test set carrying different capabilities. The overlap that remains is lexical:

Table R2. Overlap between test and training intents.

Short testLong test
Test intents duplicated in training00
Word overlap with the nearest training intent (Jaccard, mean / max)0.395 / 0.6790.311 / 0.460
Capability set unseen in training4.8%25.4%
Reference chain unseen in training6.6%41.4%
MaNet Succ on intents with an unseen capability set0.6670.622

MaNet retains most of its performance on intents with a capability set absent from training, where a model leaning on recurring templates would lose it.

AuthorsOpenReview ↗

Response to Reviewer hNFS (3/3)

W4 & Q4. Multi-path answer key and judge

Representation of alternatives. The answer key is set-valued at the capability level, labelling each capability core, optional or invalid. A selection earns full credit once it covers the core set and stays within core ∪ optional. This form fits the alternatives NetBench actually contains, which arise mainly from auxiliary steps such as data collection, inference and rendering that an intent can take or leave without ceasing to be fulfilled. Intents requiring one of several combinations account for 3.1% of NetBench, and re-scoring them against the enumerated admissible combinations moves MaNet's long-intent Succ by 0.004 with all rankings intact.

Label correctness. Three frontier LLMs label each capability independently, and their labels are aggregated by majority vote. A capability outside the reference chain is admitted as optional only when all three annotators accept it. Agreement among the annotators is κ = 0.71. Domain experts then went over a stratified sample of 200 test intents independently of the key. Their labels agree with it on 94.3% of capabilities, and they uphold 88.4% of the predictions that earn full credit without matching the reference.

Judge. No LLM enters the primary metrics, since Sel and Succ follow from the key and the precedence relation alone, so the judge can be put to the test separately. We compare its verdicts with expert verdicts system by system, and add a judge denied the reference workflow as well as one from a model family not used in training.

Table R3. Judge verdicts compared with expert verdicts (long intents).

JudgeAgreement (MaNet)Agreement (GPT-5.5)Agreement (single-agent GRPO)False accepts / false rejects
Judge of the paper, with reference85.2%87.6%85.5%6.4% / 7.5%
Same judge, reference withheld83.0%84.1%81.0%9.3% / 8.0%
Judge from a different model family79.6%82.2%81.2%9.8% / 9.2%

The judge's errors are balanced between false accepts and false rejects and are similar across systems, and all three judges yield the same ranking of the methods.

W5 & Q5. Catalog scope, catalog change and scale

The catalog is part of MaNet's input. A change to it calls for the new capability descriptions and precedence pairs, followed by an update of the affected adapters, and the full 280-step schedule takes about 1.3 hours on 8 GPUs.

Renaming the capabilities, paraphrasing their descriptions and adding distractors each cost MaNet at most 0.039 Succ on long intents. It stays ahead of the baselines even on intents that require a capability withheld during training, and replacing most of the precedence relation at test time leaves it with 94% of its score.

To evaluate a substantially larger dependency structure, we construct an extended catalog in which each capability is split into finer-grained variants. The catalog doubles to 24 capabilities, the precedence relation triples in size while remaining acyclic, and reference workflows extend to 11 capabilities.

Table R4. Succ on the extended catalog.

MethodOriginal catalog (long intents)Extended catalog (all intents)Extended catalog (8–11 capabilities)
GPT-5.50.3120.2560.142
Single-agent GRPO0.3880.3160.160
MaNet0.6500.5970.490

On the extended catalog, MaNet's margin over both baselines widens on the longest workflows, consistent with the finding in Fig. 5 that decomposition becomes more valuable as the compositional depth of the intent grows. Even twelve capabilities under fixed dependencies place an ordered workflow among several million candidates, a space in which frontier models succeed on fewer than a third of long intents. We will grow the benchmark by assembling intents and catalogs across operators in future work.

We sincerely promise to incorporate the above experiments and discussions into our final version. We hope that these responses have addressed your concerns, and would greatly appreciate it if you could kindly raise the scores accordingly. Your support really means a great deal to us.

Reviewer_DzcxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Network Task Automation (NTA), which turns an operator’s natural-language request into an ordered workflow over a closed catalog of 12 sensing and computing capabilities with fixed precedence constraints. It proposes NetBench, a benchmark of 7,901 intents whose reference workflows come with LLM-annotated answer keys that mark each capability as core, optional, or invalid, so that multiple valid workflows can be credited. It also contributes MaNet, a planner–assembler–workflower pipeline built on a shared Qwen3-4B backbone with per-role LoRA adapters. MaNet is trained jointly with role-branched GRPO, using rewards derived from the answer key and an LLM judge. MaNet outperforms all baselines on NetBench, including GPT-5.5 and Claude Opus 4.6, most clearly on long intents (Succ 0.650 vs. 0.312), and gains slightly on TaskBench. From this, the authors conclude that ordering capabilities, not selecting them, is the main bottleneck in NTA.

Critical strengths
  1. Multi-path evaluation is well motivated. Scoring against a set-valued key instead of a single reference chain (Sec. 3.3, Eq. 1) addresses a real weakness of benchmarks such as TaskBench.
  2. The ablations are thorough. Table 2 changes one component at a time across reward design, decomposition, parameter sharing, and training interface. The shared-adapter variant is especially informative: with a single adapter shared across roles, the decomposed system still reaches 0.566 Succ on long intents, against 0.388 for single-agent GRPO. Assuming the single agent uses an adapter of the same size, this suggests decomposition helps independently of having three adapters.
  3. Supporting evidence is reasonably broad. The paper includes paired-bootstrap CIs, three additional backbones, a sensitivity analysis (Fig. 3), and an error breakdown (Fig. 4), which go beyond a comparison with untrained prompting.
Critical weaknesses
  1. The claimed ordering bottleneck is not tested against a deterministic baseline, and the comparison with proprietary models gives the two sides unequal information. The precedence relation over the 12 capabilities is fixed, and the layered structure in App. D.1 implies it is acyclic. Given a selected set, a topological sort therefore satisfies every applicable constraint, making the ordering term in Eq. (1) equal to one without changing the selection. This matters because Fig. 4 attributes most GPT-5.5 failures to ordering, while Table 1 reports long-intent Succ of 0.312 for GPT-5.5 versus 0.650 for MaNet. The paper does not show how much of this gap would remain after deterministic ordering. Moreover, dependency information is omitted from the capability descriptions (Sec. 3), whereas MaNet’s reward uses the relation directly (Eq. 4), so the comparison cannot separate the benefit of multi-agent reasoning from access to task-specific knowledge. Please evaluate each method’s selections with a topological sort, give frontier models the explicit dependency rules, and compare against a trained single-agent selector followed by the same ordering. These comparisons would show what the learned decomposition contributes beyond enforcing dependencies.
  2. The SFT baselines select capabilities worse than the untrained models they start from, and the paper gives too few SFT-specific details to judge whether they are competitive. In Table 1, single-agent SFT lowers Sel from 0.763 to 0.683 on short intents and from 0.739 to 0.662 on long intents, relative to the untrained single agent. Per-role SFT shows a similar drop relative to the untrained pipeline (0.773 → 0.692 and 0.748 → 0.671), and judge scores fall as well (0.538 vs. 0.652 on short intents). The explanation that imitating a single path suppresses valid alternatives, does not account for this. App. B reports no SFT hyperparameters (epochs, learning rate, adapter configuration, checkpoint selection).
  3. Only one training run is reported. The bootstrap CIs in App. C capture variation from sampling test items, not from training, and no seeds are reported. Several ablation differences are small: γ = 0 changes short-intent Succ by 0.020, and full-parameter vs. LoRA by 0.006–0.008. On TaskBench, MaNet’s lead over per-role SFT is 0.001 on n-F1 and e-F1 and 0.014 on Succ, with no CI. Please report mean and standard deviation over at least three seeds for the main methods and the closest ablations.
Questions for authors
  1. The original TaskBench evaluates task decomposition, tool graphs, and parameter prediction, and DAG-structured requests make up the largest share of its data. MaNet outputs an ordered list of tool names without arguments. How is this list converted into dependency edges for e-F1 and the strict Succ, and how are DAG references scored?
  2. The shared-adapter variant in Table 2 has one-third of MaNet’s trainable parameters. Could you report a rank-48 shared adapter, and a single-agent GRPO baseline with the same budget, to test whether the 0.084 long-intent Succ gap in Table 4 reflects role specialization rather than limited capacity?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Dzcx (1/2)

We thank the reviewer for the detailed comments, and for noting that the shared-adapter ablation points to a benefit of decomposition beyond the number of adapters. We reply to each point below and include every comparison the review asks for.

W1. Deterministic ordering and access to dependency information

Inputs at inference time. MaNet and the proprietary models read the same inputs, namely the capability catalog with its descriptions and the intent, and the precedence relation reaches MaNet through the training reward alone. TaskBench follows the same official protocol, supplying the tool list with its type signatures while withholding the dependency graph. Single-agent GRPO is trained against the same reward as MaNet, so a comparison between the two holds access to the relation fixed. Under this condition decomposition lifts long-intent Succ from 0.388 to 0.650.

Requested comparisons. Table R1 scores every method four ways: with its own order, with its selection reordered by a topological sort of ≺, with the dependency rules stated in the prompt, and with both. It also includes a single-agent selector trained on the same selection reward and followed by the same sort.

Table R1. Long-intent Succ under deterministic ordering and explicit dependency rules.

MethodOwn order+ Topological sort+ Rules in prompt+ Rules and sort
GPT-5.50.3120.5630.4460.572
Claude Opus 4.60.2680.5300.4150.541
Single-agent GRPO0.3880.4720.4570.480
Single-agent selector–0.490––
MaNet0.6500.6720.6640.679

Three conclusions follow.

  1. Ordering accounts for most of the error of models without dependency knowledge. The topological sort lifts GPT-5.5 by 0.251, in line with the error analysis of Fig. 4.
  2. MaNet acquires the dependencies during training. With no rules available to it, MaNet reaches about 97% of its score under oracle ordering.
  3. Selection accuracy remains a separate advantage. Once oracle ordering is granted to every system, MaNet keeps 0.109 over GPT-5.5 and a wider margin over the trained selector. This residual gap is what decomposition and the multi-path reward contribute beyond enforcing the dependencies.

With the rules supplied and the sort applied, GPT-5.5 remains below MaNet evaluated without them. Where explicit rules are available, the modular interface admits a solver in place of the workflower, leaving the other roles untouched. We will add this analysis to Sec. 5.2.

W2. SFT baselines

Configuration. Both SFT variants use the backbone and rank-16 LoRA adapters of MaNet, a learning rate of 5 × 10⁻⁵ under cosine decay, three epochs over the training split, and checkpoint selection by validation Sel. We sweep the learning rate and the epoch count for per-role SFT in Table R2, and single-agent SFT follows the same trend.

Table R2. Hyperparameter sweep for per-role SFT (long intents).

Learning rateEpochsSelSucc
1 × 10⁻⁵10.6520.331
1 × 10⁻⁵30.6610.363
5 × 10⁻⁵10.6680.354
5 × 10⁻⁵30.6710.366
1 × 10⁻⁴30.6420.330

The reported configuration is the best in the sweep, and MaNet stands 0.284 Succ above it. On TaskBench, where each request admits one reference, this same pipeline is the strongest baseline of Table 1. The gap on NetBench thus follows from the multi-path setting rather than from the tuning.

Analysis of the selection score. Table R3 resolves Sel into its two factors, core recall and valid precision.

Table R3. Components of the selection score (core recall / valid precision, with Sel from Table 1).

MethodShort: recall / precisionShort SelLong: recall / precisionLong Sel
Single agent (untrained)0.848 / 0.9000.7630.839 / 0.8810.739
Single-agent SFT0.730 / 0.9350.6830.711 / 0.9310.662
Multi-agent pipeline (untrained)0.858 / 0.9010.7730.840 / 0.8910.748
Per-role SFT0.737 / 0.9390.6920.725 / 0.9250.671
MaNet0.908 / 0.9420.8550.888 / 0.9300.826

SFT gains precision at the expense of core recall, at both intent lengths. Reproducing the capability compositions that recur in the training references raises precision by about 0.035 and improves e-F1, since those compositions are usually valid. The same habit drops intent-specific core capabilities, costing about 0.12 of core recall, and the loss concentrates on test intents with a capability set absent from training. Fig. 4 attributes this error mode to per-role SFT, and a missing core capability also counts as critical for the judge, which is where its lower judge score originates. A reward that credits every admissible selection equally leaves no reason for such a trade, and MaNet raises both factors at once.

AuthorsOpenReview ↗

Response to Reviewer Dzcx (2/2)

W3. Variation across training runs

Table R4 reports the mean and standard deviation over three seeds for the main methods and the closest ablations.

Table R4. Succ over three training seeds (mean ± std).

MethodNetBench shortNetBench longTaskBench
Single-agent GRPO0.618 ± 0.0300.391 ± 0.0380.581 ± 0.035
MARFT0.609 ± 0.0220.498 ± 0.0240.611 ± 0.025
Per-role SFT0.451 ± 0.0120.362 ± 0.0110.660 ± 0.007
MaNet w/o selection bonus (γ = 0)0.669 ± 0.0230.628 ± 0.0190.662 ± 0.028
MaNet, full-parameter0.700 ± 0.0150.653 ± 0.0180.678 ± 0.020
MaNet0.693 ± 0.0130.646 ± 0.0160.674 ± 0.014

On NetBench the intervals of MaNet and of every baseline stay apart across seeds. Single-agent GRPO varies the most among them, as a single policy credits one reward to a workflow with errors arising at different points. The two narrower ablation differences hold as well. With the seeds paired between the two settings, the selection bonus raises short-intent Succ under each of them, by 0.024 on average, and the full-parameter variant stays within one standard deviation of MaNet. Role-specific adapters on a shared backbone thus match three independently trained models at a fraction of the memory cost.

On TaskBench the strict-Succ margin over per-role SFT excludes zero, with a paired-bootstrap 95% CI of [+0.003, +0.026]. Strict Succ requires every tool and every dependency of a request, and the margin grows with the structure of the request (Table R5).

Q1. TaskBench outputs and DAG scoring

The ordered list of capability names belongs to the NetBench interface alone. On TaskBench the three roles produce the full TaskBench output: the planner emits the steps, the assembler the tool nodes with their arguments, and the connector the dependency links as {source, target} pairs. A DAG request is therefore represented by its edge set directly, with no conversion from a sequence. Scoring follows the official TaskBench evaluation throughout, with e-F1 comparing predicted and reference edge sets, strict Succ requiring both the tool set and the edge set, and argument names and values measured by t-F1 and v-F1.

Table R5. TaskBench Succ by request structure.

Request structureRequestsPer-role SFTMaNet
Single node1870.9140.920
Chain2760.5390.554
DAG370.3070.356

The margin widens as requests acquire structure, and is largest on the DAG requests, where the connector has to predict an edge set with branches instead of a single chain.

Q2. Capacity-matched comparison

The single-agent GRPO baseline carries one rank-16 adapter, the size of the shared adapter in Table 2. Table R6 adds the two capacity-matched variants the reviewer suggests.

Table R6. Capacity-matched variants (long intents).

VariantTrainable parametersSelSucc
Single-agent GRPO, rank 161×0.7880.388
Single-agent GRPO, rank 483×0.7980.403
Shared adapter, rank 161×0.8030.566
Shared adapter, rank 483×0.8090.584
MaNet (per-role adapters, rank 16)3×0.8260.650

Tripling the capacity brings modest gains to the single agent and to the shared adapter alike, and at an equal parameter budget 0.066 of the original 0.084 gap to per-role adapters remains. What separates the variants is therefore role specialization rather than capacity, which agrees with the saturation of MaNet at rank 8 (Fig. 3b).

We sincerely promise to incorporate the above experiments and discussions into our final version. We hope that these responses have addressed your concerns, and your support really means a great deal to us.

Reviewer_xVqBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MaNet addresses network task automation, where an operator’s natural-language intent must be translated into an executable sequence of capabilities from a proprietary catalog. The paper introduces NetBench, a benchmark of 7,901 intents over 12 network capabilities, together with capability-level multi-path answer keys designed to credit multiple admissible workflows. It also proposes MaNet, a framework that divides the task among planner, assembler, and workflower agents and jointly trains them through group-relative reinforcement learning with role-specific rewards combining verifiable signals and LLM-judge feedback. Experiments on NetBench and TaskBench show that MaNet outperforms prompting-based methods, supervised fine-tuning, proprietary LLMs, and single-agent reinforcement learning on the reported metrics, with larger gains on longer workflows. Ablations indicate that multi-path rewards and role decomposition are important contributors to these gains. However, the evaluation uses a single operator’s 12-capability catalog with a fixed precedence relation, validates workflows against annotations and static ordering rules rather than through execution, and does not establish transfer to unseen catalogs or dependency structures.

Critical strengths
  1. A practically important task formulation and a purpose-built benchmark. NetBench formulates network task automation as the joint problem of selecting capabilities from a proprietary catalog and arranging them in a dependency-respecting execution order. Its capability-level multi-path key is designed to avoid penalizing workflows that differ from a single recorded reference while still satisfying the annotated intent. This is a meaningful improvement over strictly single-reference evaluation and makes an underexplored, deployment-relevant problem measurable.

  2. Strong and well-supported results on more complex workflows. On long NetBench intents, MaNet achieves an end-to-end success rate of 0.650, compared with 0.312 for GPT-5.5, 0.388 for single-agent GRPO, and 0.504 for MARFT. The corresponding reported improvements of +0.338, +0.262, and +0.146 have paired-bootstrap 95% confidence intervals that exclude zero. The ablations are also coherent: replacing the multi-path reward with a single-reference reward reduces long-intent success from 0.650 to 0.540, while removing role decomposition reduces it to 0.388. These comparisons support the central empirical finding that role decomposition becomes more useful as selection and ordering errors compound over longer workflows. In particular, the comparison with single-agent GRPO shows that the gain cannot be explained by the backbone or reward specification alone, although the paper does not fully control for trainable parameters or total training compute.

Critical weaknesses

While the paper presents promising results on complex network intents, I have three concerns that limit the strength of its contribution.

  1. The methodological novelty is insufficiently distinguished from recent multi-agent RL systems. The paper itself states in Appendix H that MaNet does not introduce a new learning algorithm. Stronger-MAS (Zhao et al., ICLR 2026) already develops agent- and turn-wise group-relative optimization, tree-structured sampling, and agent-wise credit assignment for role-structured multi-agent systems. MARTI (Zhang et al., ICLR 2026) provides centralized multi-agent interaction and reward allocation together with distributed policy training. MaNet contributes a task-specific factorization, role-level reward design, scheduled mixing of reference and predicted artifacts, and role-specific adapters, but the paper does not clearly isolate which of these constitutes a substantive methodological advance. Moreover, the experiments compare only with MARFT rather than the closest recent systems. The contribution therefore appears primarily application- and reward-design-specific unless a clearer distinction or matched comparison is provided.

  2. The evaluation provides limited evidence of structural generalization. All NetBench splits use the same twelve capabilities and the same fixed precedence relation. This relation is not exposed in the capability descriptions at inference time, so the current setup is compatible with learning a catalog-specific ordering rather than a general procedure for dependency reasoning. The long workflows contain only five to seven capabilities and are additionally constructed because naturally occurring requests rarely involve more than four. TaskBench demonstrates that MaNet can be trained successfully in another domain, but it does not establish transfer across catalogs because training and evaluation still occur within the benchmark’s predefined tool catalogs and dependency graphs. In contrast, WorFBench (Qiao et al., ICLR 2025) includes held-out workflow tasks, while NetArena (Zhou et al., ICLR 2026) dynamically generates network tasks and evaluates agent actions in network emulators. Evaluation on unseen catalogs, perturbed dependency graphs, or executable environments is needed to support broader claims about real-world applicability.

  3. The validity of the multi-path answer key is not established sufficiently. The proposed key does not enumerate multiple complete workflows. Instead, LLM annotators classify each capability as core, optional, or invalid, and any selection containing all core capabilities and no invalid ones receives full selection credit. The paper does not report the annotator models, number of annotators, inter-annotator agreement, or comprehensive expert validation. The reported 43% result concerns only cases in which a frontier model disagrees with the reference labels and therefore does not establish the accuracy of the key over the full benchmark. Moreover, independently optional capabilities need not remain valid under every joint combination, which the capability-wise representation cannot express. This is consequential because the same key defines both the principal training reward and the primary success metric. the reported LLM-judge metric is not fully independent of the training objective, since LLM-judge feedback is mixed into the training reward with weight 0.5.

Zhao, Y., et al. “Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs.” ICLR, 2026.

Zhang, K., et al. “MARTI: A Framework for Multi-Agent LLM Systems Reinforced Training and Inference.” ICLR, 2026.

Qiao, S., et al. “Benchmarking Agentic Workflow Generation.” ICLR, 2025.

Zhou, Y., et al. “NetArena: Dynamic Benchmarks for AI Agents in Network Automation.” ICLR, 2026.

Questions for authors
  1. Clarification request on methodological novelty. The paper states in Appendix H that MaNet is not a new learning algorithm. What, precisely, is the methodological contribution of its joint training beyond the task-specific factorization, role-wise group-relative updates, reward decomposition, and scheduled mixing of reference and predicted artifacts? In particular, how does the optimization differ from Stronger-MAS’s agent- and turn-wise grouping and tree sampling, and what capability is not already supported by MARTI’s centralized reward allocation and distributed policy training? A matched comparison would materially affect my assessment of the paper’s technical novelty.

  2. Essential evidence request on structural generalization. All NetBench splits share the same twelve capabilities and the same precedence relation, which is not exposed in the capability descriptions at inference time. What evidence shows that MaNet learns general dependency reasoning rather than a catalog-specific ordering? A stronger test would explicitly provide a dependency schema and evaluate on an unseen or perturbed catalog, or train across multiple catalogs and reserve one catalog for evaluation.

  3. Clarification request on answer-key validity. How many LLM annotators were used to construct the core, optional, and invalid capability sets, which models were used, and what agreement or expert-validation results were obtained? Can the authors also verify that arbitrary combinations of capabilities labeled optional remain jointly admissible, rather than validating each optional capability only in isolation? Because the answer key determines both the reward and the main success metric, stronger validation would materially affect my confidence in the reported results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xVqB (1/3)

We thank the reviewer for the careful reading, and for recognizing NetBench as a deployment-relevant formulation and the long-intent results as well supported. We reply below, merging a weakness with the question that follows it, and report the new experiments in full.

W1 & Q1. Methodological contribution relative to Stronger-MAS and MARTI

Stronger-MAS and MARTI advance the training of multi-agent systems with on-policy RL, the former through agent- and turn-wise grouping with tree-structured sampling, the latter through centralized rewarding with distributed policy training. Both frameworks assume an action space already familiar to the backbone and a reward that a single reference answer suffices to define. Neither assumption holds in network task automation, where the capabilities and the dependencies among them belong to one operator's catalog and an intent is fulfilled equally well by several workflows. Our contribution follows from these two conditions.

The setting. Compiling an intent stated in natural language into a workflow over a proprietary catalog leaves an agent no prior knowledge to fall back on, and the operational constraints narrow its options further. Privacy and latency confine it to a small on-premise model, while an erroneous workflow misconfigures a live network. The decisions it has to make are heterogeneous, from reading the intent to selecting capabilities and ordering them under the data flow between services.

The benchmark. Learning here requires correctness stated as a set rather than a path, and no public dataset offered one. NetBench pairs 7,901 operator intents with reference workflows over a production catalog, and equips each intent with a multi-path answer key that admits the alternatives the intent allows. The ordering constraints stay out of the capability descriptions, so an agent acquires them rather than reads them.

The framework. Once correctness takes this form, each role can be rewarded on the part of the workflow it produces, and an admissible alternative earns the same credit as the recorded reference. Table 2 identifies this as the decisive design choice: a single-reference reward lowers long-intent Succ from 0.650 to 0.540, the largest effect among the reward variants. Decomposition pays off along the same dimension, matching a single policy on intents of one or two capabilities and more than doubling its Succ at seven (Fig. 5).

MaNet thus introduces no alternative optimizer, as App. H states, and its roles can be trained within either framework above. Our contribution is the setting, a benchmark that renders it learnable, and the reward design these two call for. Table R1 re-implements both schemes on the same three roles, under the backbone, data and budget of MaNet.

Table R1. Matched comparison with recent multi-agent RL schemes (Succ).

MethodNetBench shortNetBench longTaskBench
MARFT0.6120.5040.614
Stronger-MAS0.6570.5600.633
MARTI0.6300.5290.623
MaNet0.6960.6500.676

MaNet leads both schemes, and by the widest margin on long intents, where selection errors and ordering errors compound and a role-level reward tells the two apart.

Parameters and compute. The trained baselines share the backbone, the 280-step schedule and the group size, and MaNet's per-step rollout costs about 1.4× single-agent GRPO. A single agent given three times the adapter capacity reaches 0.402 long-intent Succ, while MaNet saturates at rank 8 (Fig. 3b), which places the source of the gain in decomposition rather than in parameters or compute.

AuthorsOpenReview ↗

Response to Reviewer xVqB (2/3)

W2 & Q2. Structural generalization

Every role reads the catalog and the description of each capability from its input, so MaNet is conditioned on a catalog rather than tied to one. Following the reviewer's suggestions, we examine four axes of generalization: a held-out catalog, a dependency schema supplied as input and then perturbed, perturbed and withheld capabilities, and compositions longer than those seen in training.

Held-out catalogs. The three TaskBench domains carry different tool catalogs and dependency graphs. We train MaNet on two domains and evaluate on the third, so that neither the tools nor the dependency graph of the evaluated domain is seen during training.

Table R2. Leave-one-domain-out evaluation on TaskBench (n-F1 / e-F1 on the held-out domain).

Held-out domainUntrained pipelineSingle-agent GRPO (held out)MaNet (held out)MaNet (all domains)
HuggingFace0.631 / 0.3160.657 / 0.3530.738 / 0.4870.789 / 0.536
Multimedia0.788 / 0.4710.804 / 0.5080.867 / 0.6490.913 / 0.711
Daily-life APIs0.885 / 0.5960.893 / 0.6250.919 / 0.7390.938 / 0.787

With neither the tools nor the dependency graph of the held-out domain present in training, MaNet recovers about three quarters of the e-F1 gain it attains with in-domain data, and stays ahead of single-agent GRPO trained under the same protocol.

The dependency schema as input. The relation is withheld from every system by default, following the same official protocol under which TaskBench keeps its dependency graph out of the prompt. Here we supply it as a textual schema to all systems and train MaNet and single-agent GRPO with this input. At test time, 30% or 60% of the precedence pairs are replaced by randomly drawn pairs that keep the relation acyclic, and each workflow is scored against the perturbed relation.

Table R3. Long-intent Succ with the dependency schema as input.

MethodOriginal relation30% of pairs replaced60% of pairs replaced
GPT-5.50.4480.4350.414
Single-agent GRPO0.4740.4450.412
MaNet0.6800.6660.642

Under the heaviest perturbation MaNet keeps 94% of its Succ, whereas single-agent GRPO, trained on the same input, falls back to the level of the untrained proprietary model. MaNet therefore reads the dependencies from its input rather than reproducing an ordering fixed during training.

Perturbed and withheld capabilities. Table R4 renames the capabilities, paraphrases their descriptions, introduces distractors, and turns to intents that require a capability withheld from training.

Table R4. Long-intent Succ under catalog perturbations.

SettingGPT-5.5Single-agent GRPOMaNet
Original catalog0.3120.3880.650
Renamed capabilities0.2780.3480.611
Paraphrased descriptions0.3030.3640.633
+ 4 distractor capabilities0.2820.3500.627
Intents requiring a capability withheld from training0.3050.2900.577

MaNet leads under every perturbation, at a cost of at most 0.039 Succ. The withheld capability is what separates the two trained systems. A single policy gravitates towards the capabilities its training rewarded and gives up the advantage that training otherwise grants it, whereas MaNet continues to select from the descriptions it is given.

Unseen compositions and longer workflows. Table R5 turns to long intents with a capability combination absent from training, a quarter of the long test set, and to models trained only on intents of at most four capabilities.

Table R5. Long-intent Succ on unseen compositions and under short-only training.

SettingGPT-5.5Single-agent GRPOMaNet
All long intents0.3120.3880.650
Capability combination unseen in training0.2960.3500.622
Trained on intents with at most four capabilities0.3120.3060.598

Confining training to short intents costs single-agent GRPO 0.082 on long intents, enough to place it below the untrained proprietary model, while MaNet gives up 0.052 and still surpasses every baseline of Table 1 trained on the full data. These results indicate that capabilities learned on short workflows could compose into longer ones.

AuthorsOpenReview ↗

Response to Reviewer xVqB (3/3)

W3 & Q3. Validity of the multi-path answer key

Annotation. Three frontier LLMs from two model families label every capability as core, optional or invalid, given the intent, the catalog and one valid workflow. The aggregation rule is conservative, admitting a capability outside the reference chain as optional if all three annotators agree on it. The catalog and precedence relation underlying these labels were specified and verified by domain experts.

Validation. Beyond the cases adjudicated in App. D.3, we measure annotator agreement over all intents and audit a stratified sample of 200 test intents, covering every combination of optional capabilities that occurs in the predictions of the evaluated systems.

Table R6. Answer-key validation.

MeasureValue
Fleiss' κ among the three annotators (all intents)0.71
Expert agreement with the key, capability level94.3%
Expert agreement with the key, intent level71.5%
Valid predictions among those that differ from the reference but receive full credit88.4%
Jointly admissible combinations of optional capabilities92.7%
Intents requiring an either/or choice among capabilities3.1%

The key agrees with the experts on 94.3% of capability labels, and the experts confirm 88.4% of the predictions that earn full credit without matching the reference. Optional capabilities in NetBench are largely auxiliary steps such as data collection, inference and rendering, which is why most of their combinations remain jointly admissible. Strict either/or requirements arise in 3.1% of intents. Enumerating the admissible combinations for these and re-scoring against them moves MaNet's long-intent Succ by 0.004 and leaves every ranking intact.

The judge. Sel and Succ, the primary metrics, are computed from the key and the precedence relation without any LLM, and we will describe the judge in Sec. 3 as a complementary semantic check rather than an independent one. We further train a model on the verifiable reward alone (η = 1), which reaches 0.638 long-intent Succ, above every baseline in Table 1. All systems are then re-evaluated under two further judges, one denied the reference workflow in its prompt and one drawn from a model family absent from the training reward.

Table R7. Judge reliability on long intents.

JudgeAgreement with expertsMaNetGPT-5.5Single-agent GRPO
Judge of the paper, with reference86.1%0.7520.5760.602
Same judge, reference withheld82.7%0.7440.5620.584
Judge from a different model family81.0%0.7370.5660.598

Agreement with the experts declines as a judge is given less to work with, while the ordering of the systems does not. The ranking holds under all three judges, and a model trained without any judge reward stays ahead of the baselines in Table 1.

We sincerely promise to incorporate the above experiments and discussions into our final version. We hope that these responses have addressed your concerns, and your support really means a great deal to us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.