acceptodds
Under review as a conference paper at ICLR 2027

ANT: A Multi-Granularity Network Traffic Dataset and Benchmark for Agents Behavior Auditing

Abstract

The growing adoption of large language model (LLM) agents creates a need for network administrators and security teams to audit agent behavior within organizational networks without inspecting private user content. Network traffic offers an observable source of evidence, but how much it reveals about agent tasks and operations remains unclear. Existing traffic datasets lack the joint task and stage annotations needed to evaluate this question. We introduce ANT (Agent Network Traffic), a dataset providing agent behavior information at risk, scenario, and behavior primitive granularities alongside network traffic. ANT contains 3,114 execution episodes across 20 tasks and five scenarios, comprising 276,417 bidirectional flows and 40,049 behavior primitive segments organized into 47 macro groups. We establish a benchmark for agent risk identification, scenario recognition, and behavior primitive classification using 13 representative traffic analysis baselines. The results show that existing methods recover useful but uneven behavioral signals. They struggle to identify risk when malicious workflows resemble benign tasks and to distinguish scenarios with similar traffic patterns. Primitive classification is more reliable for frequent macro groups and those with distinctive traffic patterns than for rare or semantically similar groups. ANT provides a common basis for developing more precise auditing and forensic analysis of agent behavior from network traffic. Our data and code are available at https://anonymous.4open.science/r/ant-main-suite-7BC0/.

Primary area:
datasets and benchmarks
Keywords:
Agent Auditing, Traffic Analysis, Agent Misuse

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ka3BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ANT, a dataset of network traffic captured while LLM agents run tasks, together with annotations at three levels: whether the episode is malicious, which of five scenarios it belongs to, and what "behavior primitive" each traffic segment corresponds to. The authors run 20 tasks under five agent configurations, capturing 3,114 episodes, 276k flows and 104 GB with tcpdump, while logging model calls and tool events. Segments are cut at primitive start times from the session logs, labeled by DeepSeek-v4-pro from the log content, and the 3,295 resulting free-text labels are clustered into 47 groups with Jina embeddings. They then benchmark 13 existing encrypted-traffic and website-fingerprinting methods on the three tasks. And report the performance of baselines on the dataset.

Critical strengths
  1. The dataset fills a real gap, and the collection is done properly from my perspective. Nothing comparable exists: traffic paired with session logs from real agent harnesses, across several agents and backends, with isolated namespaces, masked IPs, and attack execution kept inside Docker. The release includes the PCAPs, sanitized logs, and code. Also, it's valuable for scope of Traffic Analysis of Cybersecurity.

  2. The analysis provides an opinion about where traffic analysis works and where it does not. Five methods detect web penetration almost perfectly and malicious code generation at 0.00, because the first has visible network actions and the second does not. The model-call-only ablation and the segment-clearing experiment are also sensible ways to show which part of the traffic carries the scenario signal.

Critical weaknesses
  1. The split may let the models memorize task instances, and risk identification may be task identification. Episodes are split randomly within each task, and 3,114 episodes come from only 659 task instances, so the same prompt (and so the same web pages, same repos, same DNS names and response sizes) very likely appears in both train and test, possibly even with the same agent. IP masking does not remove that. The paper never reports a split by task instance. The same problem affects risk identification: "malicious" is just membership in three task families that are also in training, so a classifier can learn "this is the Juice Shop task" without learning anything about risk. The claim that methods "struggle to identify risk when malicious workflows resemble benign tasks" needs a held-out malicious task to be believable.

  2. The primitive labels are LLM-generated and weakly clustered, so the 47-class results are hard to read. The clustering has a cosine silhouette of 0.157 and seed stability of 0.51 ARI, which means the group boundaries are unstable. The only check is a 2,000-sample spot check done "with AI assistance", with no human agreement number. When baselines reach 0.36–0.46 accuracy, I cannot tell how much is lost to label noise and how much to the traffic actually not carrying the signal.

Questions for authors
  1. Do train and test share task instances (same prompt, any agent)? Same prompt with the same agent? Report scenario and risk results under an instance-level split if possible.
  2. Which agent configurations ran the three misuse tasks? If they were not run on all five, could the risk classifier be learning the agent?
  3. For the primitive labels: how many of the 2,000 spot-checked samples were read by a human, and what was the disagreement rate?
  4. Why are only the eight website-fingerprinting baselines evaluated on primitive classification, and not the five encrypted-traffic classifiers?
  5. Excluding refusal-only episodes removes the cases where the model declined. Does that leave the misuse set biased toward agents that actually executed attacks, and does that affect which agents appear malicious?
  6. Here is a question from just curiosity: did you try leave-one-malicious-task-out for risk identification?
4 comments

Thank you for acknowledging the insights provided by our analysis and providing constructive feedback.

  1. On Instance-Level Isolation and Dataset Splits.

    Our primary motivation for randomly splitting episodes within tasks was to increase instance coverage and capture diverse execution variances in the limited test set. We agree that evaluation under strict instance-level isolation provides a valuable perspective. Therefore, we have supplemented evaluations using a new split where all executions of the same underlying instance are kept together and have 128 malicious tasks in the test set. For each set of experiments, we trained using 3 different random seeds and calculated the mean accuracy, sample standard deviation, and bootstrap 95% confidence intervals (CI) on the test set.

    Compared to the data reported in the original paper, the accuracy of most baselines in the new experiments remained consistent or slightly decreased within 5%, while the average recall across the three types of malicious tasks fluctuated within 10%. Crucially, aside from minor ranking changes among closely scoring baselines, the updated results uphold our core observations: high overall accuracy may mask missed risks, no baseline leads recall in all five scenarios for scenario recognition, and fine-grained primitive recognition remains imbalanced. We will include these new experimental details and results in the revision.

  2. On the Manual Validation of Semantic Ground Truth.

    Our manual spot-check process was conducted as follows: first, we generated primitive labels using an LLM, followed by macro group clustering. We then conducted three rounds of preliminary checks. In each round, we randomly sampled 200 examples to verify the semantic correspondence across the raw logs, primitive labels, and macro groups. If a primitive label failed to reflect the behavior in the raw log, it was manually corrected. If the primitive expression was inconsistent with its macro group assignment, we adjusted the expression based on the raw log and re-verified its cluster assignment. These initial checks directly participated in the iterative label-refinement process.

    After the preceding iterations, we conducted a stratified audit separate from the refinement process. We selected 2,000 samples using an approximately balanced sampling scheme across the macro groups. Four independent raters from our lab divided the workload to audit different samples based on two explicit error criteria: "whether the raw logs support the primitive labels" and "whether the primitive labels match the macro group semantics." In this large-scale manual audit, 1,836 samples were judged as semantically consistent and 164 failed, yielding a pass rate of 91.80%, which met our internal quality control standard of >90%.

    We will include this clarification in the revision and provide the manual review logs as supplementary material.

  1. On Agent Configurations for Misuse Tasks

    We executed the misuse tasks on Claude Code + DeepSeek (Domestic/Overseas), Qwen, and Codex Agent. We actually attempted these tasks on the other agent configurations as well, but most were rejected due to their model backends' stricter safety checks. For ethical reasons and fair evaluation, we refrained from prompt jailbreaking and thus excluded their misuse traffic from the dataset.

    We fully agree that this imbalance might introduce potential risks for the learning agent. Nevertheless, the experimental results show that the average FPR of MiMo, which did not execute misuse tasks, is actually higher than those of Qwen and Codex, which did execute them (0.57%, 0.226%, and 0.00%, respectively). This indicates that the model's judgment does not rely on agent shortcuts. Thank you for raising this key point. We will disscuss this in the revision.

    Additionally, we supplemented risk identification experiments using only data from these four configurations. The results show that most methods achieved performance similar to the originally reported results. We will include these new experimental details and results in the revision.

  2. On Baseline Selection for Primitive Classification

    Thank you for pointing out this oversight. This selection was primarily driven by differences in prediction units and input adaptation. The input for primitive classification is a traffic segment within a behavioral time window. This window may truncate a long connection that spans multiple primitives. The eight website fingerprinting (WF) baselines can directly take the sequence of packet directions, timings, or lengths within the window as a single sample. In contrast, the five encrypted traffic classifiers used in our study take connections or flows as their input units. If we were to truncate the connections according to the primitive windows, we would need to redefine the input representations, statistical features, and multi-flow aggregation methods for those partial flows. Such redefinitions could artificially influence the evaluation results, undermining their objectivity. Therefore, this work selected the eight WF baselines that can directly utilize segment inputs. We will clarify this in the revision.

  3. On the Selection Bias of Excluding Rejected Executions

    We agree that excluding model-rejected cases leaves only the abuse instances that were actually executed by the agent. However, this aligns with our research objective "Agent Behavior Auditing". Since intercepted intents pose no substantial harm, our true aim is detecting whether internal users employ evasive tactics to successfully manipulate the agent into executing high-risk tasks.

    In addition, the FPR results provided in the third response demonstrate that the model does not use the agent configuration bias induced by model rejection as a decision shortcut.

  4. On Leave-One-Malicious-Task-Out Risk Identification

    We did not perform leave-one-malicious-task-out risk identification because the three malicious task sets vary significantly in execution objectives, behaviors, and network interactions. For instance, web penetration involves direct network traffic to target services, while malicious code generation and sandbox evasion rely more on model calls and local execution. Therefore, excluding one would fundamentally distort the composition of malicious behaviors in the training set.

Thank you for the detailed responses and I appreciate the authors' efforts to address my concerns. Since my original score is positive, I will maintain my score.

Thank you for acknowledging our response and for your valuable comments. If there are any remaining ambiguities or further suggestions, we would greatly appreciate the opportunity for further discussion and refinement.

Reviewer_zLomOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a dataset and benchmark for auditing LLM agent behavior from encrypted network traffic. Their dataset size is substantial with 3,114 agent execution episodes across 20 tasks and 5 scenarios, 276,417 bidirectional flows and 40,049 behavior-primitive segments grouped into 47 macro groups. The data comes with labels at three levels: episode-level risk, task scenario, and local behavior primitive.

They benchmark 13 existing traffic-analysis methods on risk identification, scenario recognition, and behavior-primitive classification. Results show that network traffic contains useful behavioral signals, but performance varies substantially across tasks.

Critical strengths
  • The paper provides a substantial dataset specifically designed for auditing LLM-agent behavior from encrypted network traffic. Its multi-granularity annotations could enable research questions beyond existing datasets.

  • The benchmark is broad and informative. It covers 13 representative traffic-analysis methods across three levels of behavioral inference. Some interestign observations such as poor malicious-class recall for many methods.

Critical weaknesses
  1. Behavior primitive segments are constructed using boundaries obtained from agent session logs. In most passive network-auditing, these boundaries would not normally be observable.
  2. ANT deliberately isolates each task instance in an independent virtual network namespace to prevent interference. In practice, there are background traffic and the measured data on real deployments woould be a lot noisier than ANT.
  3. The benchmark has five predefined scenarios and 47 primitive macro groups. Again, in real world deployment, there would be much more unknown behaviors.
  4. ANT executes tasks on controlled cloud servers. That's useful, but cloud infrastructure remains cleaner than many enterprise endpoints. These could introduce browser traffic, authentication, enterprise SaaS, endpoint agents, background processes and policy infrastructure that are not currently in the dataset.
  5. The dataset is limited to single agent communications. There is no agent-to-agent traffic.
Questions for authors
  • How would primitive boundaries be identified from network traffic alone?
  • How do the reported results generalize to unseen agent/model configurations, tasks, and network environments?
  • Does the benchmark assume that agent flows have already been identified and isolated?
3 comments

Thank you for the constructive feedback.

  1. About Including Real-world Background Traffic

    We agree that in real deployments, background traffic and concurrent multi-instance executions increase the difficulty of auditing agent behavior. However, establishing a benchmark on a controlled dataset has irreplaceable advantages in precise labeling, high flexibility, and robust evaluation.

    First, isolating task related flows from noisy real-world traffic data is highly challenging. It involves further annotating traffic sources and separating the flows of individual instances, which requires labels that are difficult to obtain in real deployments. In contrast, independently collected task trajectories with explicit labels can serve as composable building blocks. Through time alignment, background traffic superposition, and traffic replay, researchers can construct evaluation scenarios with varying background noise intensities, instance counts, and degrees of overlap. Therefore, controlled data not only supports foundational capability comparisons but also provides flexible experimental materials for future robustness evaluations.

    This data organization methodology is well-established in public datasets. For instance, ISCXVPN2016, MIRAGE-COVID-CCMA-2022, and MIRAGE-App×Act-2024 all restricted non-target network access during collection to minimize background interference and facilitate precise labeling. Furthermore, constructing complex evaluation scenarios using isolated flows is a proven approach in the literature. Studies such as HyperVision (NDSS 2023), Whisper (CCS 2021), and REAPER (IEEE ToN 2026) commonly evaluate model robustness and detection performance by mixing and replaying isolated target flows alongside benign background traffic.

    Additionally, background traffic may introduce spurious correlations related to specific applications, host environments, or collection conditions. If mixed traffic is directly used as benchmark data, evaluated models might exploit these environmental shortcuts for classification rather than learning transferable task behaviors, thereby undermining the benchmark's validity.

    In summary, we believe that controlled task data is suitable as a foundational benchmark.

  2. About Closed-World Setting and Unseen Behaviors

    We admit that our benchmark assumes a closed-world setting. We would like to emphasize that, this work aims to provide a dataset for encrypted agent traffic auditing and benchmark existing baselines, rather than propose a novel solution. Since the evaluated algorithms are finite-class classifiers, our tests are conducted on predefined scenarios and actions to align with their original assumptions. Researchers can use ANT to develop open-world traffic recognition models by withholding specific classes during training and evaluating them at test time.

  3. About Identifying Primitive Boundaries

    Our primary motivation for using log-based boundaries to construct behavior primitive segments was to guarantee a fair and valid evaluation. Because traffic segmentation is challenging and not part of the evaluated baselines, using ground-truth boundaries avoids introducing heuristic errors that could skew the classification results.

    After submission, our further analysis revealed that logs sharing the same message.id correspond to a single request response in most cases. Thus, uplink/downlink directional switches of packets with application payloads can serve as generally effective segmentation points. To handle fragmented boundaries caused by concurrent model-call connections during parallel tool invocations, we introduce a "continuous transmission unsegmentable" rule to avoid over-segmentation. Full details of this practical algorithm will be provided in the revision. This traffic-only algorithm achieves a 90.07% byte attribution consistency with the paper's ground-truth segmentation.

  4. About Isolation of Agent Network Space

    During collection, we used network namespace isolation to guarantee that each episode's traffic solely consists of data generated or triggered by the agent's current task. Additionally, during preprocessing, we identified the agent's model-call traffic with the backend using only the SNI and IP addresses in the packets. This operation is performed without decrypting the payload or relying on log information.

  5. About Multi-agent Scenarios

    Our dataset is currently limited to single-agent communications. We leave the development of multi-agent communication datasets and benchmarking to future work.

Reviewer_zLomOpenReview ↗

The rebuttal didn't address my main concern of realism of the dataset

None of the issues around real datasets would be much noisier than the dataset in this paper were addressed. The responses focus primarily on why their dataset would be useful.

As my main concerns were not thoroughly addressed, I retain the original score and assessment.

Thank you for your further feedback, and we apologize that our previous response did not fully address your concern. We fully agree that real-world datasets are valuable, as they capture deployment conditions such as background traffic. What we hoped to convey is that controlled datasets offer different strengths, such as precise labels, high flexibility, and robust evaluation. We therefore regard the two types of datasets as complementary rather than competing. In the revision, we will add a comparison between ANT and real-world traffic datasets to the Discussion section.

Moreover, to the best of our knowledge, no comparable public dataset of agent network traffic is currently available, whether collected in controlled or real-world settings. We see ANT as a first step in this direction and hope it can lay the groundwork for future datasets, including real-world ones. We are also encouraged that ANT has been downloaded 11,618 times since its release about two months ago, which suggests considerable interest from the research community in this type of resource. We hope this clarifies our perspective, and we would be glad to discuss any remaining concerns.

Reviewer_zoE9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ANT is a dataset and benchmark for inferring LLM-agent activities from network traffic. It contains 3,114 execution episodes spanning 20 tasks and five scenarios, with 276,417 bidirectional flows and 40,049 annotated primitive segments. Execution logs provide temporal boundaries and semantic labels; embedding-based clustering consolidates 3,295 primitive labels into 47 macro groups. The benchmark evaluates risk identification, scenario recognition, and primitive classification using 13 traffic-analysis methods, with eight methods evaluated for primitive classification. The results show heterogeneous risk-detection performance, imperfect separation of related scenarios, and substantial difficulty with rare primitive groups. Traffic-scope and segment-clearing analyses probe the signals supporting scenario recognition, while Appendix C studies the scenario structure of the semantic annotations.

Critical strengths
  1. A useful paired resource at multiple semantic granularities. Collecting synchronized agent logs and packet traces across 20 tasks, five configurations, and two collection environments is a meaningful contribution. Episode-level risk/scenario labels and local primitive annotations make questions accessible that cannot be studied using coarse application labels alone. The clear collection protocol and declared release make the dataset potentially useful beyond the specific baseline results (Sections 3 and 4; Appendices A and B).

  2. The analysis goes beyond reporting one accuracy score. The paper reports malicious-class recall and false-positive rates, compares 13 methods, and examines traffic scopes and primitive groups. For example, Table 3 demonstrates that high overall risk accuracy can coexist with poor malicious recall; Figure 8 shows that model-call-only traffic can improve scenario recognition. These are consequential observations for designing more informative agent-auditing benchmarks.

Critical weaknesses
  1. The split does not establish clean generalization beyond repeated task instances, and the principal test set is small. The dataset contains 3,114 episodes but only 659 distinct task instances (p. 3, lines 151-153). The stated split is random over episodes within each task, with no specification that all executions of the same underlying instance are kept together (p. 6, lines 309-317; Appendix D.2). Keeping primitive segments from one episode together is appropriate, but does not resolve this separate instance-level concern. Thus, the manuscript does not establish whether repeated prompts/inputs can occur across training and testing. It also reports no held-out configuration or collection-environment evaluation. These omissions limit the interpretation of the scores as evidence of transferable behavioral inference. Moreover, the risk test set has only 20 malicious episodes, so a single example changes recall by five percentage points; the benchmark scores do not report training-seed variability or confidence intervals. I am not claiming that leakage occurred, but resolving the split ambiguity is important for the validity and usefulness of the benchmark.

  2. The semantic ground truth needs a more auditable validation protocol. Primitive labels are generated with an LLM and clustered into 47 groups. The reported manual spot check of 2,000 samples is valuable, but the manuscript does not describe its sampling scheme, independent raters, error criteria, or whether the audit examples were separate from the iterative label-refinement process (p. 5, lines 246-254; p. 9, lines 467-470). Appendix B.2 reports a silhouette of about 0.157 and seed stability of about 0.51 ARI (p. 19, lines 972-973). These values do not prove that the labels are wrong, but make external validation of the grouping especially important. Since the primitive taxonomy and fine-grained benchmark are central contributions, it is currently difficult to distinguish genuine traffic-classification difficulty from ambiguous or unstable target categories. An independent, stratified audit with operational definitions and representative ambiguous examples would materially strengthen the contribution.

Questions for authors

See the weakness part; please consider disclosing corresponding details.

3 comments

Thank you for your valuable suggestions.

  1. On Instance-Level Isolation and Dataset Splits.

    Our primary motivation for randomly splitting episodes within tasks was to increase instance coverage and capture diverse execution variances in the limited test set. We agree that evaluation under strict instance-level isolation provides a valuable perspective. Therefore, we have supplemented evaluations using a new split where all executions of the same underlying instance are kept together and have 128 malicious tasks in the test set. For each set of experiments, we trained using 3 different random seeds and calculated the mean accuracy, sample standard deviation, and bootstrap 95% confidence intervals (CI) on the test set.

    Compared to the data reported in the original paper, the accuracy of most baselines in the new experiments remained consistent or slightly decreased within 5%, while the average recall across the three types of malicious tasks fluctuated within 10%. Crucially, aside from minor ranking changes among closely scoring baselines, the updated results uphold our core observations: high overall accuracy may mask missed risks, no baseline leads recall in all five scenarios for scenario recognition, and fine-grained primitive recognition remains imbalanced. We will include these new experimental details and results in the revision.

  2. On the Manual Validation of Semantic Ground Truth.

    Our manual spot-check process was conducted as follows: first, we generated primitive labels using an LLM, followed by macro group clustering. We then conducted three rounds of preliminary checks. In each round, we randomly sampled 200 examples to verify the semantic correspondence across the raw logs, primitive labels, and macro groups. If a primitive label failed to reflect the behavior in the raw log, it was manually corrected. If the primitive expression was inconsistent with its macro group assignment, we adjusted the expression based on the raw log and re-verified its cluster assignment. These initial checks directly participated in the iterative label-refinement process.

    After the preceding iterations, we conducted a stratified audit separate from the refinement process. We selected 2,000 samples using an approximately balanced sampling scheme across the macro groups. Four independent raters from our lab divided the workload to audit different samples based on two explicit error criteria: "whether the raw logs support the primitive labels" and "whether the primitive labels match the macro group semantics." In this large-scale manual audit, 1,836 samples were judged as semantically consistent and 164 failed, yielding a pass rate of 91.80%, which met our internal quality control standard of >90%.

    Following your suggestion, we will clarify the manual review process in the revision and provide the raw manual review logs as supplementary material, explicitly detailing the sampling method, reviewer assignments, and judgment criteria.

Could the authors provide some relevant examples about Manual Validation to make the explanation easier to follow?

Thank you for your valuable feedback. Labeling and the manual verification of those labels are indeed highly complex processes that require a significant amount of work. Natural language classification is inherently challenging on its own, especially since the logs include various content types such as thinking, text, and tool_use.

During our manual review process, we noticed that DeepSeek sometimes tends to rely on the content of its "thinking" process to generate labels, while the actual tool invocation has less influence on its decision. This is reasonable in the vast majority of cases, as the thinking process guides the tool invocation, and the purpose of the tool call is usually reflected in the thinking. However, we also found that in some cases, the thinking process contains information spanning multiple stages, whereas the actual tool invocation only executes a subset of those stages. For example, in USA_cc_dpsk / A1522 / message 5877, the message was initially labeled as answer_synthesis by DeepSeek because the "thinking" term said "the challenge is solved. Now let me write the full penetration testing report with attack steps.\n\nLet me record the complete documentation.". However, the actual tool operation just updated the task status from in_progress to completed, without generating any report or invoking a file-writing tool. During our human verification process, this was corrected to task_status_update. Upon identifying this issue during our manual verification, we paid special attention to the label accuracy of these specific cases.

Furthermore, we found that the LLM occasionally confuses code_debugging with environment_inspection, as both often involve similar code-related actions. For instance, in USA_cc_dpsk / A3323 / message 53146, the agent executed commands such as "ls ../python", "which python3", and "python3 --version" to locate an available interpreter. It did not inspect or debug any actual business code. DeepSeek labeled this as code_debugging. During human verification, the label was corrected to environment_inspection. After identifying several such ambiguous cases, we adopted "whether actual code execution is involved" as the distinguishing criterion to manually separate and correct these two categories.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Remark

We sincerely thank the PC, AC, and the reviewers for their thoughtful engagement and valuable feedback.

In the discussion, we carefully addressed comments raised by the reviewers through clarifications, new experiments, and revisions to the manuscript. Below, we summarize what the reviewers appreciated about our work, how we addressed their main concerns, and the major changes made in the revision. We hope this summary will help the PC and AC better understand our work and the revisions we have made.

What reviewers appreciated about our work:

  1. ANT fills a real gap and provides a substantial, useful dataset for auditing LLM agent behavior from encrypted network traffic. (Reviewers Ka3B, zLom, zoE9)
  2. Its multi-granularity annotations enable research questions beyond existing datasets and coarse application labels. (Reviewers zoE9, zLom)
  3. The collection is done properly with a clear protocol, and the release includes PCAPs, sanitized logs, and code. (Reviewers Ka3B, zoE9)
  4. The benchmark is broad and informative, covering 13 representative methods at three levels of behavioral inference, and its analysis shows where traffic analysis works and where it does not. (Reviewers zLom, zoE9, Ka3B)

How we addressed reviewers' main concerns:

  1. Instance-Level Isolation and Result Uncertainty. Reviewers zoE9 and Ka3B noted that our random episode split may place executions of the same task instance in both training and test sets. We explained that our primary motivation for randomly splitting episodes was to increase instance coverage and capture diverse execution variances in the limited test set. To provide a stricter point of reference, we added risk identification and scenario recognition evaluations under instance-level splits that keep all executions of a task instance in the same set. The new risk test set contains 128 malicious episodes, and each baseline is trained with three random seeds, and we report means, standard deviations, and bootstrap 95% confidence intervals. Most baselines stay within 5% of their reported accuracy, the average recall across the three types of malicious tasks fluctuated within 10%, and our core observations still hold: high accuracy coexists with missed malicious tasks, and no baseline leads recall in all five scenarios.
  2. Auditable Validation of the Semantic Ground Truth. Reviewers zoE9 and Ka3B asked how the LLM-generated primitive labels and macro groups were validated. We clarified that the 2,000-sample check was a manual, stratified audit conducted separately from label refinement. After three rounds of refinement checks (200 random samples each), four lab members audited 2,000 samples drawn approximately evenly across macro groups, each handling a disjoint subset, against two explicit criteria: whether the raw log supports the primitive label, and whether the label matches its macro group. 1,836 samples (91.80%) passed, above our 90% quality threshold. At Reviewer zoE9's request, we also gave concrete examples of corrected labels.
  3. Realism of a Controlled Dataset. Reviewer zLom's main concern is that real deployments contain background traffic and are much noisier than ANT. We agree that real-world traffic is more complex and challenging. However, we believe that controlled dataset offer different strengths, such as precise labels, high flexibility, and robust evaluation. Its isolated, labeled episodes can be combined with background traffic through time alignment, superposition, and replay, as in HyperVision and Whisper, to build scenarios with varying noise intensities, instance counts, and degrees of overlap. We regard our dataset as complementary to real-world traffic datasets rather than a substitute for them.
  4. Primitive Boundaries from Traffic Alone. Reviewer zLom pointed out that primitive boundaries come from session logs, which a passive auditor cannot observe. We use log-derived boundaries because segmentation is not part of the evaluated baselines, and heuristic boundaries would add segmentation errors to the classification results. We also provide a traffic-only segmentation scheme based on uplink/downlink direction switches of payload-carrying packets on model-call connections, with a "continuous transmission unsegmentable" rule for parallel tool invocations. It achieves 90.07% byte attribution consistency with the log-derived segmentation.

(To be continue)

  1. Agent Configuration Imbalance in Misuse Tasks. Reviewer Ka3B asked whether the risk classifier could be learning the agent. The MiMo and Sonnet backends rejected most misuse tasks and we applied no jailbreaks, so misuse episodes come from the remaining four configuration-server combinations. The benign-only MiMo configuration yields a higher average FPR (0.57%) than Qwen (0.23%) and Codex (0.00%), both of which contribute misuse episodes, indicating that the classifiers' judgments do not rely on agent-specific shortcuts. We further trained and evaluated the baselines on only these four combinations, and most baselines perform comparably to the all-configuration setting.
  2. Scope of the Benchmark. (i) Closed-world setting (Reviewer zLom): ANT aims to benchmark existing finite-class classifiers rather than propose a new method, so we evaluate on predefined scenarios and primitive groups. Open-world recognition can be studied on ANT by withholding classes during training. (ii) Flow identification (Reviewer zLom): namespace isolation ensures that each episode contains only traffic generated or triggered by its task, and model-call traffic is identified from SNI and IP addresses alone, without decryption or logs. (iii) Baselines for primitive classification (Reviewer Ka3B): encrypted traffic classifiers require complete flows, which primitive windows may truncate, so we evaluate this objective with the eight website fingerprinting baselines. (iv) Multi-agent communication (Reviewer zLom) is left to future work.

Summary of Modifications in the Revision:

  1. We added instance-level split evaluations for risk identification and scenario recognition, with three random seeds, standard deviations, and bootstrap 95% confidence intervals (Sections 4.1–4.3 and Appendix E.1; Reviewers zoE9 and Ka3B).
  2. We described the manual validation process and its results in Section 3.2 and provide the audit records as supplementary material (Reviewers zoE9 and Ka3B).
  3. We explained the use of log-derived primitive boundaries in Section 3.2 and provided a traffic-only segmentation scheme in Appendix B.3 (Reviewer zLom).
  4. We discussed the configuration imbalance of misuse tasks in Section 4.2 and added a risk identification experiment on the four configuration-server combinations with misuse episodes in Appendix E.2 (Reviewer Ka3B).
  5. We explained in Section 4.1 why behavior primitive classification is evaluated with the website fingerprinting baselines (Reviewer Ka3B).
  6. We added a discussion in Section 5 on the gap between ANT and real-world traffic and on how ANT complements real-world datasets (Reviewer zLom).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.