- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ANT, a dataset of network traffic captured while LLM agents run tasks, together with annotations at three levels: whether the episode is malicious, which of five scenarios it belongs to, and what "behavior primitive" each traffic segment corresponds to. The authors run 20 tasks under five agent configurations, capturing 3,114 episodes, 276k flows and 104 GB with tcpdump, while logging model calls and tool events. Segments are cut at primitive start times from the session logs, labeled by DeepSeek-v4-pro from the log content, and the 3,295 resulting free-text labels are clustered into 47 groups with Jina embeddings. They then benchmark 13 existing encrypted-traffic and website-fingerprinting methods on the three tasks. And report the performance of baselines on the dataset.
Critical strengths
-
The dataset fills a real gap, and the collection is done properly from my perspective. Nothing comparable exists: traffic paired with session logs from real agent harnesses, across several agents and backends, with isolated namespaces, masked IPs, and attack execution kept inside Docker. The release includes the PCAPs, sanitized logs, and code. Also, it's valuable for scope of Traffic Analysis of Cybersecurity.
-
The analysis provides an opinion about where traffic analysis works and where it does not. Five methods detect web penetration almost perfectly and malicious code generation at 0.00, because the first has visible network actions and the second does not. The model-call-only ablation and the segment-clearing experiment are also sensible ways to show which part of the traffic carries the scenario signal.
Critical weaknesses
-
The split may let the models memorize task instances, and risk identification may be task identification. Episodes are split randomly within each task, and 3,114 episodes come from only 659 task instances, so the same prompt (and so the same web pages, same repos, same DNS names and response sizes) very likely appears in both train and test, possibly even with the same agent. IP masking does not remove that. The paper never reports a split by task instance. The same problem affects risk identification: "malicious" is just membership in three task families that are also in training, so a classifier can learn "this is the Juice Shop task" without learning anything about risk. The claim that methods "struggle to identify risk when malicious workflows resemble benign tasks" needs a held-out malicious task to be believable.
-
The primitive labels are LLM-generated and weakly clustered, so the 47-class results are hard to read. The clustering has a cosine silhouette of 0.157 and seed stability of 0.51 ARI, which means the group boundaries are unstable. The only check is a 2,000-sample spot check done "with AI assistance", with no human agreement number. When baselines reach 0.36–0.46 accuracy, I cannot tell how much is lost to label noise and how much to the traffic actually not carrying the signal.
Questions for authors
- Do train and test share task instances (same prompt, any agent)? Same prompt with the same agent? Report scenario and risk results under an instance-level split if possible.
- Which agent configurations ran the three misuse tasks? If they were not run on all five, could the risk classifier be learning the agent?
- For the primitive labels: how many of the 2,000 spot-checked samples were read by a human, and what was the disagreement rate?
- Why are only the eight website-fingerprinting baselines evaluated on primitive classification, and not the five encrypted-traffic classifiers?
- Excluding refusal-only episodes removes the cases where the model declined. Does that leave the misuse set biased toward agents that actually executed attacks, and does that affect which agents appear malicious?
- Here is a question from just curiosity: did you try leave-one-malicious-task-out for risk identification?
4 comments
-
On Agent Configurations for Misuse Tasks
We executed the misuse tasks on Claude Code + DeepSeek (Domestic/Overseas), Qwen, and Codex Agent. We actually attempted these tasks on the other agent configurations as well, but most were rejected due to their model backends' stricter safety checks. For ethical reasons and fair evaluation, we refrained from prompt jailbreaking and thus excluded their misuse traffic from the dataset.
We fully agree that this imbalance might introduce potential risks for the learning agent. Nevertheless, the experimental results show that the average FPR of MiMo, which did not execute misuse tasks, is actually higher than those of Qwen and Codex, which did execute them (0.57%, 0.226%, and 0.00%, respectively). This indicates that the model's judgment does not rely on agent shortcuts. Thank you for raising this key point. We will disscuss this in the revision.
Additionally, we supplemented risk identification experiments using only data from these four configurations. The results show that most methods achieved performance similar to the originally reported results. We will include these new experimental details and results in the revision.
-
On Baseline Selection for Primitive Classification
Thank you for pointing out this oversight. This selection was primarily driven by differences in prediction units and input adaptation. The input for primitive classification is a traffic segment within a behavioral time window. This window may truncate a long connection that spans multiple primitives. The eight website fingerprinting (WF) baselines can directly take the sequence of packet directions, timings, or lengths within the window as a single sample. In contrast, the five encrypted traffic classifiers used in our study take connections or flows as their input units. If we were to truncate the connections according to the primitive windows, we would need to redefine the input representations, statistical features, and multi-flow aggregation methods for those partial flows. Such redefinitions could artificially influence the evaluation results, undermining their objectivity. Therefore, this work selected the eight WF baselines that can directly utilize segment inputs. We will clarify this in the revision.
-
On the Selection Bias of Excluding Rejected Executions
We agree that excluding model-rejected cases leaves only the abuse instances that were actually executed by the agent. However, this aligns with our research objective "Agent Behavior Auditing". Since intercepted intents pose no substantial harm, our true aim is detecting whether internal users employ evasive tactics to successfully manipulate the agent into executing high-risk tasks.
In addition, the FPR results provided in the third response demonstrate that the model does not use the agent configuration bias induced by model rejection as a decision shortcut.
-
On Leave-One-Malicious-Task-Out Risk Identification
We did not perform leave-one-malicious-task-out risk identification because the three malicious task sets vary significantly in execution objectives, behaviors, and network interactions. For instance, web penetration involves direct network traffic to target services, while malicious code generation and sandbox evasion rely more on model calls and local execution. Therefore, excluding one would fundamentally distort the composition of malicious behaviors in the training set.
Thank you for the detailed responses and I appreciate the authors' efforts to address my concerns. Since my original score is positive, I will maintain my score.
Thank you for acknowledging our response and for your valuable comments. If there are any remaining ambiguities or further suggestions, we would greatly appreciate the opportunity for further discussion and refinement.
Thank you for acknowledging the insights provided by our analysis and providing constructive feedback.
On Instance-Level Isolation and Dataset Splits.
Our primary motivation for randomly splitting episodes within tasks was to increase instance coverage and capture diverse execution variances in the limited test set. We agree that evaluation under strict instance-level isolation provides a valuable perspective. Therefore, we have supplemented evaluations using a new split where all executions of the same underlying instance are kept together and have 128 malicious tasks in the test set. For each set of experiments, we trained using 3 different random seeds and calculated the mean accuracy, sample standard deviation, and bootstrap 95% confidence intervals (CI) on the test set.
Compared to the data reported in the original paper, the accuracy of most baselines in the new experiments remained consistent or slightly decreased within 5%, while the average recall across the three types of malicious tasks fluctuated within 10%. Crucially, aside from minor ranking changes among closely scoring baselines, the updated results uphold our core observations: high overall accuracy may mask missed risks, no baseline leads recall in all five scenarios for scenario recognition, and fine-grained primitive recognition remains imbalanced. We will include these new experimental details and results in the revision.
On the Manual Validation of Semantic Ground Truth.
Our manual spot-check process was conducted as follows: first, we generated primitive labels using an LLM, followed by macro group clustering. We then conducted three rounds of preliminary checks. In each round, we randomly sampled 200 examples to verify the semantic correspondence across the raw logs, primitive labels, and macro groups. If a primitive label failed to reflect the behavior in the raw log, it was manually corrected. If the primitive expression was inconsistent with its macro group assignment, we adjusted the expression based on the raw log and re-verified its cluster assignment. These initial checks directly participated in the iterative label-refinement process.
After the preceding iterations, we conducted a stratified audit separate from the refinement process. We selected 2,000 samples using an approximately balanced sampling scheme across the macro groups. Four independent raters from our lab divided the workload to audit different samples based on two explicit error criteria: "whether the raw logs support the primitive labels" and "whether the primitive labels match the macro group semantics." In this large-scale manual audit, 1,836 samples were judged as semantically consistent and 164 failed, yielding a pass rate of 91.80%, which met our internal quality control standard of >90%.
We will include this clarification in the revision and provide the manual review logs as supplementary material.