acceptodds
Under review as a conference paper at ICLR 2027

NetTrace: Evidence-Bounded LLM Reasoning for Network Traffic Investigation

Abstract

Tool-using language models can cite actual observations yet draw conclusions those observations do not support. In network investigation, support depends on the evidence set and a claim’s entity, time, traffic layer, quantifier, and strength. We formulate evidence-bounded traffic investigation and introduce NETTRACE, combining structured traffic inputs, claim–evidence supervision, and a separately trained scope-aware verifier with conflict checks and revision. On NetTraceBench’s 4,800-case test split, NetTrace achieves 98.96 macro-F1 and a 2.84% non-entailing-link rate in the supplementary batch. A same-scale text verifier with matched deterministic scope checks achieves 86.86 and 5.62%, respectively. The F1 difference is 12.10 points; separate controls show advantages under fixed observations and matched answer coverage. Counterfactual pairs, jointevidence tests, stricter splits, and an independent human audit examine the support mechanism under controlled interventions and distribution shifts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Network Traffic Investigation;Evidence-Bounded Reasoning;Large Language Models (LLMs); Structured Cyber States;LLM Agents and Tool Use;Provenance-Aware Evidence Memory;Evidence Verification and Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EBF7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NetTrace, a framework for network traffic investigation that produces evidence-bounded reports. It introduces (1) a Structure-Preserving Flow Language (SPFL) that serializes traffic into six complementary views; (2) a formal evidence-bounded release criterion requiring each substantive claim to be supported by a legal evidence set at its stated scope, with unresolved conflicts blocking release; and (3) a two-stage policy + verifier system (Qwen2.5-7B with separate LoRA adapters) performing an observe–plan–act–verify loop with eight security tools and a four-way support label. Evaluation on a new 24,000-case NetTraceBench (Host, C2, Exfil, Recon) reports 98.96 macro-F1 and 2.84% NELR on the supplementary batch.

Critical strengths
  1. A meaningful formulation of traffic-scoped evidence support. The paper goes beyond citation existence by explicitly modeling whether an evidence set jointly supports a claim under entity, time, traffic layer, quantifier, and conclusion-strength constraints, while accounting for unresolved relevant conflicts. Integrating this support relation into investigation and claim release addresses an important reliability issue in LLM-based network security investigation and constitutes a substantive contribution.

  2. Targeted mechanism-oriented evaluation. The paper uses matched comparisons to probe the contribution of explicit support modeling. Under fixed observations, NetTrace-S improves macro-F1 from 85.49 to 88.53 and reduces NELR from 5.96% to 3.32%, weakening an explanation based solely on additional evidence acquisition. Counterfactual and joint-evidence tests probe the claimed mechanism, while a blinded human audit reports fewer insufficiently supported citation sets (41/1,161 versus 77/1,159). These findings strengthen the case for the method's effectiveness.

Critical weaknesses
  1. External validity is not established: The central claim, evidence-bounded traffic investigation, rests on a benchmark that is substantially synthetic or controlled. Exfil and Recon use controlled executions with authorization records, execution receipts, and event-order references that would not exist in real deployments, and there is no evaluation on real enterprise or ISP traffic with analyst-verified ground truth. For a paper whose contribution is an operational investigation framework, this is the most consequential limitation and is understated in the main text.

  2. No comparison to any external system: All baselines (ReAct, Memory, Provenance, TextVerifier, TextScope, IndependentReAct) are internal variants constructed by the authors. There is no comparison to a published traffic-analysis or LLM-investigation system (e.g., TrafficLLM, which the paper cites). Internal controls are valuable for attribution, but they cannot establish that NetTrace improves on the state of the art.

  3. Confusing two-batch presentation: Key numbers are split across an "initial batch" and a "supplementary batch" with different F1 values (88.4 vs. 98.96), different human-audit sizes (150 vs. 240 cases), and inconsistent reporting of variance (the 98.96 F1 has no seed SD or paired interval). The relationship between the batches is never clearly explained, making it difficult to determine what NetTrace actually achieves.

Questions for authors
  1. What fraction of NetTraceBench is real traffic? Please provide a per-task breakdown of public-capture vs. controlled-execution cases, and state which of the paper's claims would hold if evaluated only on public captures. This directly determines whether the external-validity concern is fatal or addressable.

  2. How do the initial and supplementary batches relate? Are they the same benchmark and model with corrections, or different configurations? Please provide a table mapping each headline number to its batch, model, benchmark subset, and evaluation protocol.

  3. What do the failure cases look like? For the 41/1,161 human-audit insufficient-claim cases and the 21.94% of paired judgments where NetTrace-S loses to TextScope-S, what are the common patterns? Understanding when the evidence bound fails is essential for a paper about evidence-bounded reporting.

Reviewer_Y3YnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NetTrace trains a Qwen2.5-7B investigator, using structured traffic inputs (SPFL) and a separate scope-aware verifier, to release only claims jointly supported by acquired evidence. Contributions: a formal support criterion, NELR/USR metrics, and a release gate. Validated on the authors' NetTraceBench via controls, splits, and human audit.

Critical strengths

Strength 1: separates a citation exists from the evidence set jointly supports the claim within a specific scope, and formalizes the latter as a testable release criterion. This work characterizes a claim's scope along five dimensions, namely entity, time, traffic layer, quantifier, and conclusion strength, written as σ(c)=(ℓ,qc,s,e,[ts,te))\sigma(c)=(\ell, q_c, s, e, [t_s,t_e)). A claim may be released only if there exists a non-empty evidence subset E+⊆EcE^+\subseteq E_c that satisfies ScopeOK(E+,σ(c))\mathrm{ScopeOK}(E^+,\sigma(c)) and Sufficient(E+,c)\mathrm{Sufficient}(E^+,c), while ¬UnresolvedConflict(Eobs,c)\neg\mathrm{UnresolvedConflict}(E_{\rm obs},c) also holds. The formulation provides an evaluation vocabulary: NELR measures cases where citations exist but do not support the claim, USR measures scope overreach, and the distinction between SobsS_{\rm obs} and SextS_{\rm ext} separates support relative to the visible content from reliability against external truth. Even if I have reservations about some of the specific experimental results, this conceptual contribution stands, and its value is clear in settings such as security analysis, where auditability is required. H0042 case illustrates the point well.

Critical weaknesses

Weakness 1: Reliability of the core experimental results. Abstract and main table report a macro-F1 of 98.96 and a 12.10-point advantage over TextScope-S. However, paper itself acknowledges that this updated F1 comes with no seed standard deviation and no paired confidence interval. Appendix H.9 mentions only an author correction record, while the original outcome ledgers remain unchanged. This number is hard to reconcile with the rest of the paper. The controlled comparisons within the same paper show F1 gaps of only 3 to 5 points (88.53 vs. 85.49 under fixed observations; 84.97, 83.54, and 83.96 under the three strict splits, against roughly 78.7 to 80.5 for TextScope-S), the initial-batch Full model reaches 88.4, and all ablation variants (85.75 to 89.74) fall in the same range. This means the 9-to-13-point Full minus variant gaps in Figure 4(c) and Figure 20(c) derive almost entirely from the 98.96 figure, which has no uncertainty estimate. The value also appears to conflict with other metrics the paper reports. Table 1 defines over-answering as the fraction of the 840 reference-insufficient cases in which a determinate decision is output, and NetTrace-S reaches 11.98%. The recall of the insufficient class is therefore at most 1−0.1198=0.88021-0.1198=0.8802, which bounds that class's F1 by 2×0.8802×1/(0.8802+1)≈0.9362\times0.8802\times1/(0.8802+1)\approx0.936. If the macro average is taken over three decision classes, the other two classes would need an average F1 of (3×98.96−93.63)/2≈101.6(3\times98.96-93.63)/2\approx101.6, which exceeds 100 and is impossible. Paper doesn't define the macro average clearly, nor does it state whether the two metrics are taken from the same operating point, so I cannot assert the number is wrong, but the authors must explain this. Another point puzzles me as well: hierarchical JSON differs from SPFL by about 9 F1 points, but their NELR values are almost identical (2.98% vs. 2.84%), and the variant without consistency regularization shows the same pattern (89.74 F1 with 2.94% NELR). Since the paper's central claim concerns support quality, this is troubling.

Weakness 2: A mismatch between the scope of the evidence and the strength of the conclusions. All results come from a single backbone (Qwen2.5-7B) and a single tool suite. The baselines are all internal variants built by authors (ReAct-S, Memory-S, TextVerifier-S, TextScope-S, and so on), while the related work listed in Section 5, such as TrafficLLM, PentestGPT, Self-RAG, RARR, and Trust-Align, is not used as a comparison, and no stronger general-purpose LLM serves as a verifier. Section 4.2 is titled as a strong-verifier substitution, but the actual comparison is only against a same-scale 7B text verifier. The perturbation types in the counterfactual tests (layer expansion, quantifiers, key-evidence deletion, counterevidence) closely mirror the instance types used to train the verifier, so although TextScope-S uses the same training data, these tests look more like a check on whether the model has learned the relations the authors designed than evidence that the same advantage would hold in real traffic investigation. CTU-13 holdout covers only Host/C2, and the unseen-configuration test remains within controlled environments.

Questions for authors

Question 1: Could you explain how the headline macro-F1 of 98.96 was obtained, including how macro-F1 is defined? How is it consistent with the 11.98% over-answering rate on the 840 reference-insufficient cases, and with the 83 to 89 F1 range seen in the fixed-observation, strict-split, and ablation settings?

Question 2: Hierarchical JSON trails SPFL by about 9 F1 points, yet its NELR is nearly identical (2.98% vs. 2.84%), and the same holds for the variant without consistency regularization. If the central claim concerns support quality, what explains this decoupling between F1 and NELR?

Reviewer_cfzZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to make tool-using LLMs report only evidence-supported conclusions in network traffic investigation. The authors propose NetTrace. It first converts traffic into a text representation (SPFL) that preserves structure such as timing, direction, and packet order. It then fine-tunes Qwen2.5-7B in two stages to call tools, cite evidence, and write reports. A separately trained verifier checks whether each claim is fully supported by the evidence within its stated entity, time, and traffic-layer scope, and claims that fail are revised or withdrawn. The authors also build NetTraceBench, a benchmark of 24,000 cases covering four investigation tasks, and report that NetTrace outperforms a matched text-verifier baseline in both F1 and evidence support.

Critical strengths

Clear problem formulation. It separates evidence support from external correctness, and answerability from current evidence sufficiency.

Reasonable control experiments. Fixed-observation replay and matched-coverage risk comparison help rule out simple alternative explanations.

Critical weaknesses
  1. Poor presentation. The paper is hard to follow and shows clear signs of auto-research. The main text is filled with AI feedback-style statements, for example:

    • "updated paired intervals are unavailable" (repeated in the Figure 4 caption, Sec 4.3, and Figure 20)
    • "The updated F1 has no seed SD or paired confidence interval." (Sec 4.2)
    • "An accompanying author correction records Full macro-F1 of 98.96; original outcome ledgers remain unchanged." (App H.9)
    • "240 denotes the number of audited cases, not raters." (App H.7)

    I find it hard to understand why a human writer would repeatedly include such sentences in a paper. The whole paper reads more like an AI patching it point by point in response to review comments than a well-structured and coherent research paper.

  2. Lack of rigor and honesty. The paper claims the code is open-sourced, but the provided repository contains only empty files. The paper shows clear signs of auto-research, yet the AI Use Statement claims AI was used only for language polishing.

  3. The main result (98.96 macro-F1) contradicts other results in the paper and exceeds the upper bound implied by the paper's own numbers, which seriously undermines the paper's credibility.

    • Inconsistent with over-answering. On the 840 insufficient cases, NetTrace's over-answering rate is 11.98%, so recall on the insufficient class is at most 88.02% and its F1 at most about 93.6. Even if the other two classes reach a perfect F1 of 100, macro-F1 cannot exceed about 97.9.
    • Inconsistent with the ablations. Removing the consistency regularization (weight only 0.1) drops F1 by 9.2 points, while NELR barely changes (2.84% vs. 2.94%). In the initial batch, removing the same component drops F1 by only 0.9 points. All ablated variants score around 86–90, and only the full model is an outlier.
    • Inconsistent with other settings. The full model scores only 88.4 in the initial batch, 88.53 under fixed observations, and 83–85 under the stricter splits.
    • No uncertainty analysis. 98.96 is the only main result with neither a seed SD nor a CI. Appendix H.9 attributes it to an "author correction," while Figure 4(c) still keeps an old panel with a 0–6 pp axis, suggesting the original improvement was much smaller than reported.
  4. Limited experimental scope. Results use only 3 seeds, a single backbone (Qwen2.5-7B), and a single tool suite.

Questions for authors

I have no question to ask.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.