- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FragBench, a benchmark for cyberattack workflows fragmented across independent LLM sessions and linked through persistent artifacts. It constructs 24 campaign templates from 35 incident and threat-report entries, generates benign controls, and records judge verdicts, MCP execution traces, and checker-assessed completion. Validation includes judge-guided rewriting, sandbox execution, trace-based detection, graph-free controls, and edge ablations. The evaluation usefully distinguishes request acceptance from successful execution, while acknowledging limitations in attributing detection performance to malicious composition.
Critical strengths
-
A useful benchmark for cross-session misuse. Incident-informed workflows linked through artifacts complement prompt-only and shared-context evaluations. Connecting fragment plans to tool activity and outcomes makes the benchmark potentially valuable for studying both attack feasibility and detection.
-
The evaluation separates acceptance from execution. Judge acceptance, tool invocation, error-free execution, and checker-assessed completion are measured separately, and failed attempts are retained. This distinction provides useful evidence that passing isolated safety checks does not establish successful campaign execution.
Critical weaknesses
- The detection results do not isolate malicious composition from workload differences, and the cross-session interpretation needs qualification. Table 3 shows substantial class differences in session count, tool-call volume, argument length, and resource reuse. Without workload-matched controls, detection scores cannot establish recognition of malicious composition. Table 5 also partitions benign events randomly, leaving generalization to unseen benign workflows unclear. Separately, Table 6 shows GIN improving from 0.705 with the full graph to 0.874 with within-session edges alone, opposite to the other three architectures. The claim that cross-session edges provide an additional increment therefore requires architecture-specific qualification. The authors acknowledge the harmful effect of argument-similarity edges, but do not specifically explain GIN's larger degradation.
- Experimental coverage and reproducibility are insufficiently specified. Section 6.1 reports three campaigns, six styles, three judges, and five rewriters, giving 270 possible combinations, but reports only 183 evaluated cells without a complete manifest or selection and aggregation rules. A partial design is acceptable, but Table 10 reports a separate configuration and does not explain this coverage. Appendix C also lacks complete classifier configurations, graph boundaries, preprocessing, split assignments, and threshold-selection procedures. These omissions prevent independent assessment of the reported aggregate results and training/test separation.
Questions for authors
-
Can the authors provide the 183-cell rewriting manifest, selection criteria, and averaging procedure, confirming whether before/after results use identical cells and weights?
-
Clarification: What explains GIN's full-graph degradation in Table 6, and how should the cross-session benefit be qualified across architectures?
-
Evidence needed for assessment: Can the authors provide workload-matched results and complete benign-workflow or run holdouts to establish what detection signal remains beyond workload differences?
-
Reproducibility clarification: Can the authors supply the exact classifier configurations, graph-construction rules, split manifests, and threshold-selection procedures for the reported results?
2 comments
Response to Reviewer vRxS (2/2)
Q3. Workload differences and benign holdouts
We agree the detection scores do not isolate malicious composition from workload differences; §7 notes the classes are not workload-matched. F1 measures separation of campaign-associated events from the benign controls. Three points bound what the scores mean:
- The controls are executed, not synthetic. Benign workflows run through the same actor, tools, sandbox, and tracer as the malicious objectives (Table 4: GAT 0.876, GraphSAGE 0.858).
- The workload differences run in both directions. In Table 3 benign chains are larger (more sessions and tool calls per chain), while malicious chains have longer arguments (p90 6,944 vs. 3,216 characters) and less resource reuse (0.42 vs. 0.59). A detector can use either direction, so this does not remove the confound.
- We report the baseline that bounds the graph's contribution. On the campaign-disjoint split of Tables 5–6, the graph-free user-aggregate baseline reaches 0.893, within-session GAT 0.913, and full-graph GAT 0.935. The graph adds 0.042 over a baseline that itself pools workload attributes such as argument length.
Workload-matched controls and whole-workflow benign holdouts have not been run; Table 5 holds out five malicious campaigns and partitions benign events at random. We cannot yet say how much signal remains beyond workload differences, and the final version will state this scope explicitly.
On the malicious side we can add one result not in the submission. In the Table 4 setup we held out one of five campaign families at a time, so no variant of a held-out campaign is seen in training. GAT averages 0.854 F1 over the five folds (0.801 to 0.880), against 0.876 with no holdout. Benign events are still partitioned at random there, so this addresses unseen campaigns, not the benign-holdout question.
The detection analysis is secondary. The dataset and the execution results do not depend on it.
Q4. Classifier configurations, graph rules, splits, thresholds
- Architecture: 2-hop neighbor sampling (10 and 5 neighbors), hidden dimension 256, embedding dimension 128, 64-unit scoring head; GAT uses 4 attention heads.
- Optimization: Adam, learning rate 1e-3, weight decay 1e-5, cosine annealing, 30 epochs, batch size 256, class-weighted binary cross-entropy, gradient clipping at 1.0. Seed 42 fixes the split, neighbor sampling, and batch order; weight initialization was not seeded.
- Classical models (Table 4): 36 graph-derived features per event; SVM (RBF, C=10, class-balanced), MLP (128, 64; early stopping), GBT (200 trees, depth 8, learning rate 0.05).
- Graph, Table 4: nodes are call and result events. Edges link a call to its result, consecutive events in a session, and events in one chain naming the same path, file, URL, or IP address. No edge crosses a chain.
- Graph, Tables 5–6: nodes are tool-call events, with three within-session edge types (data flow, temporal order, same session) and two cross-session types (shared resource; argument similarity by SimHash, Hamming distance ≤ 8). Cross-session edges are built over the whole corpus, so they can link different chains.
- Preprocessing: per-event attributes (capability one-hot from the tool name, argument entropy, payload size, hashed tool identity) under a fixed random projection to 128 dimensions.
- Split: 80/20. Table 4 holds out whole malicious chains, so every campaign also appears in training; Tables 5–6 hold out 5 of the 24 campaigns. In both, benign events are partitioned at random at the same ratio, so a benign chain has events on both sides.
- Threshold and model selection: event-level F1 uses a fixed 0.5 threshold; none is tuned. The GNN checkpoint, however, is selected by AUC on the test events, with no separate validation split. This can favor the GNN rows, including the 0.042 margin in Q3; we will state it in Appendix C.
- Events: Table 4 uses call and result events; Tables 5–6 use tool-call events and a different split, so their F1 values are not directly comparable.
These settings, the graph-construction rules, and the split manifests will be added to Appendix C and the released code.
Thank you again. Your questions let us put the missing details on the record, including those that qualify our own numbers. We hope they address your concerns, and we would be grateful if you would reconsider your score.
Response to Reviewer vRxS (1/2)
Thank you for the careful review, and for recognizing that FragBench complements prompt-only and shared-context evaluations and that separating acceptance from execution is useful evidence. We answer your four questions below (Q1–Q2 here, Q3–Q4 in the next comment). All details will be added to the final version.
Q1. The 183-cell manifest, selection, and averaging
The design is 3 campaigns (DPRK_FRAUD, PROMPTSTEAL, QUIETVAULT) × 6 styles × 3 judges × 5 rewriters = 270 cells; 183 were evaluated:
Selection. The 183 cells are all runs of this design that had completed when the tables were built (latest run per cell). The other 87 were not run or did not finish. None was dropped because of its result.
Averaging. Yes, before and after use identical cells and weights: both means are unweighted over the same 183 cells. Each cell is one run of the rewriting loop. "Before" is the judge pass rate on the unmodified fragments at round 0; "after" is the pass rate after the final round of the same run. The 59.4% → 74.6% gain is therefore a paired, within-cell effect over the same cells and fragments.
Three details belong with these numbers, and we will add them to the paper:
Coverage is complete for Llama Guard and partial for the two Claude judges. We will report per-judge means next to the pooled mean, with and without the 72 cells above, together with the per-cell manifest. Table 10 is a separate experiment and not part of this grid: it covers two campaigns and three rewriters, and none of its cells enter the 183-cell means. Its caption will say so.
Q2. GIN's degradation and how to qualify the cross-session benefit
Cross-session edges help three of four architectures: full-graph versus within-session F1 changes by +0.021 (GraphSAGE), +0.022 (GAT), and +0.064 (GCN), and by −0.169 for GIN. The qualified claim is that the benefit is seen for the three models with mean- and attention-based aggregation and reverses for GIN. Each Table 6 entry is a single run, so the two gains of about 0.02 come without a variance estimate (§7).
GIN is the only one of the four that aggregates neighbors by an unnormalized sum, which makes it sensitive to the extra degree that cross-session edges add. The ablations are consistent with this. Dropping argument-similarity edges recovers GIN from 0.705 to 0.787, and dropping shared-resource edges instead gives 0.823; neither reaches its within-session 0.874. GIN's loss is therefore not specific to argument-similarity edges: either cross-session edge type lowers it. We offer the aggregation explanation as a hypothesis, not a verified cause, and will state the result per architecture.