acceptodds
Under review as a conference paper at ICLR 2027

FragBench: Multi-Session Cyber Attacks Hidden in Benign-Looking Fragments

Abstract

Production LLM safety guardrails check one request at a time, yet documented cyber campaigns are already decomposed across many requests. A request that appears legitimate can therefore contribute to a harmful operation whose purpose is only apparent from the broader campaign. We introduce FragBench, a benchmark that includes attacks based on reported cyber campaigns, split across multiple model API requests to be executed in separate agent sessions. A catalogue of 35 cyber-incident and threat-report entries informs 24 campaign templates. The construction pipeline generates fragments and records their dependencies, safety-judge verdicts, tool activity, and execution-checker outcomes. Judge-guided iterated rewriting increases mean bypass from to on the evaluated subset. Classifiers trained on malicious and benign MCP executions achieve event F1 up to . FragBench supports analysis of both attack execution and detectability by retaining failed attempts alongside successful runs. We release the benchmark and evaluation code at https://anonymous.4open.science/r/fragbench-FE86/.

Primary area:
datasets and benchmarks
Keywords:
multi-session attack, decomposition attack, fragmentation attack, model context protocol, agentic attack

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vRxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FragBench, a benchmark for cyberattack workflows fragmented across independent LLM sessions and linked through persistent artifacts. It constructs 24 campaign templates from 35 incident and threat-report entries, generates benign controls, and records judge verdicts, MCP execution traces, and checker-assessed completion. Validation includes judge-guided rewriting, sandbox execution, trace-based detection, graph-free controls, and edge ablations. The evaluation usefully distinguishes request acceptance from successful execution, while acknowledging limitations in attributing detection performance to malicious composition.

Critical strengths
  1. A useful benchmark for cross-session misuse. Incident-informed workflows linked through artifacts complement prompt-only and shared-context evaluations. Connecting fragment plans to tool activity and outcomes makes the benchmark potentially valuable for studying both attack feasibility and detection.

  2. The evaluation separates acceptance from execution. Judge acceptance, tool invocation, error-free execution, and checker-assessed completion are measured separately, and failed attempts are retained. This distinction provides useful evidence that passing isolated safety checks does not establish successful campaign execution.

Critical weaknesses
  1. The detection results do not isolate malicious composition from workload differences, and the cross-session interpretation needs qualification. Table 3 shows substantial class differences in session count, tool-call volume, argument length, and resource reuse. Without workload-matched controls, detection scores cannot establish recognition of malicious composition. Table 5 also partitions benign events randomly, leaving generalization to unseen benign workflows unclear. Separately, Table 6 shows GIN improving from 0.705 with the full graph to 0.874 with within-session edges alone, opposite to the other three architectures. The claim that cross-session edges provide an additional increment therefore requires architecture-specific qualification. The authors acknowledge the harmful effect of argument-similarity edges, but do not specifically explain GIN's larger degradation.
  2. Experimental coverage and reproducibility are insufficiently specified. Section 6.1 reports three campaigns, six styles, three judges, and five rewriters, giving 270 possible combinations, but reports only 183 evaluated cells without a complete manifest or selection and aggregation rules. A partial design is acceptable, but Table 10 reports a separate configuration and does not explain this coverage. Appendix C also lacks complete classifier configurations, graph boundaries, preprocessing, split assignments, and threshold-selection procedures. These omissions prevent independent assessment of the reported aggregate results and training/test separation.
Questions for authors
  1. Can the authors provide the 183-cell rewriting manifest, selection criteria, and averaging procedure, confirming whether before/after results use identical cells and weights?

  2. Clarification: What explains GIN's full-graph degradation in Table 6, and how should the cross-session benefit be qualified across architectures?

  3. Evidence needed for assessment: Can the authors provide workload-matched results and complete benign-workflow or run holdouts to establish what detection signal remains beyond workload differences?

  4. Reproducibility clarification: Can the authors supply the exact classifier configurations, graph-construction rules, split manifests, and threshold-selection procedures for the reported results?

2 comments

Response to Reviewer vRxS (1/2)

Thank you for the careful review, and for recognizing that FragBench complements prompt-only and shared-context evaluations and that separating acceptance from execution is useful evidence. We answer your four questions below (Q1–Q2 here, Q3–Q4 in the next comment). All details will be added to the final version.

Q1. The 183-cell manifest, selection, and averaging

The design is 3 campaigns (DPRK_FRAUD, PROMPTSTEAL, QUIETVAULT) × 6 styles × 3 judges × 5 rewriters = 270 cells; 183 were evaluated:

DimensionCells evaluated
Judge (of 90 each)Llama Guard 4: 90 · Claude Sonnet: 50 · Claude Opus: 43
Campaign (of 90 each)DPRK_FRAUD: 71 · QUIETVAULT: 57 · PROMPTSTEAL: 55
Style (of 45 each)command_form: 42 · compliance_audit: 34 · sysadmin: 31 · direct: 28 · helpdesk: 25 · educational: 23
Rewriter (of 54 each)35 to 39 for each of the five rewriters

Selection. The 183 cells are all runs of this design that had completed when the tables were built (latest run per cell). The other 87 were not run or did not finish. None was dropped because of its result.

Averaging. Yes, before and after use identical cells and weights: both means are unweighted over the same 183 cells. Each cell is one run of the rewriting loop. "Before" is the judge pass rate on the unmodified fragments at round 0; "after" is the pass rate after the final round of the same run. The 59.4% → 74.6% gain is therefore a paired, within-cell effect over the same cells and fragments.

Three details belong with these numbers, and we will add them to the paper:

  • Stopping rule. The loop stops early once 90% of a cell's fragments pass, so "ten rounds" means up to ten. §4 describes the rule as all fragments passing; we will correct it.
  • Rewriters that produced no rewrite. 72 of the 183 cells use the two GPT rewriters, whose API calls returned errors (HTTP 400) and produced no rewrite. These cells contribute a baseline but no rewriting effect, so the gain comes from the 111 cells using the other three rewriters, and the pooled figure is diluted by the rest.
  • Zero baselines. 24 cells start from a round-0 pass rate of 0%.

Coverage is complete for Llama Guard and partial for the two Claude judges. We will report per-judge means next to the pooled mean, with and without the 72 cells above, together with the per-cell manifest. Table 10 is a separate experiment and not part of this grid: it covers two campaigns and three rewriters, and none of its cells enter the 183-cell means. Its caption will say so.

Q2. GIN's degradation and how to qualify the cross-session benefit

Cross-session edges help three of four architectures: full-graph versus within-session F1 changes by +0.021 (GraphSAGE), +0.022 (GAT), and +0.064 (GCN), and by −0.169 for GIN. The qualified claim is that the benefit is seen for the three models with mean- and attention-based aggregation and reverses for GIN. Each Table 6 entry is a single run, so the two gains of about 0.02 come without a variance estimate (§7).

GIN is the only one of the four that aggregates neighbors by an unnormalized sum, which makes it sensitive to the extra degree that cross-session edges add. The ablations are consistent with this. Dropping argument-similarity edges recovers GIN from 0.705 to 0.787, and dropping shared-resource edges instead gives 0.823; neither reaches its within-session 0.874. GIN's loss is therefore not specific to argument-similarity edges: either cross-session edge type lowers it. We offer the aggregation explanation as a hypothesis, not a verified cause, and will state the result per architecture.

Response to Reviewer vRxS (2/2)

Q3. Workload differences and benign holdouts

We agree the detection scores do not isolate malicious composition from workload differences; §7 notes the classes are not workload-matched. F1 measures separation of campaign-associated events from the benign controls. Three points bound what the scores mean:

  • The controls are executed, not synthetic. Benign workflows run through the same actor, tools, sandbox, and tracer as the malicious objectives (Table 4: GAT 0.876, GraphSAGE 0.858).
  • The workload differences run in both directions. In Table 3 benign chains are larger (more sessions and tool calls per chain), while malicious chains have longer arguments (p90 6,944 vs. 3,216 characters) and less resource reuse (0.42 vs. 0.59). A detector can use either direction, so this does not remove the confound.
  • We report the baseline that bounds the graph's contribution. On the campaign-disjoint split of Tables 5–6, the graph-free user-aggregate baseline reaches 0.893, within-session GAT 0.913, and full-graph GAT 0.935. The graph adds 0.042 over a baseline that itself pools workload attributes such as argument length.

Workload-matched controls and whole-workflow benign holdouts have not been run; Table 5 holds out five malicious campaigns and partitions benign events at random. We cannot yet say how much signal remains beyond workload differences, and the final version will state this scope explicitly.

On the malicious side we can add one result not in the submission. In the Table 4 setup we held out one of five campaign families at a time, so no variant of a held-out campaign is seen in training. GAT averages 0.854 F1 over the five folds (0.801 to 0.880), against 0.876 with no holdout. Benign events are still partitioned at random there, so this addresses unseen campaigns, not the benign-holdout question.

The detection analysis is secondary. The dataset and the execution results do not depend on it.

Q4. Classifier configurations, graph rules, splits, thresholds

  • Architecture: 2-hop neighbor sampling (10 and 5 neighbors), hidden dimension 256, embedding dimension 128, 64-unit scoring head; GAT uses 4 attention heads.
  • Optimization: Adam, learning rate 1e-3, weight decay 1e-5, cosine annealing, 30 epochs, batch size 256, class-weighted binary cross-entropy, gradient clipping at 1.0. Seed 42 fixes the split, neighbor sampling, and batch order; weight initialization was not seeded.
  • Classical models (Table 4): 36 graph-derived features per event; SVM (RBF, C=10, class-balanced), MLP (128, 64; early stopping), GBT (200 trees, depth 8, learning rate 0.05).
  • Graph, Table 4: nodes are call and result events. Edges link a call to its result, consecutive events in a session, and events in one chain naming the same path, file, URL, or IP address. No edge crosses a chain.
  • Graph, Tables 5–6: nodes are tool-call events, with three within-session edge types (data flow, temporal order, same session) and two cross-session types (shared resource; argument similarity by SimHash, Hamming distance ≤ 8). Cross-session edges are built over the whole corpus, so they can link different chains.
  • Preprocessing: per-event attributes (capability one-hot from the tool name, argument entropy, payload size, hashed tool identity) under a fixed random projection to 128 dimensions.
  • Split: 80/20. Table 4 holds out whole malicious chains, so every campaign also appears in training; Tables 5–6 hold out 5 of the 24 campaigns. In both, benign events are partitioned at random at the same ratio, so a benign chain has events on both sides.
  • Threshold and model selection: event-level F1 uses a fixed 0.5 threshold; none is tuned. The GNN checkpoint, however, is selected by AUC on the test events, with no separate validation split. This can favor the GNN rows, including the 0.042 margin in Q3; we will state it in Appendix C.
  • Events: Table 4 uses call and result events; Tables 5–6 use tool-call events and a different split, so their F1 values are not directly comparable.

These settings, the graph-construction rules, and the split manifests will be added to Appendix C and the released code.

Thank you again. Your questions let us put the missing details on the record, including those that qualify our own numbers. We hope they address your concerns, and we would be grateful if you would reconsider your score.

Reviewer_Jr3wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper discusses the safety threats of decomposition attacks. They have synthesized the scenarios that from 35 incident reports. Then they measure the effectiveness of the decompositions as well as the disguised version. Finally, the study whether a model trained on the subtasks can detect the attacks .

Critical strengths

1- The paper is written very well and easy to follow.

2- The curated dataset is significant and fascinate further studies on decomposition attacks and how to avoid them.

Critical weaknesses

1- Apart from the curated dataset, the novelty of the paper is limited. Note that environments such as ExploitGym and CyberBench already provide a wide range of tasks to test the offensive capabilities of models. Then, many previous decomposition attacks can be readily applied in such environments.

2- Then, the question is what is the advantage of the current curated dataset, and I could not find this comparison in the paper. E.g., are the considered scenarios more realistic for decomposition attacks?

3- The paper does not argue if the fragmentation and disguise provides actual uplift to completing the task. I.e., it focuses on whether the judge allows a task or not, but the rewritten version can actually hurt the performance.

4- The detection on the defense part is done in the literature before. For instance, [1] trains a detector on the generated fragments and studies a similar F1 score.

[1] Brown et al, "Benchmarking Misuse Mitigation Against Covert Adversaries".

Questions for authors

Please read above.

3 comments

Thank you for your review. You ask what it offers beyond existing environments and decomposition attacks. We answer your four points in order, starting with what makes FragBench distinct.


W1–W2. Novelty and advantage over ExploitGym, CyberBench, and prior decomposition work.

FragBench is a different kind of object from a capability environment with a decomposition attack applied to it. To our knowledge, no existing resource combines these five properties:

  • Campaign-level, incident-derived tasks. Each of the 24 templates reconstructs a multi-stage campaign from public incident and threat reports (35 entries), with every stage tied to the report evidence it comes from. Capability environments pose individual tasks; ExploitGym, for example, asks whether an agent can turn a known vulnerability into a working exploit [4]. Fragments designed in, with explicit dependencies. Each fragment declares the artifacts it produces and consumes. Sessions share nothing except those persistent artifacts. The decomposition is part of the task definition, not the output of an attack method applied afterwards.
  • Real tool execution in isolated sessions. Fragments run through MCP tools in a sandbox, and the full tool activity is recorded. Attacker and defender evidence in one record. The same campaign carries safety-judge verdicts, execution-checker outcomes, and the traces a defender would see, alongside benign workflows executed through the same harness.
  • Failures are kept. Researchers can see where a chain breaks, not only whether it succeeded. Applying an existing decomposition attack inside ExploitGym or CyberBench, as you suggest, is a valid study, and it would give property 3. Those environments pose individual exploitation or CTF tasks, so it would not give incident-derived campaigns, predefined artifact dependencies, executed benign controls, or the linked judge-to-execution record.

The closest prior work overlaps on single properties, and we will cite each explicitly: Brown et al. study tasks decomposed into independent queries and defenses that aggregate query history [1]; DeCompBench compares monolithic and decomposed tasks with environment-state checks [2]; CSTM-Bench studies cross-session detection on generated conversation transcripts [3]. We do not claim decomposition or cross-session detection is new in itself. The contribution is the linked, incident-derived, executed record. On your realism question: the scenarios are grounded in documented campaigns, with the source reports cited per seed so that fidelity can be checked. They are sandbox reconstructions, not captured attacker prompts, and we will say so plainly.


W3. Does fragmentation and disguise give uplift on task completion?

You are right that the rewritten version could hurt completion, and the paper's own results show why the distinction matters. Rewriting raises mean judge acceptance from 59.4% to 74.6% on the evaluated subset, while in a separate execution cohort only 8.5% of malicious chains pass every fragment check. Acceptance is not completion; that is a finding FragBench makes measurable, and we report the two separately for this reason.

We do not claim completion uplift, and the final version will remove any wording that suggests it. The multi-session format is taken from the incident reports themselves, so the benchmark does not depend on fragmentation being advantageous. Each seed keeps the unfragmented prompt for every stage, so a paired uplift comparison on the same campaign is possible with FragBench; we will describe it as an evaluation the benchmark enables.


W4. Detection has been done before (Brown et al.).

We agree that stateful misuse detection has precedent in Brown et al. [1] and will credit it clearly. The two analyses use different evidence. Theirs aggregates query history of question-answering prompts and also trains a per-prompt detector. Ours classifies recorded tool activity and event relationships after execution, against benign workflows executed in the same harness. A similar score does not indicate the same task, because the data, labels, splits, and evaluation units differ. Our detection study uses standard baselines and is a secondary analysis of the benchmark's traces. We do not claim a new detector or superior detection; the contribution is the data that makes such comparisons possible.


Thank you again. We hope this makes clear why the dataset you found significant is not obtainable by combining existing environments with existing attacks.


References

[1] Brown et al., arXiv:2506.06414 | [2] Kothamasu et al., DeCompBench, arXiv:2606.13994 | [3] Azarafrooz, CSTM-Bench, arXiv:2604.21131 | [4] Wang et al., ExploitGym, arXiv:2605.11086

I thank the authors for the clarification. I understand how the data and the fragments are collected now. What I don't understand is why running a simple decomposition on top of ExploitGym, CyberBench, or any other misuse dataset cannot capture what the real data from the real scenarios does. Can you please explain this? For instance, how does this help us understand or benchmark the decomposition attacks better?

Another question is, what is the reason that the authors emphasize on real world scenarios? Where do the authors explain the advantage of these real world data (which is probably generated by existing decomposition attacks already) over synthetic data?

Thank you for the follow-up.

Q1. Why can't a simple decomposition on top of ExploitGym, Cybench, or another misuse dataset capture this?

It tests a different setting and yields a different record.

  • Setting. Existing decomposition attacks split a request into sub-prompts and reassemble them inside one context window (§2). In FragBench every fragment runs in a fresh session with no shared history; the only link is an artifact left in the sandbox, such as a file an earlier fragment wrote. That is what a per-request guardrail faces, and decomposing a prompt does not produce it.
  • Question. ExploitGym measures capability: can an agent turn a known vulnerability into a working exploit? Its main experiments run with safeguards disabled. FragBench measures what happens around the safeguards: which requests a judge accepts, what the agent executes, and what a defender can see.
  • Unit. An exploit task is one tightly coupled problem. A campaign has stages with different tools, each depending on an earlier session's output. Our split is fixed before any judge is consulted, so every defense is compared on the same fragments.
  • Negatives. Our benign workflows run through the same actor, tools, sandbox, and tracer (§3.4). An attack applied to a capability environment gives only the malicious side, so detection cannot be evaluated.

The payoff is that outcomes usually reported as one are separated. GuardBench guardrail models accepted every tested fragment individually (§1). With LLM judges, rewriting raises acceptance from 59.4% to 74.6%, and 95.4% of tool calls return without error. Yet only 8.5% of chains have every fragment pass the checker. Stopping at "the decomposed prompt was accepted" overstates the threat. The campaign objective and seed are kept, so automated decomposition attacks can also be run on the same campaigns and compared with our split.

Q2. Why real-world scenarios, and what do they offer over synthetic data?

Our fragments are not recordings of real attackers, and no decomposition attack produced them. We read published incident reports, wrote a seed for each documented campaign, split it into 4–16 fragments (drafted with Claude Opus, fixed in the seed), and had a second author check each scenario against its source (§3.2). The scenarios follow the stages the reports document, not what a dataset designer assumes an attack looks like.

The benchmarks you mention make the same choice: ExploitGym builds on real vulnerabilities, and Cybench (Zhang et al., ICLR 2025) on professional CTF competitions. Both are trusted because they are grounded in reality. We apply that principle to whole campaigns.

The motivation is in §1: reports describe real campaigns but never release the detailed data, so researchers cannot replicate them or test defenses against them. FragBench fills that gap with executable reconstructions; 13 of the 24 campaigns have attacker LLM use directly observed in the source reports (§3.1).

To be precise about the limit: we have not shown that incident-informed fragments are harder to detect than synthetically decomposed ones, and the paper will say so. What FragBench offers is relevance, a place to test whether a monitor catches the staging patterns reported in practice.

Reviewer_BttDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies harmful plans split into ordinary-looking requests across separate AI conversations. Checking each request alone may miss the harmful purpose of the combined steps. It builds 24 example campaigns inspired by reported incidents. Fresh conversations are connected through files and other saved results. It tests whether requests pass a safety judge, whether steps pass a completion check and whether activity records reveal the campaign. Rewriting raises judge acceptance from 59.4% to 74.6% on the tested subset. In a separate experiment, 8.5% of attack chains pass all step checks.

Critical strengths

The records connect the overall plan to individual requests, shared files and observed computer activity (Table 1). Failed attempts are kept as well, so researchers can examine where a sequence breaks down. This is useful material for studying attacks across several conversations. The paper identifies passing a safety check, calling a tool, receiving no tool error and passing a completion check are different outcomes. The paper keeps them separate and warns against combining rates from different experiments into one attack-success rate. This helps understand the results.

Critical weaknesses

An event is labelled by the campaign it belongs to, rather than by an independent check that this particular event was harmful. Attack and legitimate examples also differ in tool-call volume, request length and resource reuse. A detector may use these differences, as the authors acknowledge. Legitimate events are randomly divided between the data splits, so Table 5 does not establish performance on entirely new legitimate workflows.

An AI checker decides from the logs whether each step was completed. The final outcome is not independently measured. For me, passing all step checks is therefore insufficient evidence that the complete intended attack succeeded.

Questions for authors

Table 5 holds out complete attack campaigns but randomly divides legitimate events. Does detection still work when entire legitimate workflows are held out for testing? Can an independent check confirm that chains approved under the completion rule in Section 4 achieved their intended final outcomes?

1 comment

Thank you for the thoughtful review. We are glad the linked records (Table 1), the retained failures, and the separation of four distinct outcomes came through as the paper's strengths; that separation is what FragBench is built to provide. We answer your two questions directly.


Q1. Can an independent check confirm that approved chains achieved their final outcomes?

The 8.5% figure is deliberately strict and deliberately narrow. The checker is an LLM judge applied to each fragment. It reads the request, the names of the artifacts that fragment must produce and consume, and the recorded tool calls, tool results, and final response. It passes a fragment only when a successful tool call matching the requested action appears; an unclear verdict counts as a failure, and a chain passes only if every fragment passes.

We agree this is a judgment from logs and not a measurement of the final sandbox state, and we will label 8.5% exactly as "malicious chains with every fragment passing the execution checker."

This is where the benchmark's design helps. Because every chain keeps its objective, dependency plan, tool activity, and failed attempts, and because the containerized harness is released, any researcher can audit the checker's verdicts against the recorded tool calls. A final-state check would also need the shell, archive and exfiltration tools, which are simulations in our sandbox (Appendix D, Table 12), to run for real. Each campaign seed was also reviewed by a second author against its source report for scenario fidelity, fragmentation, and linkage (§3.2).

The paper's central observation survives the stricter reading you propose. Judge acceptance and error-free tool calls are both high, and chain-level completion is far lower under any check. A final-state check applied on top of the checker can only lower 8.5% further, which strengthens the conclusion that passing isolated safety checks does not establish a successful campaign. Thank you for helping us state this precisely.


Q2. Does detection still work when entire legitimate workflows are held out?

Table 5 answers the attack-side version of this question: five complete malicious campaigns are held out, so the detector is tested on campaigns it has never seen. The benign-side version, holding out whole benign workflows, has not been run; benign events are partitioned at random. We will state this scope explicitly in the final version.

Three facts bound how the current scores should be read:

  • The benign controls are real executions. We generated 24 workflow specifications, expanded them to 1,296 objectives, and executed them with the same actor and MCP harness as the malicious objectives; 514 sampled benign chains enter the detection analysis (§3.4). GAT reaches event F1 0.876 on these executed traces (Table 4).
  • We report a strong graph-free baseline. In the separate Table 5 comparison, graph-free run aggregation already reaches 0.893, against 0.935 for full-graph GAT. We include it so that high F1 is not mistaken for recognition of malicious composition.
  • Labels mean campaign membership. As you note, an event is labelled by its campaign, and the workload differences in Table 3 remain a possible signal. The reported F1 measures separation of campaign-associated events, and we will use that wording.

The detection analysis characterizes the benchmark's traces. The benchmark itself, the linked records you describe, does not depend on these scores.

Reviewer_gK3rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FragBench, which studies whether harmful cyber campaigns can evade LLM safety judges by splitting them into "chains" of smaller, benign-looking "fragments" to be executed by a separate agent across sessions.

The authors construct 24 such cyber campaigns inspired by real reports where each fragment is a prompt connected to others through shared files. They then (1) rewrite these prompts to increase acceptances by the LLM safety judges, (2) have a separate agent attempt to execute them with MCP tools, and (3) train classifiers to detect malicious campaigns.

They show that rewriting can increase the rate of acceptance by safety judges, that 8.5% of malicious chains have every fragment pass the checker, and that a graph-based detector achieves up to 0.876 event-level F1.

Critical strengths
  1. The construction of an incident-informed benchmark is timely and important. The benchmark helps address the important problem of studying the practicality of these attacks and their detectability.
Critical weaknesses
  1. The current evaluations do not adequately demonstrate increased attack success from fragmentation.

(a) The authors do not compare the same campaign submitted as (1) a single request, (2) multiple requests within one session, and (3) multiple requests across separate sessions. Without these comparisons, it is unclear what benefit fragmentation provides to an attacker.

(b) The authors do not verify whether each check in the full chain passing means the original harmful objective, before fragmentation and rewriting, was achieved. This is particularly important given that only 8.5% of malicious chains pass all fragment checks, and rewriting fragments to bypass safety checks may alter their meaning.

  1. It is unclear whether the detectors truly learn to identify malicious cross-session campaigns.

(a) The malicious and benign chains differ substantially in characteristics such as sessions per chain (8.2 vs. 10.6) and tool calls per chain (48.7 vs. 75.1).

(b) The ablations show that the addition of cross-session edges for GAT results in a small change in F1 (0.913 to 0.935), suggesting that detection does not depend strongly on cross-session relationships.

Questions for authors
  1. How does attack success compare when the same attack is submitted as one message, multiple messages within one session, or multiple messages split across separate sessions (weakness 1 (a))?
  2. Does passing every fragment's execution check mean the full attack succeeded (weakness 1 (b))?
  3. How do the authors ensure that adversarial rewriting preserves the same goal as the original fragments (weakness 1 (b))?
2 comments

Thank you for the clear review and for calling the benchmark timely and important. Your main concern is that we do not show increased attack success from fragmentation. We would like to clarify the claim first, because we think the paper is being held to a claim it does not make or need to address.

What FragBench claims. The multi-session format is not our hypothesis about what works best for an attacker. Public reports describe campaigns carried out across many requests; our session splits are reconstructions of those, not captured attacker inputs (§3.3). FragBench reconstructs those campaigns so that we can measure which fragments a safety judge accepts, what the agent then executes, and what a defender could observe. This is motivated by the fact that guardrails that check one request at a time may not catch a malicious attack campaign broken up into multi-session fragments. Hence, we believe the benchmark is valid whether fragmentation helps, hurts, or leaves attack success unchanged.


Q1. One message vs. one session vs. separate sessions.

We did not run this comparison, and it tests a claim (uplift from fragmentation) that the paper will not make; we will remove any wording that suggests it. The benchmark already contains what the comparison needs. Each campaign seed stores the unfragmented prompt for every stage next to the fragments derived from it, so the same campaign can be issued as one request, as several requests in one session, or across separate sessions, with the actor, tools, and initial sandbox state held fixed.


Q2. Does passing every fragment check mean the full attack succeeded?

No. It means the checker passed every step in the sandbox, where some tools are simulations (Appendix D, Table 12). It does not mean the overall objective was verified, and we will label it that way: "malicious chains with every fragment passing the execution checker." The checker is an LLM judge applied per fragment. It reads the request, the artifacts the fragment must produce and consume, and the tool calls, results, and final response. It passes a fragment when at least one successful tool call matches the requested action, and a chain passes only if every fragment passes.

The low rate you point to is one of the paper's findings, not a weakness of it. 95.4% of executed malicious calls returned without an MCP error, yet only 8.5% of chains passed every fragment. Acceptance and error-free execution greatly overstate campaign completion, and FragBench is what makes that gap visible.


Q3. Does rewriting preserve the original goal?

Three mechanisms tie a rewritten fragment to its origin:

  1. Reviewed seeds. A second author reviewed each campaign seed against its source report, including its fragments and their linkage (§3.2).
  2. Fixed dependencies. Each fragment's produced and consumed artifacts are declared in the seed and do not change under rewriting. A rewrite that drifts from its goal will tend not to produce the artifact the next fragment needs, so the chain breaks at execution.
  3. Retained parents. Every rewritten fragment stays attached to its original parent prompt, so any rewrite can be audited against it.

We do not claim every rewrite is semantically verified; rewriting optimizes judge acceptance and some drift is possible. This is why we report acceptance and execution separately and claim only improved acceptance on the evaluated subset.


Weakness 2. Do the detectors learn cross-session campaigns?

(a) On Characteristics: the differences you cite run in the benign direction. Benign chains have more sessions (10.6 vs. 8.2) and more tool calls (75.1 vs. 48.7) than malicious chains, so the detector is not simply flagging high activity. We agree a distribution-matched control would isolate the signal further, and we will state that it has not been run.

(b) On cross-session edges: we agree the gain is modest for GAT (0.913 within-session, 0.935 full graph), and the paper reports it as such. The edge ablation shows where the gain comes from: removing SHARED_RESOURCE lowers GAT to 0.897, while removing ARGUMENT_SIMILARITY raises it to 0.945. The useful cross-session signal is the shared artifact, which is exactly the link the campaign construction is built on.

The detection study is a secondary analysis. The contribution is the benchmark record: an incident-informed objective, its fragments and artifact dependencies, and the judge verdicts, tool activity, and checker outcomes across sessions, with failures kept. We hope this clarifies why we believe the benchmark stands on its own, and we thank you for pushing us to state the claim plainly.

Thank you for the reply.

If the authors do not claim any uplift from fragmentation of attacks, cannot confirm that passing every fragment check means the full attack succeeded, nor provide assurance that the fragmented attacks represent the original attack goal, then what, in the authors' words, is the main contribution and significance of the work, and how can we trust it?

Comments on the paper

1 comment

Thank you for highlighting the need for this comparison and we are sorry for not completely understanding your point. In our manuscript, we already mentioned about the evaluation with GuardBench in the introduction but not explicit 3 conditions as you required. For the same evaluated campaigns, submitting all steps in one request yielded 0% ASR, whereas submitting the steps as fragments yielded 100% ASR both within one session and across separate sessions. These results support an advantage from request fragmentation under the evaluated GuardBench. We will make this distinction explicit and report the three conditions together, including the evaluated models, campaign counts, and success criterion. These guardrail results are separate from our sandbox execution and campaign-completion measurements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.