- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the repeatability of harmful outcomes in tool-using language agents and its implications for replay-based attribution. A census of 120 AgentDojo configurations, repeated 20 times under two decoding conditions, measures replay disagreement alongside aggregate attack success. Additional experiments examine attribution stability, payload neutralization, and edits outside the injected payload. The results show that stable aggregate rates can coexist with variable individual outcomes and substantial sensitivity to edits outside the injected source.
Critical strengths
-
The primary census systematically evaluates every configuration in the selected suite, reducing the selection problems associated with screening configurations using only a few repetitions.
-
The paper distinguishes aggregate attack success from individual replay repeatability and complements observational measurements with explicit payload interventions.
-
The authors provide extensive methodological details, execution-order checks, secondary evaluations, and transparent discussion of negative results and limitations.
Critical weaknesses
-
The incremental contribution beyond existing agent-reliability research remains insufficiently established. Stable aggregate rates alongside variable individual outcomes largely follow from averaging, while heterogeneous task success probabilities are expected. Please clarify which findings provide new scientific insight and demonstrate how the proposed measurements change concrete attribution decisions. The distinction between measuring replay variability and establishing its practical consequences should be made more explicit.
-
Section 4.6 and Appendix A.4 rerun tasks end to end, allowing trajectories and candidate sets to change; 45.7% of starting executions are benign. This appears to conflate configuration-level variability with attribution of a fixed historical incident. Please compare this protocol with replay that preserves the original action history, as in AttriGuard (He et al., 2026), and distinguish their estimands.
-
Appendix A.4 adapts ContextCite (Cohen-Wang et al., 2024) and TracLLM (Wang et al., 2025) without sufficient calibration. The former fits binary verdicts using fixed regularization and 24 queries for up to 425 candidates. Please examine budget and regularization sensitivity, clarify which original scoring components are retained, and validate signal recovery under a controlled oracle to separate adaptation limitations from replay noise.
-
Agreement in Section 4.6 can reward uninformative outputs because all-zero scores return the first candidate. Uniform-draw chance correction does not remove this systematic tie-breaking effect, and only three of 26 contested incidents yield informative outputs across all three repetitions for the regression-based adaptation. Please report abstention rates, agreement conditional on informative outputs, and source-identification accuracy, distinguishing switches between valid injected spans from genuinely inconsistent attribution.
-
The outside-payload control in Section 4.7 matches span counts but not token length, position, or occurrence count, and deletion can remove legitimate task information. Its large effect therefore does not isolate source specificity from semantic or structural disruption. Please add matched replacements and task-preserving edits, report utility changes, and audit every modified occurrence to establish what drives the RMS comparison.
-
Execution time is confounded with configuration identity in the main census. The randomized-order check reduces dispersion from 12.94 to 9.85 but covers only 40 previously harmful configurations at temperature zero. Please re-estimate the headline flip rates, band occupancy, and intraclass correlation under randomized, interleaved execution for both temperatures. Persistence of a heterogeneity rejection alone does not validate the original numerical estimates.
-
The attribution and intervention findings remain concentrated on gpt-4o-mini and one attack family. Near-zero harm on another provider, no harmful outcomes on gpt-5-mini, and an attacker-free benchmark do not establish that these findings generalize. Please repeat the core attribution and intervention analyses on another model–attack combination producing sufficient harmful incidents, and distinguish evidence for outcome variability from evidence for attribution reliability.
Questions for authors
-
How would the Section 4.6 results change under replay that preserves the original action history, and which estimand supports attribution of a fixed historical harmful incident?
-
For the Appendix A.4 adaptations, what controlled-oracle and budget/regularization checks separate adaptation error from replay noise, when all-zero scores are treated as abstentions?
-
Does the outside-payload RMS effect persist under task-preserving edits matched for token length, position, and occurrence count, and how does task utility change?
2 comments
(Part 2/2) Done (E20, Table 15; general response): the 45 harmful temperature-0 configurations the earlier run lacked and all 94 at temperature 1, so every harmful configuration has now run interleaved. At both temperatures flip probability and band occupancy do not change detectably: flip −0.018 [−0.055, 0.015] and −0.011 [−0.027, 0.003]; band +0.022 and +0.031, intervals containing zero. The earlier non-unanimity rise (+0.125) does not recur. The intraclass correlation is lower than in the blocked census at both temperatures, a difference untested and confounded with the re-run date (general response); the pooled temperature-0 value (0.460) is a sensitivity set only. Labels: excluding the census's 15 harness re-attempts moves no headline estimand by more than 0.011, and the corrected checker moves them by at most 0.015 with every paired-change interval containing zero; its correlations are 0.638 and 0.546 (Appendix A.5). Partly. We did not repeat the attribution and intervention analyses on a second model–attack pair; Section 5 states this. Outcome variability rests on several arms (census; banking; gpt-oss-20b; tau2-bench and InjecAgent for the dispersion signature only; DeepSeek, which very probably ran at the provider's default temperature, an inference). Attribution reliability rests only on the 26 contested gpt-4o-mini incidents; intervention on the same 26 plus a banking recurrence check (Appendix A.18): the control's ratio is 1.10 [0.79, 1.75] over nine configurations, and 1.13 [0.52, 3.24] (post hoc, descriptive) on the three configurations spared by a check defect that counts any transfer to the attacker as harmful. Only 37 of 540 executions are traced and the user-task effect was not measured. A corrected banking check removes 118 of 300 harmful executions and moves the census-minus-banking correlation difference from −0.125 (all intervals contain zero) to −0.128 (BCa excludes zero), unresolved (Appendix A.5). Qwen3.8-Flash-Next failed the open-weights eligibility screen fixed before any screening execution; the fallback rule then selected gpt-oss-20b, the eligible fallback with the highest no-attack utility. Done: E18's checkpoint replay is that estimand (R3.2). With it as oracle, the top span is identical across three runs on 0.308, 0.231 and 0.192 of incidents for ContextCite, TracLLM and the control, and modal agreement is 0.654, 0.590 and 0.577, against 0.731, 0.513 and 0.474 with the fresh-episode oracle. Accuracy over every returned span is 0.410, 0.487 and 0.372, against 0.256, 0.167 and 0.038 with the fresh-episode oracle, where chance is 0.131; the comparison is descriptive, since candidate lists, oracle and run dates differ. Preserving the history leaves attribution unrepeatable at this budget, while more runs name an injected span. Dropping six ContextCite runs with a re-attempted oracle query gives a top-span identity of 0.273 and an accuracy of 0.292, abstentions counted as failures (Table 12, Appendix A.19). Done (Table 13). Synthetic incidents copy the contested incidents' candidate counts and payload positions; the oracle returns harmful with probability p1 when every payload span is present and p0 otherwise; the three implementations are called unchanged (624 incidents per cell; chance 0.131). The oracle is an assumption, not fitted to the agent. At the greedy flip rate ContextCite, TracLLM and the control reach 0.094, 0.652 and 0.379, and at a harm probability of one half 0.022, 0.095 and 0.020 (noiseless and budget figures in R3.3). Table 13 scores the span every run returns, abstained or not; in the measured runs an all-zero run abstains and never counts as correct (Table 12). The calculation does not predict the measured accuracies: TracLLM's and the control's fall between its noiseless and one-half regimes. Yes, at a level whose interval overlaps the count-matched control's. On the 20 configurations with a feasible replacement, the matched edit's corrected root-mean-square effect is 0.535 [0.317, 0.824] of payload replacement's, against 0.723 [0.467, 1.013] for the count-matched control on the same 20; no difference was tested (Appendix A.17). Utility moves by −0.052 [−0.200, 0.077] under the matched control, against −0.255 [−0.405, −0.115] for the count-matched control on the same 20; the intervals overlap, so that the replacement leaves the user task intact is a property of its design, not a measured result. Payload replacement raises utility by 0.561. Mean harm under the matched control does not change detectably (general response, Table 14, Figure 7).
(Part 1/2) Thank you for detailed requests, most of which are now experiments; each is answered below, including what was done only in part. The general response gives the shared results and the four harness and label problems we disclose; the recorded verdict remains the primary label, with corrected labels in Appendix A.5. R3.1 (contribution). That a replay can differ at temperature zero is known (Section 1). The insight is where the difference sits: over incidents, not in the aggregate (attack success moves 0.003 between conditions while a replay flips 0.087 and 0.141 of the time); after the payload is read (E18); and in attribution accuracy (R3.4, R3.8). Decisions (Section 6): report replay repeatability beside aggregate success; do not act on one flipped verdict, since a matched edit outside the payload moves single configurations by as much as 0.10 to 1.00; compare candidates with a signed contrast over replicated replays (payload first on 17 of 20, three ties); and do not define populations with a three-repeat screen, which misses 10 and 14 harmful configurations. These hold under the corrected labels of Appendix A.5. R3.2 (two estimands). Agreed; the revision measures the history-preserving estimand and defines both (Appendix A.19). For a configuration c, a recorded harmful history h and a span set S, the fresh-episode estimand is the probability that a fresh execution of c with S neutralized ends harmful (what the submission queried); the fixed-history estimand is the probability that the agent, served h verbatim up to and including the first payload-bearing tool return, with S deleted from the re-executed returns, ends harmful (what E18 queries). AttriGuard tests a proposed tool call's necessity under attenuated observations; its target is an action, not a terminal verdict, and we did not run it. The fixed-history estimand is still a probability (general response); R3.8 compares the two oracles. R3.3 (method calibration). Retained: ContextCite keeps random ablation and a sparse linear surrogate, but regresses the terminal binary verdict rather than the response log-probability, with a fixed L1 penalty of 0.05 and no cross-validation; TracLLM keeps K = 2 group splitting and scores the drop from a fresh factual verdict (Appendix A.4). Budget and regularization sensitivity, in the synthetic calculation only (R3.9; noiseless unless stated): ContextCite at penalty 0.05 abstains on 0.809 of incidents at budget 24 and 0.982 at 192; at penalty 0.005 its accuracy rises with budget from 0.209 to 0.744. TracLLM and the control reach 0.871 and 0.910 at budget 24 and 1.000 and 0.996 at 192, but at a harm probability of one half budget 192 does not recover TracLLM (0.094). So ContextCite is limited by the adaptation at this penalty, the other two by verdict noise. We did not re-run ContextCite with a tuned penalty, so its figures describe our adaptation, not the method's ceiling. R3.4 (uninformative outputs). All four are now reported. Abstention (checkpoint oracle, Table 12): 0.513, 0.038 and 0.026 for ContextCite, TracLLM and the control when every score is zero; 0.538, 0.372 and 0.026 when the top score is at most zero. Agreement on informative outputs (fresh-episode oracle): only 3 of 26 incidents have all three repetitions informative for ContextCite and 10 for TracLLM, with agreements of 0.667 and 0.400; the control's binary score saturates (Appendix A.4). With the checkpoint oracle, accuracy over informative runs is 0.658, 0.493 and 0.382. Accuracy: with the checkpoint oracle and abstentions counted as failures, in the general response; over every returned span, in R3.8. Switches between valid injected spans (fresh-episode runs, Appendix A.4): 0.077 of ContextCite's pairs and 0.064 of TracLLM's under varying seeds, and no pair otherwise. Section 4.6 now notes that overall agreement can reward uninformative runs. R3.5 (unmatched control). Done in E19 (Appendix A.19; results in the general response). The submitted control deleted count-matched spans and cut utility by 0.279 (Appendix A.17). E19 replaces content outside the payload with neutral filler of equal token length, the replaced size within 10 percent of the payload's; a span is ineligible if it overlaps user-task ground truth, the prompt, the payload or the injection's goal, changes its occurrence count across regenerations, or first occurs after the payload's last return. Six of 26 configurations had no feasible set. Occurrence audit: every arm consumed the payload in every execution, the factual and matched arms left it byte-intact, and goal-text fragments survived in 26 of 520 payload-arm executions, none harmful. Utility and magnitude: R3.10. Semantic and structural effects are still not separated (Section 5). Each E19 contrast uses E19's own factual arm, which drifted between runs.