acceptodds
Under review as a conference paper at ICLR 2027

HOW RELIABLY DOES REPLAY REPRODUCE A TOOL-USING AGENT’S HARMFUL OUTCOME?

Abstract

Counterfactual replay is a common instrument in replay-based attribution for tying a tool-using language agent's harmful action to a span of its context: neutralize a candidate span, re-execute, and read the outcome. Whether that verdict is a value or a draw, and how concentrated the draw is, has not been measured. We report a census. On the AgentDojo workspace suite with a stock prompt-injection attack and gpt-4o-mini, all 120 configurations are executed 20 times in each of two decoding conditions, 4,800 executions with no failed runs. The benchmark-level attack success rate is 0.563 under greedy decoding and 0.566 at temperature 1, against a round-to-round standard deviation of 0.019 and 0.026 – steadiness that is a property of averaging rather than evidence that an individual outcome repeats. In each condition a single replay of an observed harmful execution returns the opposite verdict 0.087 of the time (95% CI [0.053, 0.124]) under greedy decoding and 0.141 ([0.091, 0.189]) at temperature 1, and 0.306 ([0.183, 0.389]) and 0.404 ([0.292, 0.534]) of harmful configurations have an observed harm frequency strictly inside (0.1, 0.9) over 20 identical repeats. An aggregate error bar therefore does not bound the per-incident quantity, and on this census the two move independently. Trajectory comparison localizes the first divergence to a model output in 91 of 91 divergent pairs with byte-identical payloads, the population is far more polarized than a common-rate model allows, and that excess appears again on a second AgentDojo task suite, on a benchmark with no attacker, and on an independently implemented attacked benchmark whose checker, attack and decoding condition are none of ours. Executing the counterfactual side as well, over 6,120 preregistered executions, four interventions on the injected payload leave no harmful outcome on any contested configuration, while deleting count-matched spans that never touch it moves a configuration's harm probability almost as far – a noise-corrected root-mean-square of 0.44 [0.32, 0.54] against 0.57 [0.48, 0.65]. Sensitivity of that order recurs on a second task suite whose nine eligible configurations, from six user tasks, were selected independently: the ratio there is 1.10 [0.79, 1.75], passing a point rule fixed in advance at [0.20, 1.40], with the direction reversed – harm rises on seven of nine, each to 20 of 20, where it split eleven to fourteen here. Unsigned change alone therefore does not identify the injected source. This sensitivity is distinct from failure of signed candidate ranking and persists regardless of the replay budget used to estimate it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, AI safety, prompt injection, counterfactual replay, causal attribution, reproducibility, agent evaluation, tool-using agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TLSTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the repeatability of harmful outcomes in tool-using language agents and its implications for replay-based attribution. A census of 120 AgentDojo configurations, repeated 20 times under two decoding conditions, measures replay disagreement alongside aggregate attack success. Additional experiments examine attribution stability, payload neutralization, and edits outside the injected payload. The results show that stable aggregate rates can coexist with variable individual outcomes and substantial sensitivity to edits outside the injected source.

Critical strengths
  1. The primary census systematically evaluates every configuration in the selected suite, reducing the selection problems associated with screening configurations using only a few repetitions.

  2. The paper distinguishes aggregate attack success from individual replay repeatability and complements observational measurements with explicit payload interventions.

  3. The authors provide extensive methodological details, execution-order checks, secondary evaluations, and transparent discussion of negative results and limitations.

Critical weaknesses
  1. The incremental contribution beyond existing agent-reliability research remains insufficiently established. Stable aggregate rates alongside variable individual outcomes largely follow from averaging, while heterogeneous task success probabilities are expected. Please clarify which findings provide new scientific insight and demonstrate how the proposed measurements change concrete attribution decisions. The distinction between measuring replay variability and establishing its practical consequences should be made more explicit.

  2. Section 4.6 and Appendix A.4 rerun tasks end to end, allowing trajectories and candidate sets to change; 45.7% of starting executions are benign. This appears to conflate configuration-level variability with attribution of a fixed historical incident. Please compare this protocol with replay that preserves the original action history, as in AttriGuard (He et al., 2026), and distinguish their estimands.

  3. Appendix A.4 adapts ContextCite (Cohen-Wang et al., 2024) and TracLLM (Wang et al., 2025) without sufficient calibration. The former fits binary verdicts using fixed regularization and 24 queries for up to 425 candidates. Please examine budget and regularization sensitivity, clarify which original scoring components are retained, and validate signal recovery under a controlled oracle to separate adaptation limitations from replay noise.

  4. Agreement in Section 4.6 can reward uninformative outputs because all-zero scores return the first candidate. Uniform-draw chance correction does not remove this systematic tie-breaking effect, and only three of 26 contested incidents yield informative outputs across all three repetitions for the regression-based adaptation. Please report abstention rates, agreement conditional on informative outputs, and source-identification accuracy, distinguishing switches between valid injected spans from genuinely inconsistent attribution.

  5. The outside-payload control in Section 4.7 matches span counts but not token length, position, or occurrence count, and deletion can remove legitimate task information. Its large effect therefore does not isolate source specificity from semantic or structural disruption. Please add matched replacements and task-preserving edits, report utility changes, and audit every modified occurrence to establish what drives the RMS comparison.

  6. Execution time is confounded with configuration identity in the main census. The randomized-order check reduces dispersion from 12.94 to 9.85 but covers only 40 previously harmful configurations at temperature zero. Please re-estimate the headline flip rates, band occupancy, and intraclass correlation under randomized, interleaved execution for both temperatures. Persistence of a heterogeneity rejection alone does not validate the original numerical estimates.

  7. The attribution and intervention findings remain concentrated on gpt-4o-mini and one attack family. Near-zero harm on another provider, no harmful outcomes on gpt-5-mini, and an attacker-free benchmark do not establish that these findings generalize. Please repeat the core attribution and intervention analyses on another model–attack combination producing sufficient harmful incidents, and distinguish evidence for outcome variability from evidence for attribution reliability.

Questions for authors
  1. How would the Section 4.6 results change under replay that preserves the original action history, and which estimand supports attribution of a fixed historical harmful incident?

  2. For the Appendix A.4 adaptations, what controlled-oracle and budget/regularization checks separate adaptation error from replay noise, when all-zero scores are treated as abstentions?

  3. Does the outside-payload RMS effect persist under task-preserving edits matched for token length, position, and occurrence count, and how does task utility change?

2 comments

(Part 1/2) Thank you for detailed requests, most of which are now experiments; each is answered below, including what was done only in part. The general response gives the shared results and the four harness and label problems we disclose; the recorded verdict remains the primary label, with corrected labels in Appendix A.5. R3.1 (contribution).\textbf{R3.1 (contribution).} That a replay can differ at temperature zero is known (Section 1). The insight is where the difference sits: over incidents, not in the aggregate (attack success moves 0.003 between conditions while a replay flips 0.087 and 0.141 of the time); after the payload is read (E18); and in attribution accuracy (R3.4, R3.8). Decisions (Section 6): report replay repeatability beside aggregate success; do not act on one flipped verdict, since a matched edit outside the payload moves single configurations by as much as 0.10 to 1.00; compare candidates with a signed contrast over replicated replays (payload first on 17 of 20, three ties); and do not define populations with a three-repeat screen, which misses 10 and 14 harmful configurations. These hold under the corrected labels of Appendix A.5. R3.2 (two estimands).\textbf{R3.2 (two estimands).} Agreed; the revision measures the history-preserving estimand and defines both (Appendix A.19). For a configuration c, a recorded harmful history h and a span set S, the fresh-episode estimand is the probability that a fresh execution of c with S neutralized ends harmful (what the submission queried); the fixed-history estimand is the probability that the agent, served h verbatim up to and including the first payload-bearing tool return, with S deleted from the re-executed returns, ends harmful (what E18 queries). AttriGuard tests a proposed tool call's necessity under attenuated observations; its target is an action, not a terminal verdict, and we did not run it. The fixed-history estimand is still a probability (general response); R3.8 compares the two oracles. R3.3 (method calibration).\textbf{R3.3 (method calibration).} Retained: ContextCite keeps random ablation and a sparse linear surrogate, but regresses the terminal binary verdict rather than the response log-probability, with a fixed L1 penalty of 0.05 and no cross-validation; TracLLM keeps K = 2 group splitting and scores the drop from a fresh factual verdict (Appendix A.4). Budget and regularization sensitivity, in the synthetic calculation only (R3.9; noiseless unless stated): ContextCite at penalty 0.05 abstains on 0.809 of incidents at budget 24 and 0.982 at 192; at penalty 0.005 its accuracy rises with budget from 0.209 to 0.744. TracLLM and the control reach 0.871 and 0.910 at budget 24 and 1.000 and 0.996 at 192, but at a harm probability of one half budget 192 does not recover TracLLM (0.094). So ContextCite is limited by the adaptation at this penalty, the other two by verdict noise. We did not re-run ContextCite with a tuned penalty, so its figures describe our adaptation, not the method's ceiling. R3.4 (uninformative outputs).\textbf{R3.4 (uninformative outputs).} All four are now reported. Abstention\textbf{Abstention} (checkpoint oracle, Table 12): 0.513, 0.038 and 0.026 for ContextCite, TracLLM and the control when every score is zero; 0.538, 0.372 and 0.026 when the top score is at most zero. Agreement on informative outputs\textbf{Agreement on informative outputs} (fresh-episode oracle): only 3 of 26 incidents have all three repetitions informative for ContextCite and 10 for TracLLM, with agreements of 0.667 and 0.400; the control's binary score saturates (Appendix A.4). With the checkpoint oracle, accuracy over informative runs is 0.658, 0.493 and 0.382. Accuracy:\textbf{Accuracy:} with the checkpoint oracle and abstentions counted as failures, in the general response; over every returned span, in R3.8. Switches between valid injected spans\textbf{Switches between valid injected spans} (fresh-episode runs, Appendix A.4): 0.077 of ContextCite's pairs and 0.064 of TracLLM's under varying seeds, and no pair otherwise. Section 4.6 now notes that overall agreement can reward uninformative runs. R3.5 (unmatched control).\textbf{R3.5 (unmatched control).} Done in E19 (Appendix A.19; results in the general response). The submitted control deleted count-matched spans and cut utility by 0.279 (Appendix A.17). E19 replaces content outside the payload with neutral filler of equal token length, the replaced size within 10 percent of the payload's; a span is ineligible if it overlaps user-task ground truth, the prompt, the payload or the injection's goal, changes its occurrence count across regenerations, or first occurs after the payload's last return. Six of 26 configurations had no feasible set. Occurrence audit: every arm consumed the payload in every execution, the factual and matched arms left it byte-intact, and goal-text fragments survived in 26 of 520 payload-arm executions, none harmful. Utility and magnitude: R3.10. Semantic and structural effects are still not separated (Section 5). Each E19 contrast uses E19's own factual arm, which drifted between runs.

(Part 2/2) R3.6 (execution time).\textbf{R3.6 (execution time).} Done (E20, Table 15; general response): the 45 harmful temperature-0 configurations the earlier run lacked and all 94 at temperature 1, so every harmful configuration has now run interleaved. At both temperatures flip probability and band occupancy do not change detectably: flip −0.018 [−0.055, 0.015] and −0.011 [−0.027, 0.003]; band +0.022 and +0.031, intervals containing zero. The earlier non-unanimity rise (+0.125) does not recur. The intraclass correlation is lower than in the blocked census at both temperatures, a difference untested and confounded with the re-run date (general response); the pooled temperature-0 value (0.460) is a sensitivity set only. Labels: excluding the census's 15 harness re-attempts moves no headline estimand by more than 0.011, and the corrected checker moves them by at most 0.015 with every paired-change interval containing zero; its correlations are 0.638 and 0.546 (Appendix A.5). R3.7 (one model, one attack).\textbf{R3.7 (one model, one attack).} Partly. We did not repeat the attribution and intervention analyses on a second model–attack pair; Section 5 states this. Outcome variability rests on several arms (census; banking; gpt-oss-20b; tau2-bench and InjecAgent for the dispersion signature only; DeepSeek, which very probably ran at the provider's default temperature, an inference). Attribution reliability rests only on the 26 contested gpt-4o-mini incidents; intervention on the same 26 plus a banking recurrence check (Appendix A.18): the control's ratio is 1.10 [0.79, 1.75] over nine configurations, and 1.13 [0.52, 3.24] (post hoc, descriptive) on the three configurations spared by a check defect that counts any transfer to the attacker as harmful. Only 37 of 540 executions are traced and the user-task effect was not measured. A corrected banking check removes 118 of 300 harmful executions and moves the census-minus-banking correlation difference from −0.125 (all intervals contain zero) to −0.128 (BCa excludes zero), unresolved (Appendix A.5). Qwen3.8-Flash-Next failed the open-weights eligibility screen fixed before any screening execution; the fallback rule then selected gpt-oss-20b, the eligible fallback with the highest no-attack utility. R3.8 (fixed-incident estimand).\textbf{R3.8 (fixed-incident estimand).} Done: E18's checkpoint replay is that estimand (R3.2). With it as oracle, the top span is identical across three runs on 0.308, 0.231 and 0.192 of incidents for ContextCite, TracLLM and the control, and modal agreement is 0.654, 0.590 and 0.577, against 0.731, 0.513 and 0.474 with the fresh-episode oracle. Accuracy over every returned span is 0.410, 0.487 and 0.372, against 0.256, 0.167 and 0.038 with the fresh-episode oracle, where chance is 0.131; the comparison is descriptive, since candidate lists, oracle and run dates differ. Preserving the history leaves attribution unrepeatable at this budget, while more runs name an injected span. Dropping six ContextCite runs with a re-attempted oracle query gives a top-span identity of 0.273 and an accuracy of 0.292, abstentions counted as failures (Table 12, Appendix A.19). R3.9 (controlled-oracle checks).\textbf{R3.9 (controlled-oracle checks).} Done (Table 13). Synthetic incidents copy the contested incidents' candidate counts and payload positions; the oracle returns harmful with probability p1 when every payload span is present and p0 otherwise; the three implementations are called unchanged (624 incidents per cell; chance 0.131). The oracle is an assumption, not fitted to the agent. At the greedy flip rate ContextCite, TracLLM and the control reach 0.094, 0.652 and 0.379, and at a harm probability of one half 0.022, 0.095 and 0.020 (noiseless and budget figures in R3.3). Table 13 scores the span every run returns, abstained or not; in the measured runs an all-zero run abstains and never counts as correct (Table 12). The calculation does not predict the measured accuracies: TracLLM's and the control's fall between its noiseless and one-half regimes. R3.10 (task-preserving edits).\textbf{R3.10 (task-preserving edits).} Yes, at a level whose interval overlaps the count-matched control's. On the 20 configurations with a feasible replacement, the matched edit's corrected root-mean-square effect is 0.535 [0.317, 0.824] of payload replacement's, against 0.723 [0.467, 1.013] for the count-matched control on the same 20; no difference was tested (Appendix A.17). Utility moves by −0.052 [−0.200, 0.077] under the matched control, against −0.255 [−0.405, −0.115] for the count-matched control on the same 20; the intervals overlap, so that the replacement leaves the user task intact is a property of its design, not a measured result. Payload replacement raises utility by 0.561. Mean harm under the matched control does not change detectably (general response, Table 14, Figure 7).

Reviewer_UgEeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reproducibility of harmful outcomes when replaying tool-using language agents, motivated by counterfactual attribution methods that neutralize context spans and re-execute the agent. The primary experiment repeats AgentDojo workspace configurations under two decoding conditions. A replay of an observed harmful execution returns a benign verdict with estimated probabilities of 0.087 and 0.141, despite relatively stable aggregate attack success rates. Further experiments examine configuration heterogeneity, attribution self-agreement, and the effects of payload neutralization versus deletion outside the payload. The paper argues that attribution should account for replay uncertainty and that unsigned outcome changes alone cannot identify an injected source.

Critical strengths
  1. The paper addresses an important problem for agent security and debugging. Measuring incident-level repeatability provides information that aggregate attack success rates do not convey.
  2. The complete census is a useful design choice. Executing every configuration avoids excluding harmful or variable configurations missed by a shallow screen. The distinction between configuration-weighted and harmful-execution-weighted quantities is also helpful.
  3. The intervention experiments add useful evidence beyond repeated executions of the original input. Running both payload-targeting edits and deletions outside the payload allows the authors to measure how these changes affect harm probabilities.
Critical weaknesses
  1. I feel that the findings are not strong enough to establish a substantial problem for practical attribution. The paper acknowledges that earlier work has already observed replay nondeterminism, and stable averages can naturally coexist with variable individual outcomes. There is value in measuring the size of this effect, but the paper leaves open how often it leads to an incorrect source being identified.
  2. The attribution experiment in Section 4.6 is difficult to interpret. Each invocation begins with a fresh end-to-end execution, and 45.7% of those executions return benign. The experiment therefore changes the execution being explained as well as repeating the attribution procedure. So I find it hard to tell how much disagreement comes from noisy counterfactual comparisons, changes in the trajectory, or the handling of zero scores.
  3. The control experiment supports a relatively narrow conclusion. Deleting legitimate task content can change an agent’s behavior, even when that content lies outside the injected payload. The large unsigned effects are therefore not especially surprising on their own.
Questions for authors

I have some questions:

  1. Could the attribution experiment hold the observed incident fixed?

  2. How often does the control cause an incorrect source selection?

1 comment

Thank you for pressing on what an attribution method can and cannot be shown to do. The revision holds the incident fixed by replaying from a checkpoint and scores attribution accuracy against it (E18), and adds a length-matched control designed to preserve the user task (E19). The general response gives both, and the four harness and label problems we disclose; the recorded verdict remains the primary label, with corrected labels in Appendix A.5. R2.1 (practical value).\textbf{R2.1 (practical value).} We now measure it (E18, Table 12). With the checkpoint replay as oracle, the three implementations name an injected span on 0.321 to 0.474 of runs (general response); the rest abstain or name a span outside the payload, so 0.53 to 0.68 of runs miss the injected source; among runs that name a span, 0.342, 0.507 and 0.618 name a non-injected one. In a calculation holding each implementation fixed, verdict noise alone lowers TracLLM's and our control's accuracy from 0.871 and 0.910 to 0.652 and 0.379 (Table 13). The calculation does not predict the measured accuracies, and because a run's seed drives both the method and its replays, the measured errors are not split by source. Between the payload and an edit outside it, a signed contrast at twenty replays never picks the outside edit (R2.5). R2.2 (Section 4.6 experiment).\textbf{R2.2 (Section 4.6 experiment).} The revision separates the three. Trajectory changes: E18 replays from a checkpoint after the first injected tool return, so every oracle query starts from the recorded incident rather than from a fresh episode, which came back benign 0.457 of the time. Noisy comparisons remain: from the checkpoint, some pair of the nine unablated baseline replays of a contested incident disagrees on 0.692 of incidents. Zero scores are now abstentions that never count as correct: ContextCite abstains on 0.513 of runs, TracLLM on 0.038, the control on 0.026. ContextCite's zero scores come largely from our adaptation: it abstains on 0.809 of synthetic incidents even with a noiseless oracle, and since we did not re-run it with a tuned penalty, its figures describe our adaptation. With the pre-payload trajectory fixed, the top span is still identical across three runs on only 0.19 to 0.31 of incidents (Table 12). R2.3 (scope of the control).\textbf{R2.3 (scope of the control).} Agreed, and the conclusion is now narrower. Section 4.7 states that the count-matched control removed content the user task needed (utility fell by 0.279) and that an outcome flip alone does not name the injected span. The new matched control (general response) supports a narrow conclusion: mean harm does not change detectably, yet single configurations move by as much as 0.10 to 1.00, so a decision read from a single flip or an unsigned effect is at risk, while a signed contrast at twenty replays could at worst tie on these records. It does not show that the matched content is irrelevant in every respect, that its effect is smaller than the count-matched control's (untested), or anything about larger candidate sets (Section 5). E19's factual arm drifted between runs, so each E19 contrast uses E19's own factual arm. R2.4 (fixed incident).\textbf{R2.4 (fixed incident).} Yes. E18 serves the recorded conversation verbatim up to the model's first call after reading the payload; an offline gate confirmed for all 179 incidents that this conversation is identical to the record. Results are in the general response and R2.2. R2.5 (incorrect source selection).\textbf{R2.5 (incorrect source selection).} In a signed comparison of the payload against the control, never on these records, because none could occur: a reversal needs a harmful execution with the payload removed or replaced, and every such arm returned 0 of 520. Against the matched control the payload is first on 17 of 20 and tied on 3 (p = 0.000015, Table 14). Under a single-flip reading the risk is real: the matched control's harm rate differs from the factual arm's in 18 of the 20 configurations (higher in 7, lower in 11), so a procedure naming whichever edited span flips one replay could name the control span. Larger candidate sets and other attribution procedures remain unmeasured (Section 5).

Reviewer_af8DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how reliably counterfactual replay can reproduce harmful outcomes in tool-using LLM agents and what this means for replay-based attribution methods. Through extensive experiments on AgentDojo and other benchmarks, the authors show that individual agent executions can produce inconsistent outcomes even when overall attack success rates remain stable. They also find that modifying context outside the malicious payload can significantly change an agent's behavior, making it difficult to identify the attack source based on outcome changes alone. Overall, the paper highlights important limitations of relying on individual replays for understanding and attributing harmful agent behavior.

Critical strengths
  1. Well-designed and thorough experiments: I liked that the authors conducted a large number of repeated executions and carefully analyzed the results using different statistical measures. The paper also includes several additional experiments to check whether the findings hold across different settings, which makes the analysis more convincing.

  2. Interesting and practically relevant problem: The paper highlights an important issue that is often overlooked in replay-based attribution methods. I particularly liked how the authors show that stable overall attack success rates do not necessarily mean that individual agent executions are reproducible. This is a useful finding for understanding the reliability of existing attribution approaches.

Critical weaknesses
  1. No clear evidence of incorrect attribution: The paper does a good job showing that replaying the same agent can lead to different outcomes. However, I'm not convinced that these differences actually cause attribution methods to identify the wrong source of an attack. In fact, in the authors' own experiments, the actual injected payload was never ranked strictly below the outside-payload control. I would have liked to see an experiment where replay variability actually leads to incorrect attribution. Without that, I feel the paper does not fully demonstrate the practical impact of the problem it studies.

  2. Questions about the outside-payload control: I'm not fully convinced by how the authors designed this experiment. They remove parts of the context outside the malicious payload and show that the agent's behavior changes significantly. But these parts could contain useful information for the user's original task. For example, removing important task instructions could make the agent more likely to follow the malicious instructions instead. So, I don't think it's surprising that removing such content changes the outcome. I would have liked to see a better control experiment where the removed content is clearly irrelevant to the task.

  3. Concerns about the attribution method evaluation: I found the evaluation of ContextCite and TracLLM a bit difficult to interpret. The authors modify both methods to work with their setup, so it's unclear whether the instability comes from replay variability or from how these methods were adapted. Also, many runs produce no meaningful attribution scores but still return a candidate through tie-breaking. I would have liked to see a more direct evaluation of how often these methods correctly identify the actual injected content, especially since the benchmark already provides this ground truth.

  4. Limited generalizability: Most of the main experiments use GPT-4o-mini with a single prompt-injection attack. Although the authors try a few other models and benchmarks, the results are quite different. For example, DeepSeek has a very low attack success rate, while GPT-5-mini is not successfully attacked in any of the tested cases. This makes me wonder whether the findings are specific to the older model and attack being used. I would have liked to see a broader evaluation with newer models and different successful attacks to better understand how common this problem really is.

  5. Poor presentation and readability: I found the paper quite difficult to follow at times, mainly because of how the results are presented. For example, Table 1 is extremely dense and almost unreadable. I also feel that several important results could have been explained much better through simple visualizations rather than lengthy text and numerical comparisons. While the authors have conducted a lot of experiments, the presentation makes it unnecessarily difficult to understand the key findings and takeaways. I would strongly encourage the authors to improve the organization and use more intuitive figures to communicate their results.

Questions for authors
  1. The authors show that the first visible divergence occurs in a model-generated message. Is there any evidence explaining what actually causes this divergence, particularly under temperature-zero decoding?

  2. The randomized execution-order experiment reduces the measured dispersion. Why was this experiment conducted only at temperature zero? Would the same behavior appear at temperature one?

  3. The attribution experiments focus primarily on 26 configurations already identified as having intermediate harm probabilities. How representative are these configurations of harmful incidents overall?

  4. Would the results change if replay started from a fixed checkpoint immediately before the injected content, rather than restarting the entire agent execution?

  5. Since previous work has already observed nondeterminism at temperature zero, could the authors clarify the main new insight beyond quantitatively characterizing this variability?

3 comments

(Part 1/2) Thank you for a careful, engaged review. In brief, on your two central questions: the revision now measures attribution accuracy against a fixed incident and shows in a calculation that verdict noise alone lowers it (R1.1); and what is new beyond temperature-zero nondeterminism is where the variability sits and what it does to attribution decisions (R1.10). The general response gives the new experiments and the four harness and label problems we disclose (re-attempts, a count-based check, a divergence-script bug, an unstored fingerprint); the recorded verdict remains the primary label, with corrected labels in Appendix A.5. R1.1 (incorrect attribution).\textbf{R1.1 (incorrect attribution).} Agreed: the submission did not show this, so the revision measures it. With the fixed-checkpoint replay as oracle, the three implementations name an injected span on 0.321 to 0.474 of runs, against 0.132 by chance (general response, Table 12). To isolate replay noise, a calculation holds each implementation fixed and varies only the oracle: verdict noise alone lowers TracLLM's and our control's accuracy from 0.871 and 0.910 with a noiseless oracle to 0.652 and 0.379 at the greedy flip rate (Table 13). It does not predict the measured accuracies, and we do not claim that replay noise caused any particular misattribution. You are right that the payload is never ranked strictly below the control. On these records it could not be: no execution with the payload removed or replaced was harmful (0 of 520), so a control could at most tie. Against the new matched control the payload is first on 17 of 20 configurations, with three ties. What is at risk is a decision read from one outcome flip or an unsigned effect: under the matched edit, single configurations move from 0.10 to 1.00 and from 0.85 to 0.00 (Section 4.7, Figure 7). R1.2 (control design).\textbf{R1.2 (control design).} Agreed. The submitted control matched span count, not size, and removed content the user task needed: utility fell by 0.279 [0.165, 0.410] (Appendix A.17), which Section 4.7 now states. E19 adds a control designed to preserve the user task (general response); its eligibility rules exclude, for example, spans holding a ground-truth argument of the user task or the injection's goal text. The rules are chosen to leave the user task intact; they do not prove the replaced content irrelevant. Sensitivity outside the payload persists: on the 20 configurations with a feasible replacement, the matched edit moves harm 0.535 [0.317, 0.824] as far as payload replacement in root-mean-square terms, against 0.723 [0.467, 1.013] for the count-matched control on the same 20; the intervals overlap and no difference was tested. The factual arm drifted between the two runs (0.209 against 0.110 expected), so each E19 contrast uses E19's own factual arm. R1.3 (attribution methods).\textbf{R1.3 (attribution methods).} Both, and the revision separates them with a controlled-oracle calculation that calls the implementations unchanged (Table 13). ContextCite as adapted (binary verdict, fixed L1 penalty 0.05, no cross-validation) abstains on 0.809 of synthetic incidents even with a noiseless oracle, so its limit is the adaptation. TracLLM and our control work with a noiseless oracle and lose accuracy to verdict noise (R1.1). Tie-breaking: all-zero scores are now abstentions and never count as correct; with the checkpoint oracle, ContextCite abstains on 0.513 of runs, TracLLM on 0.038 and the control on 0.026 (Table 12). Accuracy against the ground truth is in R1.1. ContextCite's solver diverged in 3 of 9 runs on incidents with more than 400 candidates; re-solving moves the correct count from 32 to 33. We did not re-run ContextCite with a tuned penalty, so its figures describe our adaptation, not the method's ceiling. R1.4 (generalizability).\textbf{R1.4 (generalizability).} Partly. The attribution and intervention experiments were not repeated on a second model–attack pair: they stay on one model, one attack family and one suite, and whether a contemporary model under a successful attack behaves alike is open (Section 5). We added an open-weights arm, which measures outcome variability, not attribution (general response). Its eligibility rule, the decision to screen Qwen3.8-Flash-Next first and the fallback list were fixed before any screening execution; Qwen3.8-Flash-Next was ineligible, and the rule selected gpt-oss-20b, the eligible fallback with the highest no-attack utility. The other arms each carry a stated limit (Section 4.5): banking is a second suite of the same benchmark; InjecAgent missed its minimum yield; tau2-bench, with no attacker, supports the dispersion statistic only; gpt-5-mini was compromised in 0 of 48 executions against 41 of 48 for gpt-4o-mini. DeepSeek very probably ran at the provider's default temperature, because the installed benchmark omits a configured 0 from the request; this is an inference (Appendix A.11).

(Part 2/2) R1.5 (presentation).\textbf{R1.5 (presentation).} Done in part. Table 1 is split into (a) the aggregate an evaluation reports and (b) the per-incident quantities. Two new figures carry results that would otherwise be number-heavy text: Figure 6 plots each incident's checkpoint-replay harm rate against its census rate (Section 4.1), and Figure 7 shows harm per contested configuration under the factual arm, the matched replacement and payload replacement (Section 4.7). The new results enter the introduction and each results subsection with their headline numbers first. We did not add a conceptual diagram. R1.6 (cause of divergence).\textbf{R1.6 (cause of divergence).} Not a cause, but the revision narrows where it arises and corrects one finding. The submitted divergence script could not see tool messages; corrected, an assistant message comes first in 90 of the 91 divergent configurations, the one exception being a harness re-attempt. The first difference falls at the first model call, whose request is byte-identical across repeats, in 0.196 of pairs at temperature 0 and 0.404 at temperature 1 (per-configuration averages); later ones follow requests that also differ in provider-assigned tool-call identifiers, so the traces cannot separate those from serving nondeterminism (Appendix A.6). The census did not store the provider's system fingerprint. Post hoc, in the checkpoint replays, it separates outcomes perfectly in 16 of 29 mixed incidents at temperature 0, against 7.6 expected (p = 0.00016), and in 13 one fingerprint carries both: an association, not evidence of routing. On open weights, batching is one source of temperature-0 variation, not the only one (general response). R1.7 (order at temperature 1).\textbf{R1.7 (order at temperature 1).} Only in part. E20 runs it at both temperatures (general response, Table 15): χ2/(n−1)\chi^2/(n-1) falls from 11.56 to 9.62 at temperature 0 but not at temperature 1 (10.42 to 10.50), and the intraclass correlation falls less there. At both temperatures flip probability and band occupancy do not change detectably (flip at temperature 1 −0.011 [−0.027, 0.003]), and the earlier rise in non-unanimity does not recur. Harm shows no trend over rounds. Sections 1 and 4.2 now report the interleaved correlation beside the blocked one; the difference is untested and confounded with the re-run date. R1.8 (the 26 configurations).\textbf{R1.8 (the 26 configurations).} They are not representative of all incidents; they stand for incidents in which the user task and the injected task compete, not saturated ones. At temperature 0 the 26 carry 251 of the census's 1,352 harmful executions, and in them the agent completes the user task in 0.444 of executions against 0.091 in the other harmful configurations (Section 4.6, Appendix A.19). The replay finding does not rest on the 26: E18 covers every harmful configuration (general response). The populations were selected under the recorded verdict; under the corrected checker one more configuration would have qualified as contested (Appendix A.5). R1.9 (fixed-checkpoint replay).\textbf{R1.9 (fixed-checkpoint replay).} Yes, and E18 does this for every harmful configuration (general response). Recorded assistant messages are served verbatim up to the model's first call after reading the payload; an offline gate confirmed for all 179 incidents that this conversation is identical to the record. Variability remains (Table 11, Figure 6): at temperature 0 the unanimous share is 0.659, against 0.674 expected from independent draws at the census rate (0.575 against 0.520 at temperature 1), and the checkpoint raises mean harm by 0.057 and 0.078. When the first live call is the injected goal function, the replay ends harmful in 0.951 and 0.946 of cases (descriptive, since the first action is part of the outcome). Attribution against this checkpoint is in R1.1. R1.10 (what is new).\textbf{R1.10 (what is new).} That a replay can differ at temperature zero is known, and the introduction now says so. What the paper adds is where the difference sits and what it does to decisions: Over incidents, not in the aggregate:\textbf{Over incidents, not in the aggregate:} attack success moves by 0.003 between decoding conditions while a single replay flips 0.087 and 0.141 of the time, concentrated on some configurations (common-rate null rejected at χ2/(n−1)\chi^2/(n-1) = 12.03 and 10.42). In the episode:\textbf{In the episode:} fixing everything up to the first injected tool return still leaves 0.341 and 0.425 of incidents returning both verdicts, so the variation arises after the payload is read. In attribution:\textbf{In attribution:} accuracy against that checkpoint, and what verdict noise alone does to it (R1.1). In screening:\textbf{In screening:} a three-repeat first pass misses 10 and 14 harmful configurations. The consequences (Section 6): report replay repeatability beside aggregate success, and do not read one flipped verdict as naming the injected source; a signed contrast at twenty replays does separate it on these records (Section 4.7).

I appreciate the author's rebuttal and effort, which addressed some of my concerns. However, my main concern remains that the paper still does not convincingly demonstrate how replay variability leads to incorrect source attribution in practice, especially since the experiments do not show incorrect source ranking. I am still concerned about the validity of the outside-payload control and the limited generalizability of the findings. So, I am going to maintain my score.

Comments on the paper

1 comment

Thank you for the three reviews. All three asked what is new beyond known temperature-0 nondeterminism and whether it changes attribution in practice (R1.1, R1.10, R2.1, R3.1). Their requests: attribution of a fixed incident and the outside-payload control (all three), execution order and generality beyond gpt-4o-mini and one attack (af8D, TLST), and presentation (af8D; Table 1 is now split, and Figures 6 and 7 are new).

1. New experiments.\textbf{1. New experiments.} The revision adds three experiments and an open-weights arm: E18, fixed-checkpoint replay.\textbf{E18, fixed-checkpoint replay.} Every harmful configuration (85 at temperature 0, 94 at temperature 1) is replayed from a checkpoint after the first tool return carrying the payload; 0.341 and 0.425 of incidents still return both verdicts over ten replays. With this replay as oracle, on the 26 contested incidents, ContextCite, TracLLM and our budget-matched control name an injected span on 0.321, 0.474 and 0.372 of runs (abstentions counted as failures), against 0.132 by chance; the intervals overlap and we do not rank the methods. E19, length-matched control (1,440 executions).\textbf{E19, length-matched control (1,440 executions).} A control designed to preserve the user task: content outside the payload is replaced with filler of the same token length. On the 20 configurations with a feasible replacement, mean harm changes by 0.000 [−0.168, 0.172], not detectably, yet single configurations move from 0.10 to 1.00 and from 0.85 to 0.00. Utility changes by −0.052 [−0.200, 0.077], overlapping the count-matched control's interval, so task preservation is a property of the design, not a measured result. E19's factual arm drifted from the earlier run, so each E19 contrast uses E19's own factual arm. E20, interleaved order at both temperatures.\textbf{E20, interleaved order at both temperatures.} It covers the 45 harmful temperature-0 configurations the earlier interleaved run lacked and all 94 at temperature 1, so every harmful configuration has now run interleaved. Flip and band changes have intervals containing zero; dispersion falls at temperature 0 (χ2/(n−1)\chi^2/(n-1) 11.56 to 9.62) but not at temperature 1 (10.42 to 10.50); on the same configurations the intraclass correlation is lower than in the blocked census (0.598 to 0.448 at temperature 0, 0.530 to 0.490 at temperature 1), a difference not tested and confounded with the re-run date. Open weights.\textbf{Open weights.} The eligibility rule, the screening order and the fallback list were fixed before any screening execution. Qwen3.8-Flash-Next was attacked successfully in 3 of 360 executions and was ineligible; the rule then selected gpt-oss-20b, the eligible fallback with the highest no-attack utility (7,200 executions). Serving one sequence at a time lowers its temperature-0 flip from 0.189 to 0.075: batching is one source of temperature-0 variation, not the only one.

2. Disclosed problems.\textbf{2. Disclosed problems.} We disclose four harness and label problems found in auditing the runs (Appendix A.5): Silent re-attempts.\textbf{Silent re-attempts.} The harness silently re-attempted 15 of 4,800 census executions; excluding them moves no headline estimand by more than 0.011. Security checks.\textbf{Security checks.} Two checks mislabel outcomes: a count-based check in the census and, in the banking arm, a check that counts any transfer to the attacker as harmful. Divergence script.\textbf{Divergence script.} The script was blind to tool messages (Appendix A.6). Provider fingerprint.\textbf{Provider fingerprint.} The census never stored it.

3. Labels and code.\textbf{3. Labels and code.} The recorded verdict stays the primary label; Appendix A.5 reports corrected labels, under which the census estimands move by at most 0.015, every paired-change interval containing zero. The revised supplementary package adds the code for every new analysis and fixes an import-path defect in 22 modules of the submitted package.

4. What we do not claim.\textbf{4. What we do not claim.} We do not claim that replay noise caused any particular misattribution; a ranking of the attribution methods; that the matched content is irrelevant, or that its effect is smaller than the count-matched control's (not tested); or a ceiling for ContextCite: its figures describe our adaptation, which we did not re-run with a tuned penalty. The attribution and intervention experiments were not repeated on a second model–attack pair. The DeepSeek arm very probably ran at the provider's default temperature, an inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.