acceptodds
Under review as a conference paper at ICLR 2027

Recovering Decision Provenance from Internal Activations in LLM Agents

Abstract

Tool-using language-model agents combine information from user instructions, policies, retrieved records, memories, and tool observations when producing structured actions. Although the resulting tool call records the selected tool and argument values, it does not reveal which parts of the context influenced each field. We define field-level decision provenance through source interventions: for a fixed proposed action, we neutralize one source at a time and measure how the model's score for each realized field value changes. This yields a signed effect matrix over sources and action fields that captures both the direction and strength of source influence, but computing it directly requires repeated model evaluations. We introduce the Interventional Effect Decoder (IED), a lightweight decoder trained on offline intervention targets. Given the original context and proposed action, IED reads internal activations from a frozen backbone and predicts the complete effect matrix from a single scoring pass, avoiding repeated interventions at deployment. Experiments across diverse tool-use settings show that IED recovers informative source-level provenance while substantially reducing online attribution cost. The predicted provenance also improves the repair of incorrect tool arguments by helping a reviewer locate the context relevant to each field. Finally, we extend this representation with support and authorization supervision in a separate decision model for prompt-injection defense, enabling pre-execution detection of actions influenced by sources that lack authority over the affected fields.

Primary area:
causal reasoning
Keywords:
LLM Agents, Causal Attribution, Causal Provenance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VJ3hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines field-level provenance for structured tool calls as the change in the frozen model's mean log probability of a fixed field value when a candidate context source is neutralized. It proposes the Interventional Effect Decoder (IED), which learns to predict the direction and magnitude of these effects from pooled internal activations in one factual scoring pass. Experiments evaluate source identification and effect prediction on tool-use decisions, evidence-assisted repair of corrupted arguments, and a separately trained pre-execution defense model incorporating support and authorization supervision. The reported results show substantial online attribution savings, improved repair on BFCL, and a useful security–utility trade-off on three prompt-injection benchmarks.

Critical strengths
  1. A concrete, measurable attribution target with a practical amortization strategy. Equations (1)–(2) hold the action fixed and define signed effects separately for each field, rather than relying on a generated explanation. The decoder moves repeated interventions offline. Tables 4 and 6 provide effect-prediction evidence, and the timing study reports approximately 5.37× lower online cost than explicit intervention at six sources. This makes the proposal potentially useful for frequent pre-execution audits.

  2. The paper tests downstream usefulness and includes relevant controls. On BFCL, IED raises single-call repair recovery from 53.4% to 65.6%, with 99.7% clean retention (Table 3). Table 5 separates field supervision from influence fusion, while Appendix C.6 examines alternative neutralization strategies. These checks go beyond demonstrating that the decoder can fit its training target. The paper also reports limitations in its own results, including no repair improvement over standard review on τ²-bench.

Critical weaknesses
  1. The headline source-identification metric does not directly measure recovery of the defined intervention provenance, and the evaluation boundary is insufficiently characterized. Appendix C.1 assigns the reference source according to semantic support or justification, excludes ambiguous and unsupported fields, and manually checks 10% of labels. A source that justifies a value need not be the source whose neutralization most changes the model's score. Thus Table 1's 74.6% mean accuracy is not directly an accuracy measure for recovering the highest-effect source in Equation (2). Tables 4 and 6 do provide direct effect evidence, but the paper should also report agreement with the intervention-derived top source, the fraction of fields excluded from the semantic metric, and the outcome of the annotation audit. In addition, the stated within-domain decision split does not establish whether related decisions from the same trajectory or task are kept together. These details materially affect how much the results support reliable provenance recovery on unseen decisions.

  2. The security–utility comparison lacks enough protocol detail to assess generalization and operating-point fairness. The paper explicitly trains a separate defense model (Section 3.5), but Sections 4.1–4.3 and Appendix C.3 do not identify its training-set size/composition and task/attack separation from evaluation, or the blocking thresholds and their selection procedure for IED and attribution-based guards. These omissions matter because changing the blocking rule can strongly affect both attack success and task completion. Table 5 is encouraging, but its no-fusion control still uses a separately supervised defense model; it does not by itself establish comparability with the other attribution guards. A reproducible split description and thresholds selected on held-out data, ideally with a matched-utility comparison, are needed to interpret Table 2's advantage.

Questions for authors
  1. How often does the semantic reference source in Appendix C.1 agree with the maximum positive intervention effect? Please report intervention-derived top-source recovery, excluded-field counts, and the actual agreement/error rate from the 10% manual audit. How are cases with no positive effect handled?
  2. Are train/test decisions grouped by original task or trajectory, including reference-action prefixes and rollout records? Please specify the grouping and any cross-split overlap. If the current split is decision-level, does the result persist under a trajectory-disjoint split of the existing data?
  3. What data train the separate defense model, how are evaluation attacks/tasks excluded, and what are the exact blocking thresholds and calibration sets for each guard? Do the relative results hold at comparable benign utility?
  4. What common support/authorization machinery, if any, is used by the attribution-based defense baselines in Table 2? This determines whether the comparison isolates the attribution signal or compares different overall guard architectures.
  5. Code–paper consistency. Does the supplementary code correspond to the model and experiments reported in the paper? The decoder omits the lexical-value branch and directly regresses signed effects instead of the direction–magnitude formulation in Eqs. (7)–(10); source top-1 is evaluated against argmax(J) rather than the stated semantic-support labels; and the defense implementation lacks the independently supervised support head and effect fusion in Eq. (11). Please reconcile these differences and identify the implementation used to produce the reported results.
5 comments
AuthorsOpenReview ↗

Response to Reviewer VJ3h (1/2)

We thank the reviewer for the comments on source definition and evaluation protocol. We add intervention-target evaluation and task-disjoint results, and check the defense data, calibration, and shared components.

We will include these experiments and changes in the final revised version.

Semantic source vs. intervention source: agreement, annotation and exclusion rules (Weakness 1 / Question 1)

Thank you for helping us find this issue: the labels in Table 1 are indeed semantic reference scores given by a large language model, and they are not the intervention provenance defined in Eq. (2).

To address this, we re-evaluated the original IED model with the intervention provenance as the reference. Our three-domain top-1 mean is 80.5%, compared with 75.5% for Activation Patching. This shows that under the definition of Eq. (2), the intervention-derived top-source recovery still holds, and the advantage is clearer than under the LLM labels.

Table 1 in the paper has 6,216 test fields: 2,269 operation fields (tool names) and 3,947 argument fields. By Appendix C.1, only argument fields are evaluated. Among the argument fields, 2,835 have a positive intervention effect (max J>0.05) and form the IED denominator of the original Table 1; the three datasets in the main text share 2,797 (the 38 positive fields of AgentDojo have only one source and are not included); the other 1,112 argument fields have no positive intervention effect and are not forced to have a positive source.

The semantic-label audit batch is counted separately: from the 2,835 positive fields, we remove 191 S=1 deterministic fields and 3 fields that did not enter the batch, leaving 2,641; adding 200 control fields without positive J gives 2,841 audited fields. The overall agreement between the multi-judge semantic reference labels and the intervention provenance is 60.7%.

60.7% shows that the two definitions are not equivalent, so semantic labels cannot be used directly as intervention-provenance recovery. But the two are not unrelated, and semantic labels still have reference value for the main dataset. For the manual audit, we randomly sampled 280 of the 2,841 semantic-label fields (about 10%) and had three human annotators score them: 269 agree with human judgment and 11 do not, giving a correct rate of 96.1% and an error rate of 3.9%; we corrected the 11 inconsistent labels. Semantic labels reflect attribution direction, but they are not the ground truth of intervention effect and cannot replace intervention-provenance evaluation. The paper will report the two settings separately and correct the reference-source note and table caption.

Train/test separation by task or trajectory (Weakness 1 / Question 2)

Thank you for the correction. The split of Table 1 is decision-level: decisions of the same task or trajectory can appear in both the training and test sets, so there is overlap. We therefore add a task-disjoint split at the task/trajectory level: grouped by scene and normalized task key, the scene and normalized-task overlap across sets is zero, and the train/dev/test sizes are 5,208/1,336/1,028.

The results under the new split are as follows.

Methodττ²RolloutMean
IED79.9%82.9%76.0%79.6%
Activation Patching69.4%81.0%75.8%75.4%
TextEmb70.9%82.7%70.2%74.6%
Lexical73.0%83.7%61.4%72.7%

IED is higher than AP and TextEmb under the task-disjoint setting, so the main source-ranking result holds on unseen tasks. The old split is re-evaluated at 80.5%, and the task-disjoint setting is 79.6%, a difference of about 0.9 points.

Downstream repair and attack evaluations are also separated by task groups; details are in Question 3.

AuthorsOpenReview ↗

Response to Reviewer VJ3h (2/2)

How is the defense model trained, and is the test separation and threshold comparison fair? (Weakness 2 / Question 3)

The paper does not describe the training data, test separation, and threshold selection of the defense models in enough detail; we give them here. This part has five blocks: training data, test rollouts and separation, invalid samples, thresholds, and comparison at comparable benign utility.

Training data. IED reads the internal representations of context sources and call fields and forms a call risk. The training set is used for learning, the validation set for model and threshold selection, and the test set for final evaluation. All three benchmarks split train, validation, and test by task groups, and training and test tasks do not overlap. The training data are as follows, in decisions.

Training settingTraining rowsValidation rows
AgentDojo1,873327
ASB9,6003,200
MCP-Tox2,193681

Test rollouts and separation. The test rollouts are as follows. AgentDojo has 1,116 attack + 97 clean. ASB has 640 + 160. MCP-Tox has 249 + 249. Test separation is by task, agent, and server: AgentDojo attack test user tasks do not overlap training tasks; ASB training excludes test agents; MCP-Tox uses 27 training, 9 validation, and 9 test servers. The repair scoring model is trained with test tasks excluded, and BFCL does not participate in scoring-model training.

Invalid samples. We do not set a separate exclusion layer; the denominator of each benchmark is all attempted rollouts, and parse failures or no-action samples are counted as failures and not removed.

Blocking thresholds. Blocking thresholds are selected on the validation (calibration) set of each benchmark and fixed before test; for IED they are .97 on AgentDojo, .95 on ASB, and .97 on MCP-Tox. AP, CC, and lexical reuse the same IED guard, with the same data split; the three baselines only provide attribution scores, and the guard makes the allow-or-block decision, so the difference comes only from the front-end attribution signal. When the risk score exceeds the threshold, the candidate call is not executed; the system returns feedback and lets the agent replan; after the budget is reached, the action is still not allowed. Other complete defenses keep their native mechanisms.

Comparison at comparable benign utility. The MCP-Tox results are as follows.

MethodASR↓UA↑BU↑
IED0.4%31.3%48.6%
AP0.4%15.7%49.0%
CC0.4%12.1%47.8%
Lexical0.4%16.9%49.0%
PI Detector8.0%29.3%48.2%

All methods have BU between 47.8% and 49.0%. IED has the same ASR as AP, CC, and lexical, and UA is 15.6, 19.2, and 14.4 points higher, respectively; compared with PI Detector, IED has lower ASR and higher UA at similar BU. The task-completion advantage in the original table is not mainly obtained by lowering clean task completion.

Do the attribution-based defense baselines use the same support/authorization machinery? (Weakness 2 / Question 4)

The attribution-based defense baselines in Table 2 do share the same support/authorization machinery. AP, ContextCite, and text methods themselves provide source attribution information; to use them for defense, we connect their attribution scores to the same IED guard. AP, CC, and lexical reuse the same guard checkpoint, with the same data split. Therefore the comparison isolates the front-end attribution signal, not different defense architectures. TextEmb is trained separately with the same S/A labels.

The IED in the three benchmarks uses the same source-field modeling and support/authorization judgment process, only adapting to each tool format and context organization; the three are trained on their own data and determine operating points on their own validation sets.

Does the supplementary code correspond to the models and experiments in the paper? (Question 5)

We apologize that the supplementary code we provide is not complete, because we are still in the review stage. The code we provide is part of our iterative code, and these methods are also part of our ablation study. We will release the complete code later.

All our code is based on the paper, and the code and experiments correspond to the paper.

Thank you for the detailed rebuttal and the additional experiments. The intervention-based evaluation, task-disjoint results, and clarifications on the defense protocol address my concerns for now.

Reviewer_uNmsOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Agent tool calls often depend on multiple fields. However, heterogeneous contextual information might influence the tool-calling fields. To address the gap, the authors provide IED, a framework that identifies and evaluates which context sources influence different fields of LLMs' tool calls. IED first works by estimating the probability changes in the tool argument after a source is removed. Then the authors train a lightweight decoder that can predict such effects using the model's activations. Experiments across benchmarks demonstrate the effectiveness of the method in attribution.

Critical strengths
  • Clear problem formulation and motivation. The studied problem, i.e., understanding which information sources influence different tool-call arguments, is important to the community.
  • Efficiency of the IED decoder. The IED is efficient in inference, which would be beneficial to the community.
  • The paper does not evaluate the attribution performance but also extends the method to a defense strategy in prompt injection attacks, which is quite interesting.
Critical weaknesses
  • Generalization of the method. If I understand it correctly, the train-test split is done within each domain, limiting our understanding of whether IED can generalize to unseen task domains. For example, can you try training the decoder on dataset A but evaluate the performance on dataset B?
  • Unfair inference cost comparison. The paper reports that IED is 5.23× faster than Activation Patching in online inference. However, this does not take the training stage into consideration, while Activation Patching does not require such training. I would suggest that the authors also report the training cost and compare the costs more fairly.
Questions for authors

See my weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uNms

We thank the reviewer for the comments on cross-domain generalization and cost comparison. We add a leave-one-domain-out evaluation and a defense transfer experiment with frozen model parameters.

We will include these experiments and changes in the final revised version.

Generalization to unseen task domains (Weakness 1)

We run a leave-one-domain-out evaluation with two random seeds. The in-domain model uses a mixed training pool of τ, τ², and Rollout, with test tasks separated from training tasks. The cross-domain model removes all records of the target domain from the same pool and keeps the other settings unchanged.

Test domainIn-domain top-1Removed domainCross-domain top-1 (mean of 2 seeds)
τ79.9%τ78.4%
τ²82.9%τ²57.8%
Rollout76.0%Rollout40.6%

τ drops by 1.5 points, so source ranking has some transfer ability; τ² and Rollout drop clearly, consistent with domain dependence. This shows that source ranking has some generalization ability on τ and is more domain-dependent on τ² and Rollout.

Defense transfer results are as follows. The transfer model is trained on AgentDojo only and has frozen parameters at test time.

Train→TestSettingASR↓UA↑BU↑
ASB→ASBIn-domain IED0.0%56.4%97.5%
—→ASBNo defense90.9%4.8%97.5%
AgentDojo→ASBTransfer defense4.13%53.50%95.75%

MCP-Tox is evaluated on 498 rollouts from another 9 test servers.

Train→TestSettingASR↓UA↑BU↑
MCP-Tox→MCP-ToxIn-domain IED0.4%31.3%48.6%
—→MCP-ToxNo defense9.6%43.8%48.2%
AgentDojo→MCP-ToxTransfer defense4.42%43.78%48.59%

With frozen parameters, the transfer model is applied directly to ASB and MCP-Tox. On ASB, it reaches ASR 4.13%, UA 53.50%, and BU 95.75%; no defense gives 90.9%, 4.8%, and 97.5%. On MCP-Tox, it reaches ASR 4.42%, UA 43.78%, and BU 48.59%; no defense gives 9.6%, 43.8%, and 48.2%. This shows that with frozen parameters, IED's defense ability transfers to ASB and MCP-Tox, so our method has some generalization ability.

Overall cost including offline training (Weakness 2)

Offline cost estimates are as follows.

ItemCost and setting
Intervention target generationAbout 2 GPU hours, estimated from about 5.2k samples, 4.1 scoring runs per sample, and 0.33 s per run
Decoder trainingAbout 58 s per seed, about 174 s for three seeds
TotalAbout 2 GPU hours

IED's offline cost is mainly the one-time construction of intervention targets, which can be reused across training seeds and later online calls. AP needs no training, but repeats intervention scoring for every online attribution. With a fixed backbone and many decisions, IED lowers the per-call attribution cost through this upfront training.

At this scale, the one-time offline cost is about 2 GPU hours, and decoder training is about one minute per seed.

Break-even: with six sources (S=6), Activation Patching takes 9.49 s per attribution and IED takes 1.82 s, saving 7.67 s per call. With a one-time offline cost of 7,200 s, the total cost falls below Activation Patching after about 940 decisions; against explicit intervention, about 910 decisions. Moving the cost offline also matters in practice: detection at inference needs to be fast; once the targets are built offline, online attribution time drops substantially. This answers the cost-comparison question: online latency and offline training are reported separately, and we give the break-even point.

Thanks for the authors' rebuttal. My concerns are addressed. I am impressed by the rebuttal results. I have increased my score.

Thank you for the careful review and for updating your assessment. We are glad our response addressed your concerns. We wish you all the best.

Reviewer_hEg2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to identify which parts of an LLM agent's context influence different fields in a tool call. The authors introduce IED, a decoder trained to predict how replacing individual context sources would affect the model's scores for those fields. It uses the model's internal representations to make these predictions in a single pass, instead of running separate interventions for each source. The authors test IED on source identification, repairing incorrect tool arguments, and prompt-injection defense. IED achieves similar average accuracy to Activation Patching while being much faster. They also show that source information can help with argument repair, and explore how combining it with support and authorization information can improve security.

Critical strengths
  1. Reducing the cost of source attribution without losing much accuracy.

One of the main advantages of IED is that it avoids running separate interventions for every context source at inference time. The results show a large reduction in runtime with similar average accuracy. For example, IED achieves 74.6% mean source identification accuracy compared with 75.5% for Activation Patching (Table 1), while being substantially faster. This could be especially useful for agents that make many tool calls, where running multiple interventions for every action would be expensive.

  1. Testing whether attribution helps improve agent behavior.

I liked that the authors go beyond measuring attribution accuracy and test whether the information can help repair incorrect tool arguments and defend against prompt-injection attacks. For example, on BFCL, IED improves argument recovery from 53.4% with standard review to 65.6% (Table 3). These experiments help show the practical value of source attribution, rather than just how accurately it can identify relevant sources.

Critical weaknesses
  1. It is unclear how much of the defense improvement comes from influence prediction itself.

The prompt-injection results are promising, but the defense uses a separate decision model trained with additional support and authorization labels. In Table 5, removing influence fusion increases attack success only slightly, from 1.8% to 2.1%, while removing field-level supervision increases it to 18.7%. Influence fusion also improves task completion, so it does seem to help. However, these results make me wonder how much of the security improvement comes from the additional supervision rather than the influence predictions. I would have liked to see the ablation on all three security benchmarks, along with a model trained without the influence prediction branch.

  1. It is not clear how much of the improvement comes from using internal activations.

IED uses the model's internal representations to predict source effects, but the paper does not compare it with a learned decoder trained on the same intervention targets without using hidden activations. The lexical overlap baseline is a simple heuristic, while the rule-label ablation changes the training labels rather than the representations. I would have liked to see a comparison using text embeddings instead, to better understand what the internal activations actually add.

Questions for authors
  1. Could the authors compare IED with a learned decoder trained on the same intervention targets using text embeddings instead of hidden activations? This would help show how much the internal representations contribute.
  2. Could the authors extend the defense ablation to all three benchmarks and include a model trained without the influence prediction branch? This would help clarify how much the influence predictions contribute.
  3. How does IED perform with more than six candidate sources or with a different backbone model? It would be interesting to see whether the results hold in larger agent settings.
5 comments
AuthorsOpenReview ↗

Response to Reviewer hEg2 (1/3)

We thank the reviewer for the comments, and for the positive comments on the online efficiency and downstream uses of IED. We add a text-embedding baseline, cross-backbone experiments, and defense ablation experiments.

We will include these experiments and changes in the final revised version.

Defense improvement: influence prediction vs. extra support/authorization supervision (Weakness 1 / Question 2)

The defense gain comes from influence prediction and extra support/authorization supervision together, not from either one alone; extra support/authorization supervision is built on influence prediction. We use three experiments to separate the training supervision of J, the role of J at inference, and whether influence prediction can directly form a defense rule.

(1) Do not train the influence-prediction branch. We disable the magnitude and ranking supervision of J in training and retrain; influence fusion is not used at inference.

BenchmarkModelASR↓UA↑BU↑
AgentDojoFull IED1.8%34.9%50.5%
AgentDojoNo influence3.85%27.69%47.42%
ASBFull IED0.0%56.4%97.5%
ASBNo influence0.00%24.06%52.50%
MCPFull IED0.4%31.3%48.6%
MCPNo influence3.61%51.41%48.59%

On AgentDojo, the model without the influence-prediction branch has ASR 3.85%, UA 27.69%, and BU 47.42%; full IED is 1.8/34.9/50.5. On ASB, the model without the influence-prediction branch is 0.00/24.06/52.50; full IED is 0/56.4/97.5. On MCP-Tox, the model without the influence-prediction branch has higher UA than full IED (51.41 vs 31.3) and also higher ASR (3.61 vs 0.4).

With S/A supervision only, the model still blocks most attacks on all three benchmarks: ASR is 0 on ASB, 3.85% on AgentDojo, and 3.61% on MCP-Tox. Full IED has higher task completion on AgentDojo and ASB; the direction reverses on MCP-Tox. This shows that S/A supervision alone is enough to block most attacks; the training supervision of influence prediction mainly improves task completion.

(2) Remove only influence fusion at inference. Table 5 keeps the full model's J, S/A, and risk training supervision, sets the fusion weight to 0 at inference, and lets the S/A risk path decide alone.

AgentDojo, Table 5ASR↓UA↑BU↑
No field labels (action outcome only)18.7%35.3%58.4%
No fusion (J trained, w=0)2.1%29.1%46.1%
Full IED (J trained, w=0.3)1.8%34.9%50.5%

With action-outcome supervision only, ASR is 18.7%, much higher than full IED. Field-level support/authorization supervision is the main source of the defense gain. On top of this supervision, influence fusion raises UA by 5.8 points and BU by 4.4 points, and lowers ASR by 0.3 points. Influence prediction gives a stable gain, but it is not the main source of the defense effect. This shows that J's gain mainly comes from training, and fusion at inference only gives a small change.

One possible explanation is that the two signals are complementary: J describes which source affects the model, and S/A judges whether the source supports the value and has authority over the field.

(3) Keep only the influence-prediction front end, then add supervised training. This experiment tests whether the defense gain comes from extra supervision or from influence prediction. We remove the separate support head and the action-risk objective, keep only J as the front end, and train the backend decision with our labels on this representation. At inference, the positive predicted influence of each source is weighted by the probability that the source is not authorized, and then aggregated into a risk score by a fixed rule.

BenchmarkMethod/operating pointASR↓UA↑BU↑
ASBFull IED0.0%56.4%97.5%
ASBJ + authorization0.0%55.9%97.5%
MCPFull IED0.4%31.3%48.6%
MCPJ + authorization2.41%51.4%48.6%
AgentDojoFull IED1.8%34.9%50.5%
AgentDojoJ + authorization1.7%25.4%29.9%

After removing the separate support head and the action-risk objective, the defense is still effective. On ASB, the model has the same ASR and BU as full IED, and UA is 0.46 points lower. On AgentDojo, task completion is lower. On MCP-Tox, UA is higher and ASR is also higher.

AuthorsOpenReview ↗

Response to Reviewer hEg2 (2/3)

The result shows that support/authorization supervision and influence prediction are not two independent sources of gain. J, S, and A are all built on the same source-field content. The S and A matrices are trained with our labels, and J is computed from interventions on the same sources. They describe intervention effect, semantic support, and authorization authority, respectively; this separation lets the model explain what effect a source has, whether it supports the value, and whether it has authority over the field. Influence prediction gives the source information, and support/authorization supervision makes the allow-or-block decision on top of it.

Supervision is still needed. Influence prediction is information extraction, not an authorization judgment, so it cannot be used as a blocking criterion directly; the backend module needs to decide on top of this information.

Therefore, the source of the defense gain is not an either-or choice: support/authorization supervision presupposes the influence-prediction content.

Support and authorization labels are also used for the AP, CC, and lexical baselines, so the comparison isolates the front-end attribution signal. TextEmb is trained separately using the same S/A labels.

What do internal activations add over text embeddings? (Weakness 2 / Question 1)

We add a TextEmb baseline. Compared with IED, the intervention targets, decoder structure, and data split are kept the same; only the representation is replaced: TextEmb uses the mean of frozen text embeddings, and IED uses internal activations. Learned models report the mean over three random seeds.

Methodττ²RolloutMean
IED: internal activations79.9%82.9%76.0%79.6%
TextEmb: learned text embedding70.9%82.7%70.2%74.6%
Lexical: lexical overlap73.0%83.7%61.4%72.7%

Note: this table is on the task-disjoint split, with an equal-weight mean; the split differs from that of Table 1 in the paper.

IED gains 5.0 points in mean over TextEmb, mainly from τ and Rollout. Lexical is 83.7% on τ², higher than IED's 82.9%; its τ and Rollout are clearly lower, and its mean is 6.9 points lower.

Repair results are as follows.

Repair datasetIED recovery↑TextEmb recovery↑IED clean↑TextEmb clean↑
BFCL65.6%56.1%99.7%100.0%
τ²24.6%23.8%99.2%98.4%

IED recovery is 9.5 points higher than TextEmb on BFCL and 0.8 points higher on τ². This shows that internal activations are also better on the repair task.

In-domain and cross-domain defense results are as follows.

BenchmarkRepresentationASR↓UA↑BU↑
AgentDojoIED1.8%34.9%50.5%
AgentDojoTextEmb12.72%21.59%41.24%
AgentDojo→ASBIED4.13%53.50%95.75%
AgentDojo→ASBTextEmb84.94%8.19%94.25%
AgentDojo→MCP-ToxIED4.42%43.78%48.59%
AgentDojo→MCP-ToxTextEmb9.86%44.38%48.63%
MCP-Tox→MCP-ToxIED0.40%31.33%48.59%
MCP-Tox→MCP-ToxTextEmb9.64%42.57%48.59%

IED has lower ASR and higher UA/BU on AgentDojo, and keeps its defense ability in both transfer settings. On AgentDojo→ASB, ASR is 4.13% vs 84.94%. On AgentDojo→MCP-Tox, task completion is close, and IED has lower ASR (4.42% vs 9.86%). This shows that internal activations are better than text embeddings on source identification, repair, and defense.

One possible explanation is that the mean text embedding relies on lexical overlap and can hardly keep word order and negation scope. "Send to B" and "do not send to B" have the same recipient but opposite authorization direction. This representation also relies more on consistency of tool descriptions, field names, and formats between training and test; after cross-platform transfer, these patterns change and the learned relations do not transfer well. Internal activations are processed by the backbone over the full context and contain richer information about source roles and authorization relations. TextEmb clearly degrades on ASB, and on MCP-Tox it does not reduce ASR as IED does; it does not show stable defense transfer. The IED representation is not only raw hidden states; it also includes source roles, field values, and source-field interactions (Eqs. (5)-(7)). TextEmb has only the mean text embedding. The 5.0 points therefore come from internal activations together with these components, not from hidden states alone.

AuthorsOpenReview ↗

Response to Reviewer hEg2 (3/3)

More candidate sources and a different backbone model (Question 3)

For candidate sources, we add distractor candidates from other samples to 1,028 natural decisions. The table reports source top-1 for the model trained with six candidates and for the model further trained with distractor candidates.

# candidatesTrained with 6 candidates (frozen)After training with distractors
659.3%—
1240.3%67.2%
1836.5%—
2427.0%55.2%

At 24 candidates, the model trained with six candidates drops to 27.0%; after training with distractor candidates it recovers to 55.2%. The method can extrapolate: it works when both training and inference use more candidates, and training with six candidates still keeps some performance when more candidates are given. Performance drops as the number of candidates grows, which is expected because choosing among more options is harder.

For the backbone, we add source-identification results of 2B and 35B on the natural-decision subset, with the existing 30B weights as reference.

Backboneττ²RolloutMean
Qwen3-30B-A3B (existing weights)78.4%85.9%67.8%74.5%
Qwen3.5-2B76.1%86.6%64.5%72.2%
Qwen3.6-35B-A3B69.7%80.5%63.6%69.8%

The mean is weighted by the number of valid fields; this subset differs from the main-table test set.

2B reaches 72.2% and 35B reaches 69.8%, both below 30B's 74.5%. IED can be trained at both scales; performance varies with the backbone representation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all reviewers for their constructive feedback. We have updated the manuscript and supplemented the code release. Manuscript revisions are marked in blue.

  1. Attribution evaluation: Section 4.2, the caption of Table 1, and Appendix C.1 clarify semantic-source agreement and intervention-source recovery. We also provide field coverage, eligibility criteria, and annotation audit results.

  2. Task-disjoint evaluation and TextEmb baseline: Appendix E.1 reports intervention-source recovery with task/trajectory groups separated across splits. IED achieves 79.6% mean recovery, compared with 75.4% for Activation Patching and 74.6% for TextEmb.

  3. Defense protocol and ablations: Section 4.2 and Appendix C.3 specify training/validation sizes, evaluation separation, threshold calibration, and the guard shared by attribution baselines. Section 5.4 and Table 5 include the No influence and J + authorization ablations on AgentDojo, with corresponding ASB and MCP-Tox results in Appendix C.3.

  4. Generalization and source scaling: Appendices E.2 and E.3 report leave-one-domain-out evaluation and experiments with 12 and 24 candidate sources.

  5. Offline cost: Section 5.5 and Appendix C.4 report target-generation and decoder-training costs. Under the measured six-source configuration, the estimated break-even point relative to Activation Patching is approximately 940 decisions.

  6. Code release: We have included relevant model modules from our existing implementation and documented the training and evaluation entry points. CODE_MAP.md maps the paper’s equations to the corresponding files and functions. The Table 1 evaluator reads semantic-source annotations from JSON.

We appreciate the reviewers’ time and suggestions, which helped us clarify the evaluation and experimental protocols.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.