acceptodds
Under review as a conference paper at ICLR 2027

HarnessSHAP: Explaining LLM Agent Trajectories with Task-Adaptive Shapley Attribution

Abstract

When a large language model (LLM) agent succeeds or fails on a multi-step task, developers need to identify which parts of the trajectory contributed to the outcome. Existing attribution methods often rely on fixed judgment prompts, which may miss task-specific success criteria, and naive suppression can leave downstream actions dependent on removed steps. We introduce HarnessSHAP, a trajectory attribution method that learns a task-adaptive harness for a frozen LLM judge. The harness contains role-analysis and counterfactual-judgment prompts that estimate outcomes after dependency-consistent suppression while allowing downstream replanning. HarnessSHAP then applies Shapley attribution over dependency-valid orderings to assign contributions to evidence, reasoning steps, tool calls, and actions. The harness is optimized by XAI-Grad, a textual prompt optimizer that uses replay-derived endpoint and marginal errors to update the prompt pair without changing model parameters. We evaluate HarnessSHAP against a Fixed harness and Generic TextGrad on API-Bank, HotpotQA, ToolSandbox, and Who&When. XAI-Grad improves counterfactual outcome agreement and attribution quality across evaluated settings. On API-Bank, HotpotQA, and ToolSandbox, XAI-Grad achieves Spearman correlations of 0.74, 0.72, and 0.79, with endpoint MAE values of 0.103, 0.111, and 0.096, respectively. The ablation study shows that combining endpoint and marginal feedback reduces attribution error to 0.124. On Who&When, HarnessSHAP evaluates static failure localization separately from replay-based attribution.

Primary area:
interpretability and explainable AI
Keywords:
explainable AI, LLM agents, Shapley values, counterfactual attribution, failure attribution, prompt optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_eHXGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces HarnessSHAP, which attributes outcomes of agents to each trajectory units. Authors conduct experiments on three benchmarks and compare XAI-Grad against fixed harness and generic textgrad.

Critical strengths
  • The topic about credit assignment for trajectories for debugging or auditing is timely and important
  • The set up is somewhat promising, e.g., the context are restrict to prerequisite-closed units.
Critical weaknesses
  • I find results in Table 1/2 very suspicious: the three benchmarks increase in almost equal steps, i.e., drop rises by +0.04, validity by +0.06/0.07, top3 by +0.06. These numbers seem to be generated by LLMs instead of actual results. If I am wrong, please correct me with CIs and variances.
  • The experimental setup is very unclear, e.g., the model choice of agents, judges, etc. are not even revealed, the number of tested trajectories are not reported. The presentation is also suboptimal. For example, Table 3 does not specify which benchmark it uses.
  • The baseline selection is not fully fair. For example, a more related baseline AgentSHAP is completely ignored. Seems that all comparison are based on prompt variants of the paper's pipeline.
  • vspace is used too aggresively, the table captions are very close to the tables. And why some captions are placed under the tables? I do think this is actually allowed. Moreover, the font size of those captions which are under the table is noticeably smaller than required (e.g., l.191-195).
Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer eHXG

Thank you for the direct review. In short: (1) the revised Table 1 reports variability across five seeds and corrects the call counts. (2) We will specify the setup in full. (3) We clarify the relation to AgentSHAP. (4) We will correct the formatting.

1. Results in Tables 1 and 2.

Thank you for asking for variances. The revised Table 1 reports Drop, Valid and Top-3 as mean ± sample standard deviation across five seeds, with standard deviations between 0.006 and 0.021. For example, on ToolSandbox Drop is 0.36 ± 0.017 for the Fixed harness, 0.40 ± 0.007 for Generic TextGrad and 0.44 ± 0.012 for XAI-Grad. The submitted table also reported incorrect explanation call counts, and the revised table corrects them (general response). We apologize for this error.

2. Experimental setup and Table 3.

We will report the agent, judge, critic and editor models and the number of tested trajectories per benchmark in Appendix A, and state the benchmark of Table 3 in its caption.

3. Baseline selection and AgentSHAP.

AgentSHAP, which the paper cites, attributes outcomes to tools with Monte Carlo Shapley values over tool subsets. HarnessSHAP attributes outcomes to individual trajectory units, suppresses each unit together with its dependents and lets the agent regenerate the remaining trajectory. The two methods answer different questions at different granularity. AgentSHAP is the closest Shapley-based method, and we will add a direct discussion of the two.

4. Formatting.

Thank you for noting this. We will place all captions above the tables at the required font size and remove the spacing compression.

Reviewer_iRPBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes HarnessSHAP, a method that attributes an LLM agent's outcome to individual steps, including evidence, reasoning operations, tool calls, and actions. The authors further introduce XAI-Grad, a method based on TextGrad, to improve the harness system. Extensive experiments across diverse datasets demonstrate the effectiveness of the proposed methods compared to existing baselines.

Critical strengths
  1. This paper addresses an important problem that understand how each step in the agent rollout contribute to the final result, this is important for improving the existing agent system.
Critical weaknesses
  1. The evaluation relies heavily on metrics defined by authors and the selected baselines are limited, leaving its advantage over other attribution or failure tracing methods unclear.

  2. The practical value of the proposed method has not been fully demonstrated. It does not show whether the attributions can guide agent improvement. You can check Agentracer(ICLR26, also be mentioned in this paper's related work).

  3. The writing and presentation need significant revision.

Questions for authors
  1. What are the advantages compared to the existing methods?

  2. How can the proposed method be used in practical applications or agents?

2 comments
AuthorsOpenReview ↗

Response to Reviewer iRPB

Thank you for the review and for recognizing the importance of step-level attribution for agent systems. In short: (1) we state below how HarnessSHAP differs from existing attribution and failure-localization methods. (2) The intended use is debugging and auditing of agent trajectories. (3) We will extend the related-work comparison and revise the presentation.

Q1. Advantages compared with existing methods.

Existing methods attribute outcomes to retrieved context (ContextCite), to tools (AgentSHAP, Monte Carlo Shapley values over tool subsets) or to annotated failure sites (Who&When, AgenTracer). HarnessSHAP differs in two respects.

  • Unit-level counterfactuals that respect dependencies. It assigns a contribution to every typed unit of a trajectory, including evidence, observations, tool calls, results, actions and reasoning operations. A unit is suppressed together with the units that depend on it, and the agent regenerates the rest of the trajectory, so a counterfactual never reuses removed information. Contributions are averaged over orders consistent with the prerequisite graph (Faigle-Kern value), rather than over arbitrary subsets.
  • Task adaptation without changing the judge. XAI-Grad adapts only an external pair of prompts using errors against accepted replays, while the judge model and the attribution game stay fixed.

W1. Metrics and baselines.

The fidelity metrics are standard: Spearman correlation and mean absolute error against accepted replay outcomes, and Top-k overlap and Drop@k for selected units (Section 4.2, Appendix A.6). The current comparison holds the attribution game fixed and varies only the prompt optimizer, which isolates the effect of XAI-Grad. A direct comparison with AgentSHAP and AgenTracer would complement it, and we will discuss both methods in detail in the related work.

W2 and Q2. Practical value.

The intended use is debugging and auditing: a developer can see which units of a failed or successful trajectory most affect the outcome under dependency-consistent suppression. Using the attributions to guide agent improvement, as AgenTracer does through downstream repair, is the natural next step, and we will add it to the limitations.

W3. Writing and presentation.

The general response lists the presentation changes.

Reviewer_iRPBOpenReview ↗

Many thanks for your rebuttal

Due to the limited time before the discussion closed, I will continue to work on checking your rebuttal after the discussion period. Thanks again for your rebuttal.

Reviewer_hJyKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes HarnessSHAP, a framework for attributing LLM agent outcomes to individual trajectory steps. It uses dependency-constrained Shapley values and an LLM judge to estimate the effects of suppressing different steps while allowing downstream replanning. The authors also introduce XAI-Grad to optimize the judgment prompts using feedback from counterfactual replays. Experiments on four benchmarks report improvements in attribution accuracy and failure localization over fixed prompts and Generic TextGrad.

Critical strengths

S1. Meaningful problem formulation.

The paper addresses a relevant problem in understanding LLM agent behavior. I particularly appreciate the consideration of dependencies between trajectory steps, as simply removing individual steps may lead to invalid counterfactuals. The proposed suppression and replanning framework is a reasonable approach to this issue.

S2. Replay-based supervision and evaluation.

Using actual replay outcomes to supervise the LLM judge is a useful design choice. The comparison with fixed prompts and Generic TextGrad also helps assess the benefit of task-adaptive optimization. The reported improvements across several benchmarks are encouraging, although further validation is needed.

Critical weaknesses

W1. Insufficient experimental details and unclear computational accounting.

Several important experimental details are missing, including the models used, dataset sizes and splits, prompt configurations, replay budgets, and random seeds. This makes it difficult to reproduce or fully assess the reported results. I also noticed that Table 1 reports substantially different explanation call counts across methods, although the paper states that they share the same context ledger and make one judgment per distinct context. This discrepancy needs clarification.

W2. Limited validation of counterfactual interventions.

The reliability of counterfactual replay is central to the proposed method, but the current evidence is insufficient. In particular, Table 4 reports execution checks as not evaluated across the three replay benchmarks, while Appendix F only validates one scripted ToolSandbox scenario. It remains unclear how reliably the interventions are enforced in practice, especially when suppressing reasoning steps or preventing downstream reuse of removed information. This raises concerns about the validity of the reference attributions.

Questions for authors

Q1. Could the authors provide the missing experimental configurations, including model choices, dataset sizes, replay budgets, and random seeds? Also, why do the explanation call counts in Table 1 differ substantially when the methods supposedly use the same context ledger?

Q2. How do the authors verify that counterfactual suppression is correctly enforced across the evaluated benchmarks? Could they provide benchmark-level execution validity results and clarify how failed or rejected replays affect the attribution evaluation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer hJyK

Thank you for the careful review and for recognizing the dependency-aware formulation and the replay-based supervision. In short: (1) the call counts in Table 1 were incorrect, and the revised PDF corrects them and reports variability over five seeds. (2) We describe below how failed and rejected replays are handled. (3) The paper scopes its execution evidence to the validated scenario.

W1 and Q1. Experimental configuration and call counts.

Thank you for spotting this. The call counts in the submitted Table 1 were reported incorrectly. The revised Table 1 gives the corrected counts, which are equal across methods within each benchmark (13.8, 13.4, 15.2 and 14.3 calls per trajectory on API-Bank, HotpotQA, ToolSandbox and Who&When). Drop, Valid and Top-3 are now reported as mean ± standard deviation across five seeds. We will report the full model and data configuration in Appendix A.

W2 and Q2. Enforcement of counterfactual suppression.

Table 4 marks execution checks as not evaluated on API-Bank, HotpotQA and ToolSandbox, and Appendix F covers one scripted ToolSandbox scenario. That evidence establishes adapter behavior for the scenario, not benchmark-level enforcement, and Section 6 states that execution support must be assessed separately for each benchmark.

Failed and rejected replays affect the evaluation as follows (Section 3, Appendix A.3 and A.6).

  • Missing references. A proper context with no accepted replay has a missing reference estimate. It is never assigned zero and enters neither the endpoint nor the marginal loss.
  • Fixed population. Reference support is fixed before any candidate is evaluated. A failing prompt candidate cannot remove contexts or change the eligible support, and every prompt pair is evaluated on the same trajectories, contexts and orders.
  • Incomplete trajectories. A trajectory whose ledger contexts lack reference values stays in coverage reporting but supplies no reference attribution vector.
  • Adherence. Judge adherence is checked separately from execution. As Appendix A.2 notes, rule checks cannot guarantee detection of paraphrased information or hidden reasoning, so passing an execution check does not certify adherence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their time and apologize for the errors in the submitted Table 1. We have uploaded a revised PDF that corrects them.

Corrections in Table 1.

  • Explanation call counts. The submitted table reported incorrect call counts, which differed across methods although all methods share one context ledger and make one judgment per distinct context. The revised Table 1 reports the corrected counts, which are equal across methods within each benchmark: 13.8 calls per trajectory on API-Bank, 13.4 on HotpotQA, 15.2 on ToolSandbox and 14.3 on Who&When. Replay attempts during explanation remain zero.
  • Variability across seeds. Drop, Valid and Top-3 are now reported as mean ± sample standard deviation across five seeds. The standard deviations range from 0.006 to 0.021. For XAI-Grad on API-Bank, for example, Drop is 0.42 ± 0.006, Valid 0.82 ± 0.014 and Top-3 0.70 ± 0.012.

What the method establishes. The method is fully specified in Section 3.

  • Dependency-consistent counterfactuals. A context exempts a prerequisite-closed set of units from suppression, and the agent regenerates the remaining trajectory, so downstream steps cannot reuse information from suppressed units.
  • Attribution on a precedence-constrained game. Contributions are averaged over linear extensions of the prerequisite graph and target the Faigle-Kern value of the mean judgment game (Section 3.5, Appendix B).
  • Task adaptation with a frozen judge. XAI-Grad changes only an external pair of prompts, using endpoint and marginal errors against accepted replays. The judge model, parser, dependency rules and suppression policy stay fixed.
  • Fixed evaluation population. A context without an accepted replay has a missing reference estimate, is never assigned zero and enters neither loss. The reference support is fixed before candidate evaluation, so a failing candidate cannot remove contexts from the comparison (Appendix A.3, A.6).

Execution validity. Table 4 marks execution checks as not evaluated on the three replay benchmarks, and Appendix F validates one scripted ToolSandbox scenario. Section 6 states that replay validity, judge adherence and environment reset must be assessed separately for each benchmark, and we do not claim benchmark-level execution validity.

Presentation. We will place all captions above the tables at the required font size, remove the spacing compression, state the benchmark of Table 3 and report the model and data configuration in Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.