- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies query-conditioned agent "action attribution", where the goal is to identify historical sources relevant to a specific auditing question about an agent's action. The authors construct A3 BENCH, which comprises 1000+ queries across four categories: policy basis, parameter provenance, failure propagation, and unsafe-behavior tracing.
They propose a gradient-based attribution method that uses small open-weight proxy models to compute token-level saliency scores for the observed action. These scores are combined with semantic relevance and aggregated into step-level rankings. The final methods uses an ensemble of proposers to recover the source (and intermediate) evidence.
The authors report improvements in attribution ranking quality and inference efficiency over existing methods, along with competitive results against frontier LLM baselines.
Critical strengths
-
On query-conditioned attribution: I think explicitly conditioning attribution on an auditing question is a useful extension of existing work. Different questions about the same action can require different historical evidence, and the benchmark attempts to capture this through query-specific source and intermediate-evidence annotations. The human validation of these annotations is also useful though I'm still not sure if such a ground truth exists for complex agentic actions.
-
Attribution methodology: The use of query-conditioned gradients from smaller proxy models is an interesting approach, esp when the acting model is only accessible through an API. The method is also cheaper than perturbation-based attribution, and the ablations suggest that query-conditioned gradients contribute meaningfully to performance. The experiments also show that the rankings change when the auditing query changes (ie. it is sensitive to the query).
Critical weaknesses
-
On the task formulation and whether attribution is well-defined: I am not entirely convinced that the benchmark captures the complexity of agent action attribution. The authors assume that an action can be traced to an identifiable historical source and a sequence of intermediate evidence. However, agent behavior can arise from multiple interacting influences, diffuse information across a trajectory, or the model's learned behavior rather than a particular historical step. For example, how would this framework handle emergent misalignment, or agent errors that are not directly attributable to an earlier interaction?
Although the general formulation allows multiple sources, every benchmark instance has exactly one annotated source. This seems like a fairly restrictive assumption, especially for the kinds of complex behaviors that motivate agent auditing. This makes me question how much the task actually tells us about why an agent took a particular action.
-
On the proxy-gradient attribution: The method computes gradients with respect to a separate proxy model rather than the actual agent that performed the action. The gradients measure how historical tokens affect the proxy model's preference between the observed action and a constructed alternative. It is unclear to me why this should faithfully identify the historical information that influenced the acting model, rather than simply identifying information the proxy finds relevant to predicting the action.
The experiments show that this is a useful ranking heuristic, but do not show that the recovered evidence actually influenced the original agent. I also think the baselines could be stronger, since existing attribution methods are evaluated without comparable query conditioning. This makes it difficult to separate the contribution of the proposed gradient methodology from the benefit of explicitly providing an auditing query.
Generalization across attribution types also seems uneven. The method performs well on policy-based attribution but underperforms Claude Sonnet 4.6 on parameter provenance. The authors also find that many of these errors come from selecting downstream mentions rather than the actual information source. This seems like a limitation of using gradient sensitivity.
Questions for authors
-
On task definition: How does the proposed framework handle actions with multiple contributing sources or diffuse historical influences? What happens when an action arises from an agent error without an identifiable source in the visible trajectory? Why does the benchmark restrict every instance to exactly one source despite the general formulation allowing multiple sources?
-
On proxy attribution: Why should gradients computed through a separate proxy model recover the historical factors that influenced the acting agent? Can the authors validate this with interventions on the open-weight acting model, or investigate how sensitive the attributions are to the choice of proxy model?
-
On baselines: Could the authors compare against query-conditioned versions of existing attribution methods or simpler query-aware retrieval baselines?
-
On generalization: Why does the method perform substantially worse on parameter provenance compared with policy-basis attribution?
3 comments
Follow-up on Rebuttal and Additional Results
Thank you for your time and effort in reviewing our paper. As the author-reviewer discussion period is coming to an end, we would like to kindly follow up and ask whether our rebuttal and additional results have adequately addressed your concerns.
For convenience, we briefly highlight the main areas addressed in our rebuttal:
- Task formulation (W1/Q1)
- Intervention study (Q2)
- Query-conditioned baselines (W2/Q3)
- Parameter provenance (W3/Q4)
Thank you again for your time and consideration!
We would like to kindly follow up once more as the discussion period approaches its end. We would greatly appreciate it if you could share whether our rebuttal and additional experiments have addressed your main concerns, and whether they might warrant reconsideration of your initial assessment.
Thank you again for your time and consideration.
Key Takeaways from Our Rebuttal and Additional Experiments
We would like to briefly summarize four key findings from our rebuttal and revised manuscript:
Thank you again for your feedback. We hope these additional results help address your concerns.