- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TraceAttention, a training-free evaluator for context attribution that reuses full-context prefix states and masks target-to-context attention, avoiding repeated context reconstruction under physical deletion. Dual TraceAttention further reduces computation by selecting a compact reasoning target. Experiments evaluate attribution quality using held-out physical deletions and measure latency and memory across multiple models and context settings.
Critical strengths
-
The efficiency improvement is substantial and addresses a meaningful bottleneck in long-context attribution. Avoiding repeated context reconstruction makes repeated attribution queries considerably cheaper, which could make large-scale model analysis more practical.
-
The evaluation explicitly acknowledges that attention masking differs from physical deletion and tests attribution quality against held-out physical deletions. Evaluating both quality and computational cost across multiple settings provides useful evidence for the proposed trade-off.
Critical weaknesses
-
The practical boundary of this trade-off remains insufficiently clear. Quality degradation is task-dependent: for example, Table 6 reports LDS decreasing from 0.749 to 0.579 on SWE-agent trajectories with the full reasoning-and-answer target. Although the paper acknowledges these gaps, it offers limited guidance on when the approximation is reliable enough for diagnosis and when physical-deletion verification remains necessary.
-
The connection to actionable LLM development remains underexplained. The metrics and case studies demonstrate attribution behavior, but do not directly establish whether the faster attribution helps developers identify or resolve model failures. A clearer account of the intended diagnostic workflow would strengthen the practical significance of the systems contribution.
Questions for authors
-
Could TraceAttention serve as a screening stage, followed by physical-deletion verification of candidate spans in the original context? A small evaluation reporting total latency, attribution quality, and missed influential spans would help clarify whether this combination offers a useful operating point relative to either method alone. This is a suggestion for validating practical use, rather than a claim that such a combination must improve quality.
-
What concrete development decisions are the attributions intended to support, and which existing results demonstrate that they provide sufficiently reliable diagnostic evidence? Please also clarify the architectural assumptions relevant to extending the intervention to vision-language or other multimodal models. I regard multimodal experiments as future work; a discussion of applicability and limitations would suffice here.
-
Appendix A.4 reports that only 46 of 256 Nemotron InfiniteBench examples yield valid evaluation targets because the remaining responses lack the required reasoning–answer delimiter. Could the authors discuss how this filtering affects the representativeness of that setting and provide uncertainty estimates for its reported quality results?
Overall, I lean toward weak acceptance. The efficiency contribution is convincing, while clearer guidance on applicability, practical diagnostic use, and evaluation coverage would strengthen the paper.
2 comments
Summary of Revisions and Updates for Reviewer 6fhL
Dear Reviewer 6fhL,
Thank you very much for your helpful suggestions on our paper's practical use and evaluation. We wanted to briefly note that we have posted our response above and updated the manuscript (Appendix F) to address each of your points.
To summarize, we tested the two-step workflow you suggested, which uses TraceAttention to quickly shortlist candidate blocks and then checking those candidates with physical deletion, recovering 96.0% of Top-1 removal utility while cutting attribution time by 4.6x on SWE (Appendix F.6, Table 26). We also explained how this workflow supports practical model debugging and multimodal setups (Appendix F.7), and added confidence intervals plus a 14-setting exclusion check for the Nemotron InfiniteBench subset (Appendix F.1, Tables 17–18).
We hope these updates address your concerns and make the practical trade-offs much clearer. Should you have any further questions before the discussion period concludes, we would be glad to discuss them!
Sincerely,
The Authors
Response to Reviewer 6fhL
W1. Practical use and deletion verification
TraceAttention's selected blocks need not exactly match deletion's ranking to remain useful. Across the 15 settings, TraceAttention recovers 79.1% of deletion's Top-5 block IDs while retaining 96.0% of its physical-removal utility. Different selected sets therefore often have similar measured impact.
This supports using TraceAttention to prioritize sources for investigation. When a development decision depends on the effect of removing a particular source, physical deletion can then verify that candidate directly. Our additional screening experiment in Q1 measures the cost and coverage of this workflow and is now reported in Appendix F.6 (Table 26).
W2. Actionable development use
For a suspected hallucination or prompt-injection failure, a developer can use TraceAttention to investigate which retrieved passages or tool messages influenced the problematic output. The shortlist helps decide which source to inspect, remove, or correct before rerunning the model.
Case C illustrates rapid candidate discovery: TraceAttention locates the relevant code location (B12) and reproduction output (B17) in 0.39 s of query time, versus 11.33 s for deletion. Its complete Top-5 set has lower removal utility (Table 16), motivating physical checks of the candidates before deciding which source to remove.
The systems benefit is making this inspect–test–revise workflow inexpensive enough to repeat across long histories. We have expanded this workflow discussion in Appendix F.7.
Q1. TraceAttention screening followed by physical-deletion verification
Yes. On 32 Qwen3.5 SWE examples, TraceAttention screening plus physical reranking retains 96.0% of exhaustive LOO's mean Top-1 removal utility while reducing attribution compute time by 4.6x.
Recall measures overlap with exhaustive deletion-LOO rankings, with k capped at the available block count. Top-1 utility measures the physical likelihood drop from deleting the selected single block; the verified workflow's retention has a paired-bootstrap 95% CI of [89.2%, 99.6%].
The unchanged shortlist rule selects the top min(n,max(5,⌈0.10n⌉)) TraceAttention-ranked blocks, averaging 29.7% of blocks. Verification improves their ordering. In 25/32 examples, the shortlist contains exactly five blocks, so its Top-5 membership stays fixed even when its internal ranking improves.
The 6.76 s includes both stages and excludes initial prefill. Complete-batch wall time, including loading and prefill, decreases from 1,057.3 s to 373.8 s (2.83x). We have added the screening workflow and its evaluation to Appendix F.6 (Table 26).
Q2. Development decisions and multimodal applicability
The workflow above supports concrete decisions such as correcting a misleading retrieved passage or testing removal of an instruction embedded in a tool response. TraceAttention identifies candidates cheaply; physical deletion can check their effect on the original response before the developer changes retrieval or tool handling. Case C provides an existing example of locating the relevant code and tool-output sources.
For multimodal decoders with token-addressable visual keys and values, the same replay mechanism can mask selected visual-token blocks while scoring a text response. The visual prefix is computed once and reused across queries. Information already mixed into other cached visual features remains available, so the same shared-state limitation applies. We have added this discussion to Appendix F.7.
Q3. Nemotron evaluation coverage
The exclusions occur during response parsing, before attribution: 210 responses end without a complete parseable reasoning-and-answer target. For the 46 format-valid responses, TraceAttention's LDS retention is 102.4%, with a paired-bootstrap 95% CI of [97.0%, 110.2%].
Importantly, the aggregate result remains above 90% after removing this setting entirely:
Thus, the headline retention result does not depend on the small Nemotron/InfiniteBench subset. Its individual estimate describes the format-valid responses. We have added these uncertainty and exclusion results to Appendix F.1 (Tables 17–18). Appendix A.4 now states the parsing criterion explicitly.