- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DeltaTrace decomposes the change in a fixed response's log-probability, caused by replacing selected source tokens with EOS, into signed token contributions that sum exactly to that change. It builds finite-difference rules for attention, softmax, RMSNorm, and Gated DeltaNet, and uses tiled/chunked computation to keep auxiliary memory linear in sequence length. It is evaluated on 13 tasks (1,243 examples) released with FlashTrace, using Qwen3-8B and Qwen3.5-9B.
Critical strengths
-
Contributions are signed and sum exactly to the measured score change.
-
One procedure covers both attention and Gated DeltaNet, so it works on hybrid models.
-
Strong results with linear auxiliary memory. On Qwen3-8B, DT has the best RISE on 10/13 tasks, the best MAS on 11/13, and the best NIAH/VT recall among seven methods. On Qwen3.5-9B, it has lower RISE and MAS than FlashTrace on 9/13 and 12/13 tasks.
Critical weaknesses
-
Only EOS is used as the reference, and no alternative reference is tested. The deletion evaluation also masks with EOS, and its fully masked endpoint is the same input as DT's reference.
-
Only two models, both from the Qwen family, are used.
-
Settings and hyperparameters for the baselines other than FlashTrace are not reported, and it is not stated whether their numbers were rerun.
-
For scalar nonlinearities, the secant rule has the same form as DeepLIFT's Rescale rule, and the symmetric product split coincides with Shapley/IG for a single product. Both DeepLIFT and IG are cited, but this overlap is not discussed, and neither appears in Tables 2-4.
Questions for authors
-
Which EOS token is used? Contexts filled with EOS may be out of distribution. How do results change with other references (e.g., PAD, mean embedding, deletion) or masking schemes?
-
Do the results hold on other model families or scales?
-
What are the settings of the baselines other than FlashTrace, and were their numbers rerun?
-
How does DT compare with DeepLIFT and IG under the same setup?
-
The local maps are not unique. How do results change with other choices, e.g., an asymmetric split for attention or a per-product symmetric split for the GDN recurrence?
-
What are the generation settings and length distribution of the stored responses? For Qwen3.5-9B, were the responses generated by the same model? How does attribution quality vary with length?
-
How well does a sentence's score sum match the effect of masking only that sentence?
We report reference and deletion tests, four more models, the baseline settings, DeepLIFT and IG, two other allocation rules, response length, and the sentence-level correlation.
Thank you for your insightful review. Addressing your comments has improved the paper. Where no task is named, tables report averages over all thirteen tasks.
W1, Q1. EOS is
<|im_end|>, theeos_tokenof the tokenizer. PAD is<|endoftext|>. Deletion shifts positions, so we trained one embedding vector so that replacing a span matches deleting it. The masked endpoint of the test is indeed the DT reference, so we also ran the test with PAD masking and with deletion. Test cells give RISE / MAS. KL is the divergence between the responses after replacing and after deleting a span, times 1000.EOS, PAD and the learned vector give close results, and the mean embedding is worse. EOS is closer to deletion than the other tokens we tried, and the learned vector is closer still. With these three references DT is below FlashTrace on both metrics under every test, except RISE under the PAD test, where EOS and the learned vector match FlashTrace. The deletion test has no reference as its endpoint, and Recall does not use the test. We will add these tests.
W2, Q2. We ran DT and FlashTrace on four more models. Kimi-Linear-48B is a second linear attention model. FlashTrace is set as in the paper. Cells give DT / FlashTrace.
DT has higher recalls in every cell. DT has lower RISE and MAS on Qwen3-14B and Qwen3-32B, Llama-3.1-8B and Kimi-Linear-48B. FlashTrace on Kimi is our port.
W3, Q3. The five baselines use the code and settings released with the FlashTrace benchmark. Perturbation and CLP replace ablated units with EOS, and REAGENT samples replacements from
longformer-base-4096. Units are sentences on MQ-Q2 and MoreHopQA, and segments withsource_k20 elsewhere. IFR useschunk_tokens128 andsink_chunk_tokens32, and AttnLRP usesneg_handlingdrop andnorm_modenorm. Tables 2 and 3 and the NIAH rows of Table 4 report the released results. We ran the VT and HotpotQA rows ourselves with the same settings. We will add this to App. C.W4, Q4. Both points are correct, and we will state them in Sec. 2.2. The table compares DT with DeepLIFT with the Rescale rule on the activation functions (Shrikumar et al., 2017) and with IG with 100 steps, under the same reference, target and tests.
IG is better than DT on RISE for three of thirteen tasks, on MAS for five, and on Recall for one of six NIAH tasks, and DT is better on the rest.
Q5. We ran both. The asymmetric split is ΔY=ΔPV0+P1ΔV. The other GDN rule applies Eq. 18 to each product of Eq. 15. The table gives the change from the paper rule.
The GDN rule changes almost nothing. The asymmetric split lowers RISE and MAS, which is better, and both recalls, which is worse. We will report both and state that the split is a choice.
Q6. The responses come with the benchmark. They were generated by
qwen3-235b-a22b-2507, not by Qwen3-8B or Qwen3.5-9B, and the release gives no decoding parameters. The table gives their length in tokens and DT results for the shorter and longer half of each task.RISE shows no trend with length. Recall is lower for longer responses on NIAH and unchanged on VT.
Q7. We masked each sentence alone and measured the drop in response log probability. The table gives its Pearson correlation with the DT sentence sum within each example.
The two agree on most tasks, least on MQ-Q8 and MATH. The sum is a share under joint masking, so the two need not be equal.