- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how loss functions and token aggregation affect language-model data attribution. It introduces CATT, a controlled benchmark for tracing synthetic facts learned during fine-tuning, and evaluates several existing attribution methods. The main finding is that cross-entropy can suppress signals from confidently predicted tokens, while aggregation over an entire document can dilute useful local signals. Using negative log-odds (NLO) gradients and selecting high-scoring token positions substantially improves retrieval in several settings. The paper also includes an external remove-and-retrain evaluation.
Critical strengths
-
The paper is well written. The motivation is clear, and the token-level analysis makes the failure cases easy to understand. The relationship between cross-entropy and NLO gradients provides a useful explanation for the observed confidence effect.
-
The controlled benchmark and comparisons across multiple attribution methods are valuable. Studying redundancy, distractors, loss choice, and aggregation together helps identify practical design choices that can otherwise be overlooked.
-
The findings could help practitioners audit which fine-tuning documents support a learned fact and choose more effective attribution scores. I also appreciate the external retraining experiment, which evaluates changes in model behavior beyond document retrieval.
Critical weaknesses
-
Long-context practicality remains unclear. Appendix C.2 provides useful runtime measurements, so the computational overhead is acknowledged. However, attribution documents are truncated to 512 tokens, and the Function Binding documents are mostly only a few hundred tokens long. Could the authors provide a small runtime and peak-memory comparison across input lengths, including a longer-context setting, to show how much additional cost token-level scoring introduces? I don't think it hurt the main contribution though, just good to have a sense of how longer context contribute to the computation time.
-
The interpretation of replacing training gradients with NLO needs clarification. The model is trained with cross-entropy, and several methods retain cross-entropy curvature when their training gradients are replaced. This changes the perturbation being scored. The resulting ranking can still be useful, but its relationship to the influence of reweighting the original training documents is less clear. Could the authors state the corresponding perturbation explicitly and clarify when this interpretation is appropriate?
-
Document length explains a substantial part of the distractor result. In Table 14, summing NLO token terms raises EK-FAC recall from 0.18 to 0.99, nearly matching top-10 pooling at 1.00. The authors acknowledge this, but it deserves more emphasis in the main explanation. Please distinguish the gains from NLO, length normalization, and token selection; a length-matched distractor control would help isolate the last component.
-
The external evidence supports the proposed components unevenly. The strongest FTRACE retraining result comes from removing TRAK’s confidence factor , while both dense TRAK variants already use NLO gradients. The top-10 retraining variant also changes the projection, and Table 17 does not show an improvement from top-10 pooling over full aggregation in the matched layerwise retrieval comparison. Please clarify which component each external result validates and qualify the broader claims about token selection accordingly.
Questions for authors
- Please see above
2 comments
Response to the rebuttal.
After carefully reading the response from the author, most of my concerns are resolved. However, I believe my original score already reflect the merits of the paper. Therefore, I would retain my score.
Response to Reviewer EgMJ
Thank you for highlighting our writing, the clarity of our token-level analysis, as well as the value of CATT’s controlled comparisons and the external remove-and-retrain evaluation! We address weaknesses in detail below.
Long-context practicality
As suggested, we tested how scoring cost scales with document length by extending 32 training documents from 256 to 2,048 tokens and recording per-document scoring time and peak GPU memory for the same 20 queries on an 80 GB A100. We find that NLO essentially does not change scoring time/memory, while computing per-token scores for top-k pooling takes 2.1 to 4.4 times as long and costs additional memory overhead for EK-FAC. See the table below for full results.
Cells show scoring time (seconds per document) / peak GPU memory (GB)
Interpretation of NLO loss in IF
Thank you for asking about the theoretical interpretation of using NLO training gradients in influence functions. Holding the cross-entropy curvature fixed, the NLO influence score estimates the first-order change in query loss from adding a small weight of the document’s NLO loss to the CE training objective (see equation below).
CE perturbation:NLO perturbation:d′∈D∑LCE(d′;θ)+ϵLCE(d;θ),d′∈D∑LCE(d′;θ)+ϵLnlo(d;θ). We clarify this interpretation in Section 2.3 and provide the full derivation in Appendix A.2.
Separating distractor length effects from token selection
Thank you for pointing this out. As we now clarify in Section 3, full-document averaging dilutes the signal in longer ground-truth documents more than in our short distractors, so top-token pooling partly helps by removing this length penalty. However, top-token pooling also improves recall for several methods over full-document summation, which does not divide by document length (Appendix D, Table 15).
Regarding the suggested length-matched control, a key design constraint is that the distractors must follow the format of the held-out queries, and preserving this match would require artificially long queries for a simple function-output task. Adding filler to the distractors would keep queries short, but introduce additional content that could itself affect attribution.
Generalization of loss and top-token interventions
Thank you and reviewer 4UoY for asking how well each intervention generalizes beyond CATT. To address this, we evaluated three additional attribution methods on FTRACE-TREX and added a second external evaluation, BackdoorLLM [1].
On FTRACE-TREX, with full-document scoring, negative log-odds improved all methods, increasing the accuracy drop after removal and retraining by up to 32 percentage points. With loss held fixed, top-10 pooling yielded small gains for EK-FAC and Grad-Dot, but reduced the drop for Layerwise TRAK (Section 3 and Appendix E of the revision).
To test whether our interventions extend from facts to behaviors, we use BackdoorLLM, where attribution methods must trace a harmful behavior (jailbreaking or refusal) back to poisoned finetuning examples. The NLO loss correction improved attribution across the board for both behaviors, while top-10 pooling helped jailbreak attribution under cross-entropy and gave mixed results for refusals (Section 3 and Appendix F of the revision).
Across these external evaluations, the loss correction improves attribution consistently, while gains from top-token pooling are more variable, being dependent on how the attribution signal is spread within documents. As discussed in the limitations, we leave dynamic thresholding of the top-token budget to future work.
[1] Li et al. BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models.
Thank you again for taking the time to review the paper and providing helpful feedback! Do the above explanations and revisions address your concerns with the paper? If not, what further clarification or modifications could we make to improve it?