- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an interaction-aware influence function that characterizes how training examples influence the target function. The authors challenge the additivity of first-order influence scores and propose to expand the target function to second order, yielding group influence estimator consisting of the standard first-order term plus a pairwise interaction term. The interaction term is then used for redundancy-aware greedy data selection. The paper evaluates the estimator on small-scale group attribution benchmarks and on Llama-3.1-8B instruction-tuning data selection.
Critical strengths
- The paper is clearly written and easy to follow.
- The paper studies an important and well-motivated limitation of standard influence functions.
- The derivation is clean. The resulting pairwise interaction term is interpretable and the estimator is intuitive.
- The proposed interaction-aware data selection method appears practical at Llama-3 scale.
Critical weaknesses
None
Questions for authors
- Can a simple diversity regularizer work as well when down-weighting similar examples for data selection? For example, if an example has many neighbors in its representation/gradient space, decrease its score.
- Is the method compatible to scalable methods such as LoRIF[1]?
None are critical questions that materially affect my assessment though.
[1] LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution. https://arxiv.org/abs/2601.21929
3 comments
Thank you for adding the new experiments. I would like to maintain the score.
We sincerely thank the reviewer for the positive assessment and the helpful suggestions. The comparison with diversity regularizers that the reviewer suggested made it clearer what the interaction term adds beyond diversity, and we are grateful that the review led us to run it.
We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that our method addresses an important limitation of standard influence functions through a clean derivation and is practical at Llama-3 scale.
We have uploaded a revised version of the paper. To make the new results easy to find, we collected all rebuttal experiments in Appendix J at the end (pages 34 to 39). In a future revision, we will move them to the appropriate places in the paper.
[Q1] A simple diversity regularizer vs. the interaction term
Adding a diversity mechanism to F lowers its held-out test loss, but our method still gives the lowest loss. We added this comparison to the selection experiment of Section 5.1 as Figure 12 on page 39. It includes the suggested regularizer and two other diversity mechanisms, class balancing and k-center. These mechanisms make the selected subsets cover the classes more evenly, but a gap to our method remains, and it is largest when few examples are selected. Our method reaches a lower loss than even class balancing, which selects from the classes most uniformly by construction. This suggests that the interaction term does more than diversify the selection.
[Q2] Compatibility with scalable low-rank influence methods such as LoRIF [1]
Our method is compatible with scalable low-rank influence methods. The interaction term is built only from gradients and the Hessian, and most scalable low-rank influence methods are designed to compute exactly these quantities efficiently [1,2,3,4]. In fact, our LLM experiments use the gradient projection of LoGra [2].
[1] Li, Shuangqi, et al. "LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution." NeurIPS 2026.
[2] Choe et al. What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions. NeurIPS 2025.
[3] Grosse et al. Studying Large Language Model Generalization with Influence Functions. arXiv:2308.03296, 2023.
[4] Schioppa et al. Scaling Up Influence Functions. AAAI 2022.