acceptodds
Under review as a conference paper at ICLR 2027

Interaction-Aware Influence Functions for Group Attribution

43 views

Abstract

Influence functions approximate how removing a training example changes a quantity of interest, called the target function, such as a held-out loss. To estimate the influence of a group of examples, the standard practice is to sum the individual influences of its members. However, this sum does not capture how examples jointly affect the target: a pair of examples may be redundant or complementary, but the sum cannot distinguish these cases. We propose an interaction-aware influence function that characterizes how interactions between examples influence the target. By expanding the target to second order around the trained parameters, we obtain an estimator that augments the standard sum with a pairwise interaction term that captures the alignment between two examples' effects on the target. We empirically evaluate our estimator in two settings. First, on six dataset-model pairs spanning logistic regression, MLPs, and ResNet-9, our estimator tracks leave-group-out retraining substantially better than first-order influence across all settings. Second, when used as a greedy algorithm for offline selection of instruction-tuning data on Llama-3.1-8B, it outperforms prior influence-based and representation-similarity baselines on five of seven downstream tasks, in a regime where standard influence-based selection underperforms random selection. Code is available at https://anonymous.4open.science/r/Interaction_IF-45D6.

Primary area:
interpretability and explainable AI
Keywords:
influence functions, data attribution

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pyUUOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an interaction-aware influence function that characterizes how training examples influence the target function. The authors challenge the additivity of first-order influence scores and propose to expand the target function to second order, yielding group influence estimator consisting of the standard first-order term plus a pairwise interaction term. The interaction term is then used for redundancy-aware greedy data selection. The paper evaluates the estimator on small-scale group attribution benchmarks and on Llama-3.1-8B instruction-tuning data selection.

Critical strengths
  • The paper is clearly written and easy to follow.
  • The paper studies an important and well-motivated limitation of standard influence functions.
  • The derivation is clean. The resulting pairwise interaction term is interpretable and the estimator is intuitive.
  • The proposed interaction-aware data selection method appears practical at Llama-3 scale.
Critical weaknesses

None

Questions for authors
  • Can a simple diversity regularizer work as well when down-weighting similar examples for data selection? For example, if an example has many neighbors in its representation/gradient space, decrease its score.
  • Is the method compatible to scalable methods such as LoRIF[1]?

None are critical questions that materially affect my assessment though.

[1] LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution. https://arxiv.org/abs/2601.21929

3 comments

We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that our method addresses an important limitation of standard influence functions through a clean derivation and is practical at Llama-3 scale.

We have uploaded a revised version of the paper. To make the new results easy to find, we collected all rebuttal experiments in Appendix J at the end (pages 34 to 39). In a future revision, we will move them to the appropriate places in the paper.

[Q1] A simple diversity regularizer vs. the interaction term

Adding a diversity mechanism to FF lowers its held-out test loss, but our method still gives the lowest loss. We added this comparison to the selection experiment of Section 5.1 as Figure 12 on page 39. It includes the suggested regularizer and two other diversity mechanisms, class balancing and kk-center. These mechanisms make the selected subsets cover the classes more evenly, but a gap to our method remains, and it is largest when few examples are selected. Our method reaches a lower loss than even class balancing, which selects from the classes most uniformly by construction. This suggests that the interaction term does more than diversify the selection.

[Q2] Compatibility with scalable low-rank influence methods such as LoRIF [1]

Our method is compatible with scalable low-rank influence methods. The interaction term is built only from gradients and the Hessian, and most scalable low-rank influence methods are designed to compute exactly these quantities efficiently [1,2,3,4]. In fact, our LLM experiments use the gradient projection of LoGra [2].

[1] Li, Shuangqi, et al. "LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution." NeurIPS 2026.
[2] Choe et al. What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions. NeurIPS 2025.
[3] Grosse et al. Studying Large Language Model Generalization with Influence Functions. arXiv:2308.03296, 2023.
[4] Schioppa et al. Scaling Up Influence Functions. AAAI 2022.

Thank you for adding the new experiments. I would like to maintain the score.

We sincerely thank the reviewer for the positive assessment and the helpful suggestions. The comparison with diversity regularizers that the reviewer suggested made it clearer what the interaction term adds beyond diversity, and we are grateful that the review led us to run it.

Reviewer_rGq3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies group data attribution and selection, where additive influence estimates cannot express redundancy or complementarity among examples. It retains a first-order parameter-shift approximation but expands the target loss to second order, producing pairwise interactions between inverse-Hessian-preconditioned gradients and an efficient greedy offline selector. Experiments report improved attribution correlations across six dataset–model settings and the best scores on five of seven Llama-3.1-8B instruction-selection tasks.

Critical strengths
  1. Non-additivity of group effects is a known limitation of IFs, and redundancy in top-k selection is a real failure mode in practice. The correction is closed-form and easy to add to existing IF pipelines.
  2. The method scales to an 8B model with a 270K-example pool, with LLM results over five seeds.
  3. The paper is well organized and easy to follow.
Critical weaknesses
  1. GREATS [1] and second-order In-Run Data Shapley [2] already model interactions, although their update/trajectory objectives differ from retraining effects. F+α\alpha·self and H_f→I controls would clarify the specific cross-term contribution beyond the existing self/cross ablation.
  2. The omitted response term needs a full-expansion check, while Sec. 5.1's complement-removal interpretation requires verification under weight decay.
  3. JI$^2$S [3] also adds pairwise joint-influence terms, weighted by a test-set Hessian, to LESS-style [4] warmup gradients and selects a fixed subset offline, yet it is only cited. It would be better to compare with it in Sec. 5.2, together with an offline adaptation of GREATS [1], to show what H⁻¹ preconditioning and greedy updates add.
  4. Attribution uses only 50 output-neighbor groups of one size per setting, reporting Spearman without uncertainty. Alternative groups, magnitude/sign accuracy, and paired intervals would strengthen the evidence.

[1] Wang et al. GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration. NeurIPS 2024.

[2] Wang et al. Data Shapley in One Training Run. ICLR 2025.

[3] Wei et al. JI$^2$S: Joint Influence-Aware Instruction Data Selection for Efficient Fine-Tuning. EMNLP 2025.

[4] Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.

Questions for authors

See weaknesses.

4 comments

We sincerely thank the reviewer for the thoughtful review and for the time devoted to evaluating our paper. We also appreciate that the reviewer highlighted that our closed-form correction addresses a real failure mode of top-kk selection and scales to an 8B model.

We have uploaded a revised version of the paper. To make the new results easy to find, we collected all rebuttal experiments in Appendix J at the end (pages 34 to 39). In a future revision, we will move them to the appropriate places in the paper.

[W1] Contribution of the cross term with F+α⋅selfF+\alpha\cdot\text{self} and Hf→IH_f \to I controls

Both controls improve over FF but perform worse than our method in all six settings. The results are in Figures 9 and 10 on page 38.

  • F+α⋅selfF+\alpha\cdot\text{self} control. This control removes the cross term and keeps only the self term scaled by α\alpha. With the best α\alpha it improves over FF in every setting. However, our method achieves a higher correlation than this control in all six settings, and the gap is large except on MNIST and FMNIST.

  • Hf→IH_f \to I control. This control keeps the cross term but replaces HfH_f with the identity matrix. The correlation still improves over FF in all six settings, which shows that accounting for interaction matters by itself. Using HfH_f adds a further gain in all six settings, and the gain is large except on MNIST (MLP) and FMNIST (MLP).

[W2-1] Full-expansion check of the omitted response term

We computed the full expansion in logistic regression, and using it did not lead to an improvement in correlation, as the table below shows. We restricted the check to logistic regression because the full expansion contains the third derivative of the training loss, which is hard to compute for neural networks. For ReLU networks this term is not well defined, and it lacks established substitutes such as the EK-FAC approximations available for the Hessian. Five of our six settings use ReLU networks, and logistic regression is the one setting of ours in which the third derivative can be obtained exactly. We added this table as Table 10 on page 34.

SettingFFF+BF+BF+I+BF+I+BFull expansionOurs (F+IF+I)
MNIST (LR)0.540.600.880.880.90

[W2-2] Complement-removal interpretation of Sec. 5.1 under weight decay

We would first like to clarify that the paper does not claim a complement-removal interpretation for Sec. 5.1. As the reviewer points out, however, training on the selected subset SS alone is equivalent to removing its complement from the pool, and this is an insightful point.

Without weight decay, adding SS under our selection score and removing its complement from the pool give the same estimate. With weight decay β\beta they differ. The complement-removal estimate is obtained by replacing the target gradient ∇θf(θ^)\nabla_\theta f(\hat\theta) in our score with ∇θf(θ^)−βHfH−1θ^\nabla_\theta f(\hat\theta) - \beta H_f H^{-1}\hat\theta, while the interaction term stays the same.

We recomputed the selection of Sec. 5.1 with this correction. The first-order scores with and without it have a Spearman correlation of 0.92 on MNIST and 0.98 on FashionMNIST. After retraining on the new subsets, the held-out test loss closely follows the original curve. We added this comparison as Figure 13 (a) on page 39.

[W3] Comparison with JI$^2$S and an offline version of GREATS

Our method performs significantly better than JI$^2$S and an offline version of GREATS on six of the seven tasks. The results are in the table below, which we added as Table 13 on page 36.

MethodGSM8KAQuAARC-EHellaSwagPIQAECQASQuADAvg.
Offline GREATS47.22 ± 0.4323.78 ± 0.6688.99 ± 0.1258.78 ± 1.0468.05 ± 1.0869.94 ± 0.2885.00 ± 0.1263.11
JI²S35.72 ± 0.9324.57 ± 0.7788.67 ± 0.2162.12 ± 0.6769.06 ± 1.2069.60 ± 0.2585.03 ± 0.1262.11
Ours52.69 ± 0.9630.94 ± 0.7290.44 ± 0.1163.43 ± 0.3479.39 ± 0.3271.17 ± 0.1979.87 ± 0.3466.85

Offline selection asks how the model will turn out once it is trained on the selected data. Our method approximates the effect of full retraining, whereas the two methods approximate the effect of a one-step update [1, 2]. We therefore believe our method is the better fit for offline selection, and the results support this. For selection performed at every training step, the one-step approximation of GREATS may be better suited. Appendix I discusses the difference between the approximation targets of GREATS and our method in more detail.

[W4] Alternative groups, paired intervals, and magnitude and sign accuracy

Alternative groups. Our estimator matches FF on random groups and stays above FF on output-neighbor groups as they become more diverse. We added this comparison as Figure 11 on page 39. It compares FF and our estimator on random groups and on output-neighbor groups built around one, two, or three centers, where more centers make a group more diverse.

In a random group the examples are unrelated, so the interactions between them are weak and average out, and the total interaction is nearly the same for every group. Our estimator therefore matches FF on random groups. When a group is built around a single center, the examples are similar, so their interactions are strong and reinforce each other. The advantage of our estimator remains as the groups become more diverse. It is higher in all six settings with one center and in five of six settings with two or three centers.

Section 4 evaluates groups of examples with similar outputs because our target application is data selection, where the group that matters is the selected subset. Selecting the top-kk examples by influence produces subsets of similar examples, as the class entropy in Section 5.1 shows.

Paired intervals. The gain of our estimator over FF in Figure 1 is statistically significant in five of the six settings. We computed 95% paired bootstrap intervals for the difference in Spearman correlation on the groups of Figure 1 and added them as Figure 13 (b) on page 39. The interval excludes zero in every setting except MNIST (MLP).

Magnitude accuracy. Neither FF nor our estimator predicts the magnitude of the loss change well. We regressed the retrained loss change on the predicted change and report the slope below, where a slope of 1 means that the magnitude is right. The slope is close to 1 only for our estimator on MNIST (MLP). We added this table as Table 11 on page 35.

MNIST (LR)MNIST (MLP)FMNIST (MLP)Concrete (MLP)Parkinsons (MLP)CIFAR-10 (ResNet-9)
Slope of FF12.584.260.038−0.580.281.8e−3
Slope of Ours9.080.840.0380.190.211.0e−5

This is a known behavior of influence functions, which misestimate the magnitude of group effects but still rank them well [3]. Attribution methods are therefore commonly evaluated by rank correlation [4], and ranking is also what applications such as data selection require. Predicting the magnitude accurately remains important future work.

Sign accuracy. Our estimator predicts the direction of a group’s effect more reliably than FF in all six settings. The table below reports how often the predicted sign matches the sign of the retrained loss change. We added it as Table 12 on page 35.

MNIST (LR)MNIST (MLP)FMNIST (MLP)Concrete (MLP)Parkinsons (MLP)CIFAR-10 (ResNet-9)
FF0.740.720.540.720.570.61
Ours0.850.770.710.880.631.00

[1] Wang, Jiachen T., et al. "Greats: Online selection of high-quality data for llm training in every iteration." NeurIPS 2024
[2] Wei, Jingyu, et al. "JI2S: Joint Influence‐Aware Instruction Data Selection for Efficient Fine‐Tuning." EMNLP 2025
[3] Koh, Pang Wei W., et al. "On the accuracy of influence functions for measuring group effects." NeurIPS 2019
[4] Park, Sung Min, et al. "Trak: Attributing model behavior at scale." ICML 2023

Thank you for conducting the additional experiments and revising the manuscript. The new control experiments and comparative analyses have strengthened the work and addressed my concerns. I am therefore raising my rating to 3.

We sincerely thank the reviewer for the constructive feedback and the positive assessment. The suggested control experiments and comparative analyses were highly informative, and we are grateful that the review pushed us to carry them out.

Reviewer_WK4SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work presents an interaction-aware influence estimation method, which is based on a second-order Taylor expansion of the target function change. The authors propose a greedy selection algorithm based on the proposed influence estimator, which gradually collects a subset of training samples. Validation of the method is performed in a leave-group-out setting and demonstrates high correlation between the loss change and the estimated influence. Testing on small-scale models demonstrates that the method selects good training samples for a smaller allocated budget (Figure 3). The authors also apply the method to the task of instruction tuning on the 5% most influential examples and demonstrate better performance than the selected influence-based methods.

Critical strengths
  1. The authors propose a mathematically formalized estimator of sample influence and a greedy algorithm for selection. On a set of applications, the authors demonstrate the usefulness of the estimator and the algorithm.
Critical weaknesses
  1. The work barely touches the question of the limitations of the method (Formula 7). The authors reference the work of Basu et al. and provide additional derivations in Appendix D.1, but do not provide practical details of when the second (interaction) term of Formula 7 is meaningful. For |S| << N the first-order term dominates. But when |S| ~ N, the third-order and Basu terms may play an important role. Therefore, practical guidance on when the approximation is applicable would be useful.

  2. Figure 1 conveys little information. It would be better to present scatter plots of estimated influence vs. loss effect. This would demonstrate better how the second-order corrections I and B shift the point distribution.

  3. Table 1. It is strange that the performance of the other methods is below the Random baseline. Could you consider additional LLMs to run the experiment?

Questions for authors
  1. Could your method be applied to the task of detection of detrimental/noisy samples? It would be nice to see the performance of an LLM in such a setting in comparison to other influence estimators like DataInf, EK-FAC, TracIn, for instance.

  2. Figure 3 (class entropy) shows that first-order selection F suffers from a diversity problem: it picks the most individually influential examples but with a strong class skew. Since the interaction term acts as a redundancy penalty, it is unclear whether its gain comes from the curvature-weighted pairwise structure or simply from enforcing diversity. Could you compare against first-order influence combined with some explicit diversity mechanism (class balancing, k-center, etc)?

4 comments

We sincerely thank the reviewer for the thoughtful review and the time devoted to our paper. We also appreciate that the reviewer highlighted the usefulness of our mathematically formalized estimator and selection algorithm across a set of applications.

We have uploaded a revised version of the paper. To make the new results easy to find, we collected all rebuttal experiments in Appendix J (pages 34 to 39). In a future revision, we will move them to the appropriate places.

[W1] Practical guidance on when the approximation is applicable

We recommend using our approximation for groups smaller than about 10% of the training set. Figure 6 on page 37 shows the Spearman correlation of the influence function methods at group sizes of 4%, 8% and 16% of the training set. At 8%, ours is the most accurate in all six settings.

However, at 16% the correlation of every estimator drops sharply on MNIST and FashionMNIST. Influence functions are a local approximation around the trained parameters, and we conjecture that removing a group of this size moves the model outside the region where this approximation holds. As the reviewer notes, the Basu term and the third-order term are effective for large groups in a general Taylor approximation, but influence functions break down before the group is large enough for these terms to matter.

[W2] Scatter plots of estimated vs. ground-truth influence

We added the scatter plots as Figure 8 on page 38. The F+BF+B panels look almost the same as the FF panels. The F+IF+I panels differ, most visibly for groups with a large ground-truth influence. In several settings, FF underestimates the influence of these groups with a prediction near zero, so they sit along the upper part of the y axis. With our method this underestimation is reduced, and these points shift to the right, toward the trend line. The gain in Spearman correlation reported in Figure 1 is therefore also visible in the scatter plots.

[W3] Why the baselines fall below Random in Table 1

LESS, NV-Embed and RDS+ fall below Random on average in Table 1 because each of them collapses on a single task, and the results of prior work also show similar drops for several selection methods [1]. In our experiments, LESS and NV-Embed collapse on AQuA, and RDS+ collapses on SQuAD, while on the other six tasks each of them is above Random on average. We note that our method achieves the highest average performance even when AQuA and SQuAD are excluded.

The collapses come from the candidate pool, which mixes data from several sources. When a method selects data whose answers look different from those of the target task, the fine-tuned model stops answering in the format the task expects. AQuA, for example, is answered with one of five option letters, and the models fine-tuned on the LESS and NV-Embed selections often fail to follow this format, so their accuracy drops sharply.

[Q1] Detecting detrimental or noisy examples

In an LLM label-noise experiment, our method detects the noisy examples about as well as the other methods, but removing the examples it flags and retraining lowers the loss on held-out examples the most. We flipped the answers of 20% of the ARC-Easy training examples (450 of 2,251) and fine-tuned Llama-3.1-8B on them. Each method then selects the 450 training examples it estimates to be most harmful to the target loss, which is the loss on 570 held-out ARC-Easy examples. The results are in the table below, which we added as Table 14 on page 36.

OracleDataInfEK-FACTracInAdditive IFOurs
Detection precision1.0000.8670.8710.8670.8730.871
Change in target loss after removal-0.031-0.030-0.032-0.034-0.031-0.036

All methods detect the flipped examples about equally well. The best and the worst method differ by three examples out of 450. When we retrain without the 450 selected examples over three retraining seeds, EK-FAC, TracIn and our method lower the target loss more than the oracle, which removes exactly the 450 flipped examples, and our method lowers it the most in all three seeds. We conjecture that some correctly labeled examples hurt the target more than some of the flipped ones, and that our method finds them.

[Q2] First-order influence with an explicit diversity mechanism

Adding a diversity mechanism to FF lowers its held-out test loss, but our method still reaches the lowest loss. We added this comparison as Figure 7 on page 37. Class balancing and k-center both reduce the tendency of FF to select from only a few classes, but a gap to our method remains, and it is largest when few examples are selected. Our method reaches a lower loss than even class balancing, which selects from the classes most uniformly by construction. This suggests that the interaction term does more than balance the classes.

[1] Ivison, Hamish, et al. "Large-scale data selection for instruction tuning." arXiv preprint 2025.

I appreciate the response and the additional work provided.

[W1] The weakness of the provided suggestion is that it is based on empirical results only, on several selected datasets.

Figure 6 also demonstrates that the 10% threshold may not hold. On Parkinsons and Concrete the correlation does not drop at 16%, and on CIFAR-10 the proposed estimator remains the best at 16% but is the least accurate at 4%.

I suppose, a better analysis of the approximation limitations could provide some theoretical bounds, through the ratio of terms in the Taylor expansion.

[W2] Figure 8 addresses my request.

The proposed interaction term I shifts the distribution more than the second-order term B, which barely affects the influence scoring. It seems that the magnitudes of the ground-truth influence and the approximated one are not aligned, only the ranking is. Please add axis scales and the identity line to the panels. Including the Spearman correlation in the chart would make it even more informative.

[W3] I do not find this response sufficient.

The explanation that the methods fail to follow the format is stated as a hypothesis. Neither the response nor the revised appendix provides numerical evidence for it. Plausible evidence would have been the rate of format-following answers per method and task, and the accuracy on correctly formatted answers.

If the answer format is indeed the cause, then Table 1 partly reflects the format alignment of the selected data and not only its quality, which is important to state in the work.

The response also does not cover Additive IF, the baseline closest to the proposed method. The request to repeat the experiment with an additional LLM was not addressed. My concern about this experiment remains.

[Q1] My question was addressed partially.

The authors conducted an experiment on the ARC-Easy benchmark with flipped labels. The method has the same detection level as the other known methods. Several points that were missed:

  1. Accuracy on the target classification task before and after filtering (a third row in Table 14);
  2. Standard deviations over the 3 random seeds, to see if the differences are meaningful.

ARC-Easy may be too easy to make the comparison sensitive. An additional analysis of the overlaps between the detected sets would have allowed a more detailed comparison of the methods.

[Q2] The question is addressed well.

Figure 7 shows that the proposed method reaches a lower loss than F with class diversification methods, especially for small budgets.

We are sincerely grateful to the reviewer for clarifying in detail what was addressed and what was not, and for giving us the opportunity to discuss these points further. The feedback was truly helpful, and we appreciate the effort put into evaluating our paper.

We have uploaded a revised version of the paper with the additional results on pages 40 and 41, and with the updated Table 14 on page 36. As with the results added in our previous revision, we will move them to appropriate places in the paper later. Due to the limited time, some of the explanations in this response are not yet reflected in the paper, and we will include them in the next revision.

[W1] Practical guidance on when the approximation is applicable

We agree with the reviewer that the 10% threshold we proposed as practical guidance does not hold across all datasets, and we thank the reviewer for suggesting a theoretical analysis.

Following the suggestion, we measured the ratio of the Taylor terms ∣I∣/∣F∣|I|/|F| for groups of 4%, 8% and 16% of the training set in the six settings of Figure 1 (Figure 15 on page 41). A Taylor approximation is generally expected to become less accurate as the second-order term grows relative to the first-order term. However, we observed this trend clearly only on MNIST (LR), so the ratio is difficult to use as a criterion.

We therefore keep the 10% guidance for now, and we will present it in the next revision only as a conservative baseline that does not hold for all datasets. Our estimate still works well above 10% on Concrete and Parkinsons, but it breaks down at 16% on MNIST and FMNIST, so 10% is a conservative choice. At 4% and 8%, the Spearman correlation of our estimate with the ground truth is between 0.28 and 0.85 in every setting except CIFAR-10 at 4%.

[W2] Scatter plots with axis scales, identity line and Spearman correlation

We added the axis scales, the identity line and the Spearman correlation to the scatter plots as Figure 14 on page 40. To make the scatter plots richer, we measured 100 groups for these plots, so the correlations differ from those in Figure 1. Our method still has the highest Spearman correlation in all six settings.

The magnitudes are not well aligned. This is a known behavior of influence functions, which misestimate the magnitude of group effects but still rank them well [1]. Attribution methods are therefore commonly evaluated by rank correlation [2], and ranking is also what applications such as data selection require. Predicting the magnitude accurately remains important future work.

[W3] Empirical evidence for the answer format hypothesis in Table 1

We agree with the reviewer that our previous answer was a hypothesis and that it needs empirical evidence to support it. To provide this evidence, we measured the rate of format-following answers and the accuracy on correctly formatted answers, as the reviewer suggested.

Rate of format-following answers. On AQuA, LESS and NV-Embed follow the answer format in only 31.6% and 29.3% of the answers, while the other methods do so in at least 73.7% (Table 17 on page 40). This supports our hypothesis that these two methods score low on AQuA because they fail to follow the answer format. On the other multiple-choice datasets (ARC-E, HellaSwag, PIQA and ECQA), every method follows the format in at least 99.7% of the answers, so the performance differences on those tasks do not come from the format. We did not measure the rate for GSM8K and SQuAD because they have no format constraint.

Accuracy on correctly formatted answers. When only the correctly formatted answers are evaluated, our method still has the highest average performance, and the averages of LESS and NV-Embed rise above Random. The results are in Table 15 on page 40 of the revised paper. As the reviewer pointed out, Table 1 partly reflects how well each method follows the format, and we will state this in the next revision.

Why RDS+ fails on SQuAD. RDS+ scores low on SQuAD because it selects data with more words than the other methods, while SQuAD measures how accurately a model produces short answers, whose median length is two words [3]. With RDS+, the median length of the generated answers is 8.2 words. With the other methods, it is between 2.0 and 3.0 words (Table 18 on page 41).

Why Additive IF falls below Random. The drop of Additive IF below Random shows that selecting data by individual influence alone is not enough. Prior work reports similar drops [4, 5, 6] and gives a similar analysis [4, 5]. Additive IF is the ablation of our method that removes only the interaction term, so it selects the subset with the largest sum of individual influences. The same happens in Section 5.1, where selection by the first-order term alone also falls below random selection.

Additional LLM. The suggested experiment on an additional LLM requires more than 200 fine-tuning runs, and we did not have the capacity to run it alongside the other experiments during the rebuttal period. We kindly ask for the reviewer's understanding.

[Q1] Standard deviations, target accuracy and overlap in noisy-example detection

We revise the conjecture of our previous response, thanks to the analyses that the reviewer suggested. In our previous response, we saw that our method lowers the target loss the most and conjectured that it also finds correctly labeled examples that hurt the target. The results below do not support this conjecture.

Standard deviations and target accuracy. With the standard deviations and the target accuracy added (Table 14 on page 36), our method has the lowest target loss, but the target accuracy of all methods is between 88.3% and 88.6%, which is the same within the standard deviations. Therefore, from this result alone, we cannot say that our method detects flipped labels better than the other methods. We believe this is because interaction information is not much needed to detect outlier examples. An outlier example can be identified by looking at that example alone.

Overlap between the detected sets. Table 16 on page 40 shows the number of examples shared between the sets detected by each pair of methods. Every pair of the five methods shares at least 424 of the 450 examples. Our method and Additive IF, which removes the interaction term, share all selected examples except five. That is, the interaction term had almost no effect on the selection in this experiment.

[1] Koh, Pang Wei W., et al. "On the accuracy of influence functions for measuring group effects." NeurIPS 2019
[2] Park, Sung Min, et al. "Trak: Attributing model behavior at scale." ICML 2023
[3] Rajpurkar, Pranav, et al. "Squad: 100,000+ questions for machine comprehension of text." EMNLP 2016
[4] Wang, Jiachen Tianhao, et al. "Data shapley in one training run." ICLR 2025
[5] Dai, Qirun, et al. "Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities." EMNLP (Findings). 2025
[6] Jiao, Cathy, et al. "DATE-LM: Benchmarking data attribution evaluation for large language models." NeurIPS 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.