acceptodds
Withdrawn from ICLR 2027

Beyond Scalar Rankings: Nonlinear Attribution Aggregation for Data Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting high-quality training data is increasingly important when annotation, storage, and computation budgets limit how much of an available data pool can be used. Data-attribution methods support this process by assigning a value to each training example, but they are commonly paired with a top-m rule that compresses an example's effects into a scalar score, which can lead to redundant or poorly balanced subsets. In this paper, we introduce NADA (Nonlinear Aggregation of Data Attribution Components), a method-agnostic selection framework that preserves component-level attribution information and selects complementary subsets through nonlinear aggregation to obtain more informative training data. We demonstrate the effectiveness of our method on standard logistic-regression and ridge-regression models, as well as prompt-based fine-tuning models, showing that NADA outperforms the corresponding scalar top-score selector in most matched settings and demonstrating the advantage of nonlinear component aggregation in mitigating the information loss caused by scalar attribution ranking.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Data Selection, Training-data Attribution, Data Valuation, Nonlinear Aggregation, Influence Functions, Diversity-aware Subset Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TYnAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NADA, a nonlinear subset-selection method for component-level data attribution. Instead of reducing each training example to a single scalar attribution score, NADA applies an increasing concave function to accumulated component-wise scores, allowing a candidate's priority to depend on the examples already selected. The method is applied to influence functions, TracIn, and leave-one-out attribution, and is evaluated on logistic and ridge regression benchmarks as well as prompt-based fine-tuning tasks, where it generally outperforms the corresponding scalar top-score baseline.

Critical strengths
  1. The paper presents a clear and practical formulation that separates the subset-selection stage from the underlying attribution estimator. Using the same attribution matrix for both Top and NADA makes the comparison well controlled and allows the effect of the selection rule to be evaluated directly.
  2. The experiments also show that preserving component-level attribution information can outperform naive scalar top-score selection across several attribution methods and model families. This suggests that scalar aggregation may discard information that is useful for subset construction, although this empirical result alone does not establish strong novelty.
Critical weaknesses
  1. The main novelty is limited. NADA mainly extends the nonlinear component aggregation used in NASH from Shapley or semivalue scores to generic component-level attribution matrices. Closely related work has already considered imbalance, redundancy, diversity, and joint influence in data selection. I do not see a sufficiently strong conceptual or theoretical advance beyond this generalization, which is the main reason for my rejection recommendation.
  2. The empirical evaluation also does not establish a clear advantage over stronger existing selection methods. Most comparisons are against scalar Top selection, which performs poorly in many settings, while NADA is approximately on par with Random overall on the prompt fine-tuning benchmarks. Without direct comparisons to stronger joint or diversity-aware selection methods, it is difficult to tell whether NADA provides a meaningful improvement over existing approaches or mainly fixes a weak scalar aggregation baseline.
Questions for authors
  1. Could the authors clarify what is conceptually new in NADA beyond extending NASH-style nonlinear component aggregation from Shapley or semivalue scores to generic attribution matrices such as influence functions, TracIn, and LOO?
  2. Could the authors compare NADA directly with stronger joint or diversity-aware selection methods, such as NASH, BIDS, Quad, or related influence-aware subset-selection approaches?
  3. Are the validation examples used to construct the component attribution matrix disjoint from those used to report the final validation performance? If not, could the authors provide results on an independent held-out test set?
  4. How was the fixed choice of λ=2.5\lambda=2.5 selected, and how sensitive are the results to λ\lambda and to global max-absolute normalization? In particular, does the method remain stable when component scales differ substantially?
  5. Attribution-informativeness is defined through a calibration map fc,mf_{c,m}, while NADA generally optimizes a different concave function FcF_c. Under what conditions, if any, does optimizing the NADA objective improve or approximate the true subset utility, especially in the signed-score settings used in the main experiments?
Reviewer_7zndOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method-agnostic framework (NADA) that performs data subset selection using component-level attribution via nonlinear aggregation. The motivation behind this framework is to address the biases and information loss that occur during “top-k” data selection. Specifically, the framework accounts for component-level contributions using a “diminishing returns” objective. Experiments show that this method outperforms standard “top-k” selection across different attribution methods for both logistic regression models and fine-tuning tasks.

Critical strengths

The strengths of the paper are as follows:

  1. Effectively selecting the best subset is a common data attribution problem, and it is difficult to obtain that signal solely from individual components due to non-additive effects. This paper presents a lightweight framework built on top of existing individual attribution scores, avoiding heavy re-computation or expensive “group-level” attribution methods (such as AirRep, GREATS, etc.).
  2. The paper conducts experiments on both classic logistic regression models and LM fine-tuning, showing that the method is effective in both settings.
Critical weaknesses

My main concern with this paper is the lack of comparisons against appropriate baselines. In particular, there are two classes of baselines that I believe the paper should compare against:

  1. Probabilistic sampling: In many cases, weighted sampling over individual attribution scores mitigates the biases that arise in top-k selection by introducing diversity. Prior work has shown this (DATE-LM, MATES, etc.). In addition, probabilistic sampling is very lightweight, making it a natural comparison for the proposed framework.
  2. “Group-level” selection methods: These are methods that take into account the relationships between data samples and their attribution scores during the selection process. Examples include GREATS, AirRep, etc.

References

  1. DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models. NeurIPS 2025.
  2. MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models. NeurIPS 2024.
  3. GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration. NeurIPS 2024.
  4. Enhancing Training Data Attribution with Representational Optimization. NeurIPS 2025.
Questions for authors

None

Reviewer_uGbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NADA, a data-selection method that keeps component-level attribution scores instead of first reducing them to one score per training example. It then selects examples greedily using a concave objective, so that repeatedly covering the same components has diminishing value. NADA is applied to influence functions, TracIn, and leave-one-out attribution. It beats the corresponding Top rule in 134 of 144 logistic/ridge-regression comparisons and all 69 language-model fine-tuning comparisons.

Critical strengths

The idea is simple and broadly applicable. NADA leaves the attribution method unchanged and only replaces the final aggregation and selection rule. The comparison with Top is especially clean because both methods use the same attribution matrix. The motivation is also convincing. Summing each row into a scalar can hide whether the selected examples cover the same validation components repeatedly. The experiments cover several datasets, noise levels, and selection budgets, and the paper is candid that NADA does not consistently outperform random selection in fine-tuning.

Critical weaknesses
  1. Top stays at chance at every budget in many fine-tuning settings, which suggests it selects mostly one class. NADA is almost tied with Random there on average (75.68% versus 75.50%). The headline gains mainly show that NADA fixes Top, not that it beats random selection, except under label noise in the small regression experiments.

  2. The experiments omit NASH, the closest prior work, and standard coverage methods such as facility location and k-center. It is unclear whether NADA's attribution objective is better than generic diversity. Each nonrandom method is also run only once. There is no sensitivity analysis for λ\lambda, the concave function, or normalization.

  3. The fine-tuning selections are loaded from precomputed archives. The paper does not explain how exact LOO or influence scores were computed for the 7B and 20B models. Exact LOO as defined requires one retraining run per candidate. No runtime, memory use, or end-to-end cost is reported.

  4. The validation examples appear to be used both as attribution components and for evaluation. If so, NADA selects data for the same examples used to score it.

Minor comment: Liu et al. (2024), cited in Sec. 2, is "Rethinking Tabular Data Understanding with Large Language Models" (NAACL 2024), which studies how LLMs reason over tables. It does not support the claim that influence functions, TRAK, LESS, or Data Shapley underperform random selection. The citation should be corrected or the claim narrowed.

Questions for authors
  1. What are the label distributions of the subsets selected by Top, NADA, and Random? How does NADA compare with class-stratified Top and with Random across multiple seeds?
  2. How does NADA compare with NASH and with standard coverage or diversity methods under the same candidate pools and budgets? How sensitive is it to λ\lambda normalization, and the choice of concave function?
  3. Were the examples used as attribution components also used to report performance? If so, can the method be evaluated on a separate held-out set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.