acceptodds
Withdrawn from ICLR 2027

High Influence Is Not New Influence: A No-Arbitrage Analysis of Multi-Capability Data Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When is one influence score per example enough to select data for several target capabilities? At rank one with a positive direction, fixed prices, task-wise max, round-robin, capability-level BIDS and greedy gap maximization select the same data, and near rank one a fixed linear price keeps a guaranteed share of every capability's best gain, tight up to a constant for two capabilities. On real payoffs, far from this regime, a fractional coverage ceiling bounds what one score loses. Beyond rank one, high influence is not new influence: to first order, a candidate is non-redundant only if no cost-matched mixture of selected data matches it on every capability. The replication program's dual prices capabilities by scarcity; with an inactive cap, the gap is the candidate's one-step, price-weighted advantage over every such mixture. On image benchmarks, the rules coincide on payoffs deformed to rank one; the one-step identity holds for small steps but stops predicting within 3% of the parameter distance training travels; and scalar influence falls below random sampling in all 32 targeted comparisons, a failure that first-order coverage flags, while gradient distance to the targets also ranks the other rules. Gradient matching is the most accurate rule that does not know the targets; our greedy rule is a probe, and we do not fine-tune language models.

Primary area:
learning theory
Keywords:
data selection, influence functions, data attribution, multi-capability data selection, linear programming duality, regret-minimizing sets, instruction tuning, learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1HUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when a single influence score is sufficient for data selection across multiple target capabilities. The authors first analyze the geometry of first-order capability improvements, showing that common selection rules coincide under a positive rank-one structure and deriving performance guarantees for near-rank-one cases. They then introduce replication cost and arbitrage gap to characterize whether a candidate provides additional first-order improvements beyond those achievable by the selected data, using LP duality to establish a local learning interpretation. Experiments on CV benchmarks, including RotatedMNIST, PACS, and OfficeHome, examine these theoretical findings. The results show that scalar influence can underperform random selection, while the predictive value of arbitrage gap is limited to local updates. Furthermore, first-order coverage helps identify poorly performing selection methods, whereas target-gradient distance better distinguishes stronger methods. Overall, the paper provides theoretical and empirical insights into the limitations of scalar influence-based data selection.

Critical strengths

The problem studied seems interesting. The experiments are extensive, with theory also provided.

Critical weaknesses
  1. The paper is a bit hard to read and follow. For example, the abstract is written in an ambiguous way. Taking the second sentence, i.e., "At rank one with a positive direction, fixed prices, task-wise max, round-robin, capability-level BIDS and greedy gap maximization select the same data, and near rank one a fixed linear price keeps a guaranteed share of every capability’s best gain, tight up to a constant for two capabilities." as an example, I personally cannot understand those terms such as "positive direction", "two capabilities" mean in their contexts. There are many similar reading gaps, making a human reader feel frustrated and the paper hard to follow. A throughout pass could improve this paper's readability significantly.

  2. The theoretical results look rather simple, and it's hard to decipher the connection with later experiments.

Questions for authors

A general recommendation is to do a careful sanity check and an overall proofread to polish and reorganize this paper so its presentation can be further improved.

What are the practical examples where the developed theory can be applied? How can the characterized region in Fig 1, with the detailed influence metrics, such as 0.565 vs.0.51 be used to improve data selection?

Reviewer_d2AQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the impact a singular training examples have on models with multiple capabilities, finding gains over simple influence rankings, and more mixed results against other ranking methodolgies.

Critical strengths

Setting: With the advent of multi-modal LLMs, selecting a small subset of data points which are representative of the entire population on a wide variety of tasks has become more important than ever.

Critical weaknesses

Resulting Model: The paper does not fully characterize the setting it studies in that there is no theoretical and experimental improvement of the models trained on the data their algorithm selects as opposed to other data sets.

Experiments The algorithm proposed does not outperform random sampling and gradient matching, and suggests that further factors not discussed in the paper are at play in this problem.

Questions for authors

Given that gradient matching performs better than the proposed algorithm in every setting, is there any reason to use the proposed algorithm? How does this method perform when scaled to modern data set sizes? How do models trained on the different data selection techniques differ?

Reviewer_6ZzLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study different ways to compute the influence of a data point in a multi-criteria setting where inclusion of data can change multiple different capabilities (coordinates).

the authors observe that scalar valued influence functions can be suboptimal in this setting, unless the capability vectors are collinear.

The authors therefore study a linear program that aims to reconstruct a target vector from existing vectors. This leads to a notion of arbitrage cost, the difference between the cost of recreating a vector from existing ones and the cost of including it.

Critical strengths

The first figure gives a helpful illustration of the core problem.

It makes sense to study influence in a multi-criteria setting.

The authors put forward a notion of arbitrage gap, which is the cost of replicating a data point from existing data points versus the cost of including it. The gap can be written as an LP, assuming points can be included in fractional ways. This is a reasonably natural notion.

The paper has some potential to inform data selection rules, even if not direclty so yet.

Critical weaknesses

Lots of AI-written passages that lack clarity for a human reader. It's not terrible and I can tell that the authors at least put in multiple iterations with Claude. But it's still not "deslopped" enough for a quality conference paper. The main problem is that AI tends to use notions and constructs internal to its reasoning trace and exposes them to the reader without a clear introduction or motivation. As a result, I found this paper pretty hard to read for something that's probably a relatively simple idea.

The main theorems and propositions are highly technical statements that follow from the linear programming formulation. I didn't find any of them interesting enough to really work through the proofs, or even to fully parse the statements exactly. It's the sort of technical statement AI would prove without sufficient high-level guidance or iteration on the statement.

The experiments are largely simulations of the theorems. Experiments like that are never very illuminating, because they recreate the conditions of the theorems and then show that the conclusions hold. But assuming the theorems are correct, this doesn't really add new insight. It would be more illuminating to test the theory empirically in new or unknown regimes.

I'm leaning weak rejection, but wouldn't oppose acceptance if others find the statements more illuminating than i do.

Questions for authors

BIDS is used without introduction. Would be good to briefly say what it is when first used.

Can you extract high-level statements from your theorems and propositions that are worth exposing? What are the main lessons the reader should take away?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.