acceptodds
Under review as a conference paper at ICLR 2027

When Is Gradient-Based Data Selection a Property of the Predictor?

Abstract

Gradient-based data selection is commonly interpreted as identifying examples that are informative for a given predictor, yet neural-network parameters are not unique representations of the predictor. We study predictor-level well-definedness of the resulting discrete selection under function-preserving reparameterizations. Across BADGE, CRAIG, and GradMatch, equivalent predictors replace – of a query batch while their logits remain unchanged to numerical precision; in end-to-end LoRA fine-tuning, factor-coordinate selection replaces – of MiniLM queries and, in the analytical factor-basis comparison, on average for ModernBERT. Per-example gradients transform as covectors, so a fixed parameter-space metric generally generally changes the geometry seen by the selector. We characterize when Gram geometry is preserved exactly or up to scale, and give selector-specific constructions in which the resulting distortion crosses a discrete selection boundary. For Fisher-based selection, we further show that a coordinate-fixed prior can break invariance even when the Fisher terms transform covariantly. Finally, repeated acquisition reveals that query non-identifiability and downstream utility are distinct: acquisition trajectories can diverge strongly while terminal accuracy remains similar in the settings studied. These results identify parameterization as a source of data-selection non-identifiability that is separate from conventional questions of subset utility.

Primary area:
optimization
Keywords:
gradient-based data selection, active learning, data subset selection, identifiability, reparameterization invariance, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_NXSCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses gradient-based data selection methods such as CRAIG and GradMatch. It aims to show that these methods do not actually identify informative samples needed for learning the function, but are more biased towards the model's parameters. To prove this, the paper applies function-preserving reparameterizations (for example, scaling RELU) to models. So while these are computationally invisible (the models still return the same result), the paper shows that the subsets selected using these models and gradient-based data selection varied. The accuracies of these models trained on these subsets remain similar, however. So the paper argues that data "identifiability" and "downstream utility" are fundamentally disconnected.

Critical strengths

The paper is easy to follow and uses mathematical theorems to prove points where needed. It discusses the difference between picking specific data points and actually training a successful model. Discussing this difference is important because data efficiency is a major topic in machine learning right now.

Critical weaknesses

The paper has a few key weaknesses to consider. For example, the paper assumes that because the algorithms select completely different data when the weights change, the methods must be flawed. But in redundant datasets, it could be possible that different data subsets form the same decision boundary. This point needs to be explored and discussed. Along the same lines, the paper studies gradient-based selection methodologies, which are approximations. So, while discussing the difference between data points and training a model, there should be a discussion with regard to true data estimation methods such as true influence functions that require the Hessian inverse at optimal weights. Additionally, many of the experimental results include simple datasets like MNIST or Fashion-MNIST, which are very simple and don't reflect complex real-world situations.

Questions for authors
  1. Since different data subsets can easily form the same decision boundary without hurting the final accuracy, is the selection method actually flawed, or is it simply picking equally useful examples from redundant datasets?
  2. Isn't it already obvious that data selection changes when the model's weights change, considering that these gradient methods are mathematically built entirely on those weights?
  3. Because these gradient methods are only approximations, how do they actually compare to true data estimation methods, such as true influence functions that require the Hessian inverse at optimal weights?
  4. Does this problem actually affect the massive generative language models widely used today? Could there be some empirical evidence added to the paper?
4 comments
AuthorsOpenReview ↗

Response to Reviewer NXSC (1/2)

Thank you for your review. You found the paper "easy to follow", using "mathematical theorems to prove points where needed", and considered its distinction between picking specific data points and training a successful model important "because data efficiency is a major topic in machine learning right now". Your questions on influence functions (Q3) and on generative language models (Q4) led to two new experiments (App. L.11, L.12).

Your commentShort answerWhere in the revised PDF
W1 / Q1. Is the selector flawed, or picking equally useful examples from redundant data?Consistent with the latter; we do not claim the selectors are flawed. Our claim concerns which examples are pickedSec. 1, ll. 73–74; Sec. 6.2, Table 3, ll. 469–478; App. L.8
Q2. Isn't it obvious that selection changes when the weights change?That gradients change is standard (Remark 1); whether and how much the selection changes is notSec. 1, ll. 71–73; Secs. 5.2, 5.4, 5.5
W2 / Q3. How do these methods compare with true influence functions?Exact influence is invariant, with two caveats; now tested: 0% change vs. up to 15% on average for damped influence and gradient similaritySec. 7, ll. 486–496; App. L.9, L.11
W3 / Q4. Simple datasets; does this affect large generative language models?Added CLE on the CIFAR-10 ConvNet and a decoder-only LM (Qwen2.5-0.5B); same patternSec. 4.2, Table 1; Sec. 6.1, Table 2; App. L.12, L.13

W1 / Q1. Different subsets of redundant data can form the same decision boundary; is the selection method flawed, or is it picking equally useful examples?

Short answer. Our evidence is consistent with the latter, and the revision says so directly; we do not claim that the selectors are flawed.

Details.

  • Invariant control vs. raw selection (Table 3): −0.08, +0.04, −0.11 and +0.00 points, every 95% interval containing zero; random selection is 0.4–3.2 points worse.
  • Across twelve equivalent representations of each MNIST checkpoint at CLE-like strength, the standard deviation of downstream accuracy is 0.14 and 0.12 points for CRAIG and GradMatch (0.03 and 0.01 after subtracting fit-seed variance), against 0.22 across random subsets (App. L.8).
  • Further comparisons (selector-returned weights, CLE, pretrained features, BAIT, MiniLM; App. H, K.3) have mixed directions with no consistent pattern.

So, as you suggest, equivalent representations pick different examples without a consistent loss of accuracy. The paper's claim is narrower: which examples are picked is not determined by the predictor. This matters when the identities themselves are used, e.g. to order annotations, to audit a selection, or to reproduce a query from a stored checkpoint; we did not measure those costs (Limitations, ll. 518–519).

Changes in the paper. The introduction (ll. 73–74) now says that "different subsets of redundant data can be equally useful". Sec. 6.2 concludes (ll. 476–478):

Turnover therefore does not imply accuracy loss. These descriptive comparisons establish neither equivalence nor non-inferiority, and the case for invariant selection rests on reproducibility of identities, not on accuracy.

Q2. Isn't it already obvious that data selection changes when the model's weights change?

Short answer. That the gradients change is standard, and we label it as background (Remark 1). Whether the selection changes, and by how much, does not follow from it.

Details. The reparameterizations preserve the predictor analytically; in the vision rescaling audits, the maximum logit difference is 1.3×10−131.3\times10^{-13} (Figure 2). Three findings are not implied by the gradient change:

  • Changed gradients often leave the selection unchanged. Orthogonal LoRA basis changes give 0% raw replacement, and conformal changes preserve BADGE, CRAIG and unregularized GradMatch (Sec. 5.2).
  • In BAIT the Fisher terms are already covariant; all of the dependence comes from the fixed prior (Theorem 1).
  • How much a stochastic selection changes depends strongly on the sampler. Under cross-layer equalization, the standard inverse-CDF sampler gives 44–58% replacement and a keyed sampler of the same law 1–4% (Table 1, Proposition 3). This corrects our own earlier presentation (general response, Section 3).

Changes in the paper. The introduction calls the gradient change "a familiar fact in invariant optimization" and then separates the steps (ll. 71–73):

This does not by itself change the selected examples: the altered distances or scores must also cross a decision boundary, a point at which the selection outcome switches. For a stochastic selector, the outcome also depends on how random numbers are mapped to candidates.

AuthorsOpenReview ↗

Response to Reviewer NXSC (2/2)

W2 / Q3. Gradient methods are approximations; how do they compare with true influence functions that use the inverse Hessian at the optimum?

Short answer. Exact influence is on the invariant side of our characterization, with two caveats, and we have now tested it directly: exact-influence selections do not change, while damped influence and gradient similarity do.

Details. For a loss invariant under a linear reparameterization with Jacobian JJ, H′=J−⊤HJ−1H' = J^{-\top} H J^{-1}, so exact influence ziH−1zj⊤z_i H^{-1} z_j^\top is invariant, whereas raw gradient selectors use the identity in place of H−1H^{-1}. The two caveats: at a minimum with a nonzero symmetry direction, HH is singular; and a damped inverse (H+λI)−1(H+\lambda I)^{-1} is invariant for all gradient sets only if JJ is orthogonal.

The experiment (App. L.11) uses the classifier layer of the five MNIST and five Fashion-MNIST acquisition models (650 parameters; exact float64 Hessian at the trained weights). We selected the 50 candidates with the largest score g P zi⊤g\,P\,z_i^\top, where gg is the held-out validation-loss gradient and PP is H+H^{+}, (H+λI)−1(H+\lambda I)^{-1} or II, before and after cross-layer equalization and unit rescalings. Replacement (%):

ScoreMNIST, CLEMNIST, σ=0.5Fashion, CLEFashion, σ=0.5
Exact influence, P=H+P=H^{+}0000
Damped, P=(H+λI)−1P=(H+\lambda I)^{-1}, three λ\lambda2.4–3.29.6–13.22.8–4.010.7–14.6
Gradient similarity, P=IP=I0.46.32.410.2

Exact-influence selections are unchanged (exact score invariance also needs gradients in the range of HH). The Hessian is singular in every model (numerical nullity 101–173 of 650, including the 65 softmax-shift directions), hence the pseudo-inverse. On the original models, exact influence and gradient similarity share 18–42% of their selections.

Changes in the paper. New paragraph "Exact influence versus gradient similarity" in Sec. 7 (ll. 486–496; proofs in App. L.9; experiment in App. L.11, Table 31). It ends:

On our MNIST and Fashion-MNIST classifier layers, exact (pseudo-inverse) influence selects the same examples under CLE and rescalings, while damped influence and gradient similarity change up to 15% on average (Appendix L.11).

W3 / Q4. Many results use simple datasets such as MNIST and Fashion-MNIST; does the problem affect large generative language models?

Short answer. The revision adds cross-layer equalization on the CIFAR-10 ConvNet and a decoder-only generative language model (Qwen2.5-0.5B with LoRA); both show the same pattern. We do not extrapolate to much larger models.

Details. Beyond the vision benchmarks, the paper fine-tunes LoRA end to end in MiniLM (three ranks) and ModernBERT-base on SNLI (23 trained instances), now with a standard rebalancing operation and float32 write-back. The new decoder has rank-8 LoRA on every attention query and value projection, with SNLI posed as next-token prediction of label words (five instances). Under LoRA rebalancing (Table 2, % replaced):

ModelBADGE, inverse-CDFBADGE, keyedDeterministicBalanced / merged
MiniLM, r = 4 / 8 / 1643.9 / 31.4 / 37.04.2 / 3.2 / 4.213.2 / 9.2 / 10.00
ModernBERT-base, r = 856.81.83.80
Qwen2.5-0.5B (decoder), r = 850.27.915.20

In the decoder, as in the encoders, orthogonal basis changes leave raw selection unchanged. On the CIFAR-10 ConvNet, cross-layer equalization replaces 57.8% (inverse-CDF), 2.6% (keyed), 12.0% (CRAIG) and 23.6% (GradMatch), and 0% with QuotientGrad. The LoRA factor symmetry and the bound in Proposition 3 do not depend on model or pool size.

Changes in the paper. Sec. 6.1 (ll. 417–422) and Table 2; App. L.12; CIFAR-10 column of Table 1 and App. L.13; scope in Limitations (ll. 515–517). Sec. 6.1 ends (ll. 455–456):

The decoder shows the same pattern in a small generative model; we do not extrapolate to larger ones.

If these answers address your concerns, we would be grateful if you would consider raising your score. If any concern remains that affects your rating, please let us know which, and we will address it during the discussion period.

I thank the authors for providing this information. This answers my questions. I will revise my scores accordingly.

Thank you for reading our responses and for letting us know. We are glad they answer your questions.

Reviewer_ZAuuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper examines whether gradient-based data selection methods produce consistent selections under changes to model parameters that leave predictions unchanged. It studies two such transformations: rescaling hidden ReLU units and changing the internal basis of LoRA factors as both alter per-example gradients and can distort the Euclidean geometry used for selection. The authors show that distances between gradients remain unchanged under orthogonal transformations or when the distance metric is adjusted to account for the change in parameterisation. They distinguish geometric distortion from changes in discrete selections, construct examples where the first selected example changes, and trace BAIT’s sensitivity to its fixed regulariser.

Empirically, these transformations replace between 19 and 63% of a 50-example query in small vision networks and between 30 and 70% under LoRA basis changes in MiniLM and ModernBERT. The tested coordinate-invariant alternatives retain identical selections. Across five active-learning rounds, the acquired sets diverge substantially, while final accuracy changes little. The paper therefore distinguishes consistency in which examples are selected from the usefulness of those selections.

Critical strengths
  1. The question is new and well posed. While prior work focuses on whether a selected subset is useful, this paper asks whether equivalent parameterisations of the same model produce the same selection. By making the relevant symmetry group explicit and holding randomness fixed, it provides a clear, reusable framework for assessing whether selection is well defined. The analysis carefully distinguishes whether a transformation changes gradient geometry, whether that distortion appears among the observed gradients, and whether it alters a discrete decision. It also offers the non-obvious insight that BAIT’s sensitivity arises from its fixed regularizer rather than its Fisher terms.
  2. The methodology is well-executed and provides interesting insights Selector randomness is coupled, model equivalence is verified to 1e-13, and invariant controls return 0% swap, pinning the cause on coordinates rather than the predictor. The design also yields findings beyond the headline: elementwise weight-times-gradient fixes survive axis rescaling but not dense LoRA re-basing, large query turnover leaves accuracy unchanged, and reported negative results show the effect is no larger than the selector’s own noise.
Critical weaknesses

The effect is not shown to arise in real pipelines First, the authors deliberately introduce every reparameterisation. They do not examine a practical workflow in which an equivalent parameterisation arises naturally, such as adapter merging or refactorisation, or weight rescaling for quantisation. Second, the measured downstream effects are small. Final accuracy changes by at most 0.27 percentage points after five acquisition rounds. In the reported BADGE comparison, changing the random draws while keeping the weights fixed alters the selected set more than reparameterising the weights.

Questions for authors
  1. Can the authors identify a setting where two equivalent parameterisations of the same trained model actually occur in a pipeline and report query replacement, rather than under synthetic rewrites?
  2. Given that downstream accuracy changes little and reparameterisation alters selections less than BADGE’s own run-to-run variability, what concrete guidance does the paper offer practitioners? Should gradient-based selectors use a Fisher-based metric by default? If so, why are Fisher-based corrections evaluated only for BAIT, without corresponding experiments for BADGE, CRAIG and GradMatch?
7 comments
AuthorsOpenReview ↗

Response to Reviewer ZAuu (1/2)

Thank you for your review. You found the question "new and well posed", described the explicit symmetry group and fixed randomness as "a clear, reusable framework for assessing whether selection is well defined", and pointed to the "non-obvious insight" that BAIT's sensitivity comes from its fixed regularizer. Your observation that a new seed changes a BADGE query more than reparameterization does (W3) led to the main change in the revision (general response, Section 3).

Your commentShort answerWhere in the revised PDF
W1 / Q1. Every reparameterization is introduced by the authors; does one arise in a real pipeline?Yes: cross-layer equalization (before quantization) and LoRA rebalancing (balanced LoRA training), including float32 write-backSec. 4.2, Table 1; Sec. 6.1, Table 2; App. K.1, L.4, L.5, L.12, L.13
W2 / Q2a. Downstream effects are small; what should practitioners do?Agreed, now stated as a result; guidance depends on whether identities must be reproducibleSec. 6.2, Table 3; Sec. 8, ll. 500–511
W3. A new seed changes BADGE more than reparameterizationCorrect; the same-seed number depends on the sampler, so we now report two samplers of the same lawSecs. 3.3, 4.3, 5.5; Tables 1–2
Q2b. Fisher metric by default? Why only BAIT?No default recommendation; BAIT needs only a covariant priorSec. 5.4; Sec. 8, ll. 509–511

W1 / Q1. Every reparameterization is introduced by the authors; can one be shown in a setting where equivalent parameterizations actually occur?

Short answer. Yes. We now test two operations that practitioners already apply to trained models and whose directions we do not choose.

Details.

  • Cross-layer equalization (Nagel et al., 2019), the weight-rescaling step applied before quantization. Equalized and original models agree to 1.1e-13 in logits, with no prediction changes.
  • LoRA rebalancing, the product-preserving projection that balanced LoRA training applies after every optimizer step (Castin et al., 2026). On five ModernBERT checkpoints their algorithm and our balanced factorization give the same Gram matrix to 2.5e-16 and identical ordered selections in 50 of 50 runs. We also write the rebalanced factors into float32 checkpoints and recompute all gradients (physical-writeback row of Table 2).

Replacement (% of a 50-example query) across MLPs, a ConvNet, two encoders and a decoder-only language model:

OperationBADGE, inverse-CDFBADGE, keyedDeterministicInvariant
Cross-layer equalization44–581–42–240
LoRA rebalancing31–582–84–150

The two samplers are explained under W3.

Changes in the paper. Sec. 4.2 (ll. 262–271) and Table 1 (App. K.1, L.13); Sec. 6.1 (ll. 404–456) and Table 2 (App. L.4, L.5, L.12). Sec. 4.2 now opens:

We apply the scaling step of cross-layer equalization (Nagel et al., 2019), which sets per-channel scales from weight ranges and iterates over consecutive layers to convergence (Appendices K.1 and L.13); no directions are chosen by us.

W3. In the reported BADGE comparison, changing the random draws alters the selected set more than reparameterizing the weights.

Short answer. Correct, and this comparison led to the main change in the revision: the same-seed BADGE number depends on how the seed is mapped to candidates, so we now report two samplers of the same law side by side.

Details. Table 1, cross-layer equalization (% replaced):

ComparisonMNISTFashionCIFAR-10
Same seed, inverse-CDF43.7654.8857.76
Same seed, keyed1.364.162.64
Different seeds, inverse-CDF64.5673.9277.96
Different seeds, keyed65.6073.7276.52

With inverse-CDF sampling, a small change in the D² probabilities can shift many cumulative boundaries, so replacement is already 17% at the smallest rescaling tested (Figure 1c) (η≈0.01\eta\approx0.01) and 44–53% for η≥0.1\eta\ge0.1. With a keyed sampler, a step changes with probability at most 2v/(1+v)2v/(1+v) given a common prefix, vv being the total-variation change of the D² law (Proposition 3; the bound is due to Daliri et al., 2024), and replacement grows gradually with the distortion (0.6% to 10% over the same range). The different-seed baseline is similar under both samplers. We no longer present the inverse-CDF number as the size of a geometric effect.

Changes in the paper. Samplers defined in Sec. 3.3 (ll. 195–201); distortion analysis in Sec. 4.3 (ll. 275–285) and Figure 1c; Proposition 3 in Sec. 5.5 (ll. 363–400; proof in App. L.2). Sec. 4.2 now states (ll. 270–271):

Independent BADGE runs on the unchanged model differ by 65–78% under either sampler, so keyed sampling does not make the selector less random; it reduces same-seed replacement.

AuthorsOpenReview ↗

Response to Reviewer ZAuu (2/2)

W2 / Q2a. The measured downstream effects are small; what concrete guidance does the paper offer practitioners?

Short answer. Agreed; Sec. 6.2 now states this as a result, and Sec. 8 gives guidance that depends on whether the selected identities themselves matter.

Details. In Table 3, QuotientGrad is within 0.12 points of raw selection in every row, while random selection is 0.4–3.2 points worse; further comparisons in the appendix have mixed directions with no consistent pattern. The case for invariant selection therefore rests on reproducibility of identities, not on accuracy. The guidance in Sec. 8:

  • If accuracy is the only goal, our results do not justify a blanket switch to invariant selection; utility differences depend on the task and training protocol.
  • If equivalent stored forms of a model must return the same examples (annotation orders, audits, reproducing a query from a checkpoint): (i) key the D² sampler by candidate ID; (ii) use a representation that transports the metric (QuotientGrad for unit rescalings, balanced or merged coordinates for LoRA, a covariant prior for BAIT); (iii) compare returned coefficients as well as IDs.

These choices reproduced identities in the tested settings. Keyed sampling has the same asymptotic per-step cost as inverse-CDF sampling. Step (i) substantially reduces same-seed replacement in a single selection but does not guarantee an identical query; step (ii) kept the tested five-round trajectories identical (200/200 runs; Sec. 6.2).

Changes in the paper. Sec. 6.2, Table 3 and ll. 469–478; Sec. 8, ll. 500–511, which now opens:

For predictive performance alone, these results do not justify a blanket switch to invariant selection; utility differences depend on the task and training protocol.

Q2b. Should gradient-based selectors use a Fisher-based metric by default? If so, why are Fisher-based corrections evaluated only for BAIT?

Short answer. No, we do not recommend a Fisher metric by default, and Sec. 8 now says so explicitly.

Details. BAIT already contains Fisher terms, and its dependence on the parameterization comes only from the fixed identity prior (Theorem 1), so the correction there is a covariant prior; this is why BAIT is the only selector with a Fisher-based correction. For BADGE, CRAIG and GradMatch the evaluated control is multiplicative (QuotientGrad); in our MNIST timings, selection took similar time with and without it (App. K.4). We did not benchmark full-Fisher versions of these three selectors. The closest new evidence is the influence experiment requested by Reviewer NXSC: scores that use the exact (pseudo-inverse) Hessian select the same examples under CLE and rescalings, while damped influence and gradient similarity change up to 15% on average (Sec. 7; App. L.11). A fixed damping term breaks invariance by the same mechanism as BAIT's fixed prior.

Changes in the paper. Sec. 8, ll. 509–511:

We do not recommend a full-Fisher metric by default: BAIT already contains Fisher terms and needs only a covariant prior, and for BADGE, CRAIG, and GradMatch the evaluated control is multiplicative; full-Fisher versions were not benchmarked.

If these changes address your concerns, we would be grateful if you would consider raising your score.

Reviewer_ZAuuOpenReview ↗

Answer to new results and framing

I thank the authors for answering my questions and for their rebuttal.

Overall my opinion is that their rebuttal has improved the precision of their contributions but reduced the size of the claimed effect, whilst the real-world applicability remains unproven.

Firstly, the authors have pinpointed the inverse-CDF sampler as the reason for the high query turnover result they originally obtained for BADGE. Under a different sampling technique (keyed-race sampling, i.e. linking sample selection to individual datapoint IDs rather than a point on the cumulative distribution over the datapoints), the shift in candidate selection by BADGE is greatly reduced. The new abstract and introduction now state the effect of the inverse-CDF sampler more directly. This is an interesting result, since the authors state that inverse-CDF is the rule used in the official BADGE implementation. In some way this is qualitatively akin to the BAIT result, which showed that a numerical stability component was responsible for the difference in data selection rather than the main information-theoretic component (Fisher term).

Secondly, the authors test two real world versions of their weight-rewriting pipelines. The first is cross-layer equalization, where ReLU weights are rescaled before quantization, the second is LoRA balancing, where the LoRA factors are rebalanced to be the same size, without changing their product. Overall, in my opinion the paper would benefit from a brief description of how these real-world pipelines are used in practice to better contextualize the experiments. An interesting related result is that computing gradients from balanced LoRA leads to consistent picks, regardless of how the LoRA factors were originally written. If I understand the implication correctly, this would mean that keeping LoRA balanced during fine-tuning would make data selection independent from the LoRA parametrization. Could the authors comment on whether my reading is correct and whether this would have any further implications for active and continual learning pipelines ?

Finally, following the new results detailed above, the authors have softened the claim of large differences in gradient-based data selection and reframed the paper around reproducibility. After showing the conditions under which the selected sets are non-identifiable, even when the model's predictions are kept fixed, the authors propose fixes to address these and present reproductibility as the practical reason for adopting them. They cite annotation orders, audits, or queries reproduced from a checkpoint as the practical applications. In my opinion the new framing seems reasonably-suited, however its practical relevance remains unproven. The new "A recipe for representation-reproducible selection" paragraph of section 8 offers useful guidance to practitioners (in response to my earlier review), and the fixes are validated in the authors’ experiments. However they are not specifically applied to the aforementioned reproducibility-related use cases, for which the authors have not measured the real cost and benefit of their method.

As such I find that my original score (3) is appropriate for this submission. It is an interesting paper with some interesting results, the practical relevance of which remains largely unproven for the use cases identified by the authors. To raise my score, I would want to see the authors’ fixes applied the use-cases they identified, clearly demonstrating the cost of non-reproducible selection and the benefit of their proposed solutions.

Reviewer_d8jEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper identifies the fact that in gradient-based data selection methods that assign scores to training data samples regarding their role in the current model decisions, the selected important samples should be invariant to any model function preserving weight changes. It finds that up to 65% selected samples can change in response to weight-scaling operations that leave logits invariant in current approaches like BADGE, CRAIG or GradMatch. This is also demonstrated in LoRA fine-tuning, where basis changes can yield similar differences in the selected samples while preserving model function. The paper then theoretically characterize cases where the gradient-inner product structure remains invariant and cases where it changes for current methods, yielding different selected sets. In both the standard and LoRA fine-tuning cases the paper provides variants which are not sensitive to such function preserving weight changes, and demonstrate the same. Lastly, the authors note that while changes in parameterization can significantly affect the chosen set, the quality of the set, as measured by the re-training test accuracy downstream, does not show any significant change for all methods.

Critical strengths
  • The core principle enforced here of selected set invariance to function-preserving weight changes is valid, as it should be a function of the final model function and not of its weight parameterization.
  • The observation regarding the current methods' selection quality not changing that much to such function preserving weight changes is relevant.
Critical weaknesses

There are many issues here, mainly on the clarity/presentation aspects but also the empirical aspects. Some of them are:

  • The writing is really poor, choice of words for well known terms seems to flip-flop through the writing as well, making reading the paper a significant chore to read.

  • The introduction jumps into the terminology without any proper build up and background, and definition, examples are all over but just in the first paragraph authors just dedicate one sentence to describe the problem, and that too, quite underwhelmingly. Subsequently, writing jumps into terms like "subset utility", "candidate pool", "predictor" without actually defining any of these. These are non-conventional terms (at least to me), and while a reader understands after the fact what these may be, it needs dedicated time for "inferring" which makes the paper slow and hard to read. Especially as the underlying concepts here are quite standard and straightforward

  • Some other examples in the introduction alone is: LoRA is introduced without reference, the background dicussion from 44-48 is very roughly summarized, and undefined, non-conventional use of terms continue: "gradient coordinate dependence", "implementation invariance", "geometric distortion", "selector decision boundary", "selected identities". This trend actually continues throughout this paper.

  • Unfortunately, these trends continue through the math as well, just a few examples: it is not clear what ξ would represent, and z_i(theta) is suddenly introduced without any proper definition and bad terminology. Subsequently, the methods are just roughly summarized within the gram geometry framework (eq (2)) in a single sentence without proper exposition. There needs to be more details to do due diliegence to literature. The related work section later on is very dense and also quite hard to read/connect in the context of this work.

  • It is clear that, much if not most of this work heavily uses AI-writing, and that is okay if it does not hurt readability and presents the paper in a coherent and clear manner, in this case it is quite the opposite. The paper will need a thorough rewrite, at least.

  • While sec 4 presents some relevant findings, many details are missing, and the writing continues to be poor. "ten hidden-unit rescalings": what exactly does this mean? the output of the units are rescaled? and 10 means randomly chosen? "candidate pool of 1000" is this candidate pool randomly chosen? I assume only 1k samples are chosen due to compute constraints for the gradient norms? While I understand what candidate pool is referring to, is it a subset of the training samples after training the classifier on the whole datasets, or is the model trained on the subset itself? "stochastic side information is coupled" again, it would be helpful to point out exactly what types of side information is being frozen or point to it int he Appendix if it is contained there.

  • Sec 5: "selector-consumed parameters". Also why do the authors keep referring to z_i as the "covector" it is the vector of gradients for the ith samples. Also Lemma 1 is not setup correctly as it is not self-contained. One needs to read the intro to sec 5 to make sense of both Lemma 1 and Theorem 1. It would be better to first define the setting and the problem formally and then at least refer to it in the theoretical results.

  • Also, lemma 1 and theorem 1 appear to be quite obvious, I would just put them in a remark.

  • In 5.3, (9) is completely abrupt. Not sure what t is, what delta is, what even is being discussed here. It is very abrupt.

  • In proposition 1, "ReLU-realizable" is weird, again non-standard terminology that's hard to interpret first hand.

  • In 5.4 a bunch of notation is directly introduced in some equations without any definition. Not sure what H_S, F_L, I_x, J_R all are. It thus becomes unncessarily tough to read and interpret theorem 2.

  • There are no proper experiment sections, some of them are introduced within theory sections without any proper setup or detail.

  • In the LoRA experiments, the authors haven't properly explained what their "balanced and merged" approach for ensuring invariance is.

  • One thing the authors point out is that large changes in the selected sets does not imply changes in accuracy for all methods. Meaning that even if the function-preserving parameter change changes the selected set significantly in some cases, the accuracy of re-training does not change significantly at all, implying the quality of the new selection is just as good as the one before. This is fine, but there seems to be a glaring omission in all this: I could not find anywhere where the actual quality of the selected set is measured by comparing the accuracy of their fixes against the baselines. So even though the selected set remains invariant, no quality analysis of the actual selected set is provided, at least in the main paper.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer d8jE (1/3)

Thank you for the detailed review. You considered the core principle, that a selected set should be a function of the model function rather than of its weight parameterization, valid, and the observation that selection quality changes little under function-preserving weight changes relevant. Your presentation comments led to a rewrite of the main text, and your point that the fixes were never compared in quality with the baselines led to Table 3, now in the main text. We number your points W1–W13 in the order of your review and give submission numbering where it changed.

Your commentShort answerWhere in the revised PDF
W1. Hard to follow; terms change; needs a rewriteAgreed; rewritten, one term per concept, defined at first use; new overview figureSecs. 1–8; Figure 1
W2. "Subset utility", "candidate pool", "predictor" undefinedDefined in the first paragraph, which now states the questionll. 34–44
W3. LoRA without reference; rough background; non-standard termsLoRA cited; background rewritten; terms removed or definedll. 45–53, 70–84
W4. ξ\xi and ziz_i unclear; selectors in one sentence; dense related workSymbols defined; one description per selector; prior work placed in the introductionll. 124–161; ll. 76–84; App. M
W5. Sec. 4: rescalings, pool, training data, what is held fixedAll stated in the setup paragraphll. 213–215 and 246–251; App. B.1
W6. "Selector-consumed parameters"; "covector"; Lemma 1 not self-containedTerm replaced; covector explained; setting stated firstll. 142–145, 293–298
W7. Lemma 1 and Theorem 1 obviousNow Remark 1 and Proposition 1ll. 302–315
W8. Eq. (9) abruptNow Eq. (7), with tt and Δ(t)\Delta(t) defined firstll. 325–332
W9. "ReLU-realizable"Removed; now Proposition 2ll. 333–337
W10. HSH_S, FLF_L, IxI_x, JRJ_R undefinedDefined before the theorem (now Theorem 1)ll. 340–351
W11. No proper experiment sectionsExperiments in their own sectionsSecs. 3.3, 4, 6
W12. "Balanced and merged" unexplainedDefined with formulasll. 412–416; App. G
W13. No quality comparison of fixes vs. baselinesAdded to the main textSec. 6.2, Table 3; App. L.8

We start with W13, the substantive gap you identified.

W13. The quality of the sets selected by the fixes is never compared with the baselines.

Short answer. Agreed; this comparison is now Table 3 in the main text.

Details. Accuracy of the invariant control (QuotientGrad) and of random selection relative to raw selection, in percentage points with 95% intervals over five acquisition seeds and three paired fits each:

ProtocolSelectorRaw accuracy (%)QuotientGrad − rawRandom − raw
One query, MNISTCRAIG88.015−0.079 [−0.226, +0.062]−0.527 [−0.845, −0.262]
One query, MNISTGradMatch87.860+0.041 [−0.286, +0.386]−0.372 [−0.692, −0.069]
Five rounds, MNISTBADGE94.735−0.111 [−0.356, +0.143]−3.160 [−3.307, −3.013]
Five rounds, Fashion-MNISTBADGE83.275+0.001 [−0.248, +0.399]−1.807 [−1.935, −1.666]

QuotientGrad is within 0.12 points of raw selection and all four intervals include zero; random selection is 0.4–3.2 points worse. Across twelve equivalent representations of each MNIST checkpoint at CLE-like strength, the standard deviation of accuracy is 0.14 and 0.12 points for CRAIG and GradMatch (0.03 and 0.01 after subtracting fit-seed variance), against 0.22 across random subsets (App. L.8). Further comparisons in App. H and K.3 have mixed directions with no consistent pattern.

Changes in the paper. Table 3 and Sec. 6.2 (ll. 433–478), which conclude:

Turnover therefore does not imply accuracy loss. These descriptive comparisons establish neither equivalence nor non-inferiority, and the case for invariant selection rests on reproducibility of identities, not on accuracy.

W1. The writing is hard to follow, wording for well-known terms changes, and the paper needs a thorough rewrite.

Short answer. Agreed; the main text was rewritten.

Details. One term per concept, every term defined at first use, standard terms where they exist, and setup before results. The introduction now follows one line: the problem, why it matters and is hard, what prior work does not cover, our approach, and the contributions; a new Figure 1 shows the problem at a glance. W2–W12 give the change for each passage you listed.

Changes in the paper. Secs. 1–3 rewritten, Sec. 5 reorganized, experiments in Secs. 4 and 6 (a large rewrite, so locations rather than quotes).

AuthorsOpenReview ↗

Response to Reviewer d8jE (2/3)

W2. The introduction states the problem in one sentence and uses "subset utility", "candidate pool" and "predictor" without definition.

Short answer. Agreed; the first paragraph now defines these terms and then states the question.

Details. After the definitions come the four selectors, a one-line gloss of Fisher information (l. 40), and the question (ll. 42–44): with the predictor and candidates fixed, does the selected set depend only on the predictor, or also on how its parameters are written down?

Changes in the paper. ll. 34–37:

Data selection chooses a small set of examples to label or train on; in active learning this set is called the query. A predictor is the model's input-to-output function; the candidate pool is the finite set offered for selection; and subset utility is the predictive performance obtained by a specified training procedure using the chosen data.

W3. LoRA is introduced without reference, the background is rough, and several terms are non-standard.

Short answer. Agreed; LoRA is now cited and written out, and each term you listed is removed or defined.

Details. LoRA is introduced with Hu et al. (2022) and the product BABA of two thin factors (ll. 48–51). "Gradient coordinate dependence", "implementation invariance" and the acronym PEFT no longer appear in the main text; "distortion" is a defined quantity η\eta (Sec. 4.3); "decision boundary" and "selected identities" are defined where they first appear.

Changes in the paper. "Decision boundary" is defined at ll. 71–72, "selected identities" at ll. 51–53:

If the identities of the selected examples (selected identities) change under these operations, then an annotation order, an audit, or a query reproduced from a checkpoint depends on something other than the predictor.

W4. ξ\xi and zi(θ)z_i(\theta) are not properly defined, the selectors are summarized through Eq. (2) in one sentence, and related work is dense.

Short answer. Each symbol is now defined where it is introduced, each selector has its own description, and related work is shortened.

Details.

  • zi(θ)=∇θBℓi(θ)⊤∈R1×pz_i(\theta)=\nabla_{\theta_B}\ell_i(\theta)^\top\in\mathbb{R}^{1\times p} is the ordinary per-example gradient on the parameter block θB\theta_B used by the selector, written as a row (ll. 142–145).
  • In Eq. (2), ΦD\Phi_D is the selection algorithm and Kij=ziMzj⊤K_{ij}=z_iMz_j^\top compares the gradients of examples ii and jj (ll. 150–152). BADGE, CRAIG and GradMatch each get their own description (ll. 152–161); App. Table 4 lists the geometry and control for each.
  • The introduction now says how prior work differs from ours (ll. 76–84); Sec. 7 adds LoRA geometry, the coupling result and influence functions; the longer survey is in App. M.

Changes in the paper. ll. 125–127:

ξ\xi records the random numbers, the rule mapping them to candidates, the label convention (predicted or true labels), the per-class budget allocation, candidate order, and tie rules. With these fixed, selection is a deterministic map.

W5. Sec. 4 lacks details: the "ten hidden-unit rescalings", whether the pool of 1000 is random and chosen for compute, whether the model is trained on the full data, and what "side information" is held fixed.

Short answer. Each is now stated in the setup paragraph of Sec. 4.1 (ll. 213–215 and 246–251), with further detail in App. B.1.

Details.

  • Rescalings. Each transformation multiplies the incoming weights and bias of every one of the 64 last-hidden units by sjs_j and divides the unit's outgoing weights by sjs_j, with log⁡sj=clip(0.5 uj,±log⁡10)\log s_j=\mathrm{clip}(0.5\,u_j,\pm\log 10), uj∼N(0,1)u_j\sim N(0,1). Outputs are unchanged (maximum logit difference 1.3×10−131.3\times10^{-13}). "Ten" means ten independent draws of the scales per model.
  • Training data. Each model is trained only on 500 stratified initial labels (1000 for CIFAR-10) from a fixed 10,000-example subset of the training set, not on the full dataset; this is the active-learning setting.
  • Pool. Yes, random: 1000 examples sampled uniformly without replacement from the remaining examples of that subset; 50 are queried. Yes, the size was set by computation, though the main cost is not the gradient norms: CRAIG computes within-class pairwise distances, GradMatch refits an active set, and every audit is repeated in float64 for ten transformations. In the five-round protocol (Sec. 6.2, App. L.10), selection uses the full remaining pool of 8,500–9,500 candidates and shows the same pattern: inverse-CDF sampling replaces 76–83% of the first batch, keyed sampling 3–16%, QuotientGrad none. Proposition 3's one-step keyed bound does not depend on pool size.
  • Held fixed. Everything in ξ\xi (ll. 125–127), listed again in Sec. 3.3 (ll. 193–194): candidate order, pseudo-label convention, class allocation, tie rules and random numbers.

Changes in the paper. Sec. 4.1, ll. 213–215 and 246–251; App. B.1 (seeds, pool construction, reason for the pool size).

AuthorsOpenReview ↗

Response to Reviewer d8jE (3/3)

W6. "Selector-consumed parameters" is unclear; ziz_i is just the gradient, so why "covector"; Lemma 1 is not self-contained.

Short answer. Agreed: the term is replaced, ziz_i is described as the ordinary gradient, and the setting precedes all results.

Details. "Covector" only names how ziz_i changes under a change of coordinates; the text says it is "not a different derivative" (ll. 144–145). The map φ\varphi, its Jacobian JJ, the preserved predictor and the differentiability conditions precede Remark 1 (ll. 293–298). The Jacobian is now JJ, so it no longer clashes with the LoRA factor AA.

Changes in the paper. ll. 293–296, Eq. (4):

Let φ:U→U′\varphi: U\to U' be an invertible differentiable reparameterization of the parameter block θB\theta_B used by the selector, with Jacobian J=Dφ(θ)J=D\varphi(\theta), such that fφ(ϑ)=fϑ ∀ϑ∈Uf_{\varphi(\vartheta)}=f_\vartheta\ \forall\vartheta\in U, (4) with loss-relevant state transported.

W7. Lemma 1 and Theorem 1 appear obvious and could be remarks.

Short answer. Agreed: Lemma 1 is now Remark 1, marked as standard background, and Theorem 1 is now Proposition 1, marked as elementary.

Details. Proposition 1 stays numbered because the invariant controls rest on its necessity direction.

Changes in the paper. Remark 1 ends "This is standard background." (l. 307). After Proposition 1 (l. 315):

The statement is elementary; its role is to fix what an invariant control must do.

W8. Eq. (9) in Sec. 5.3 is abrupt; tt and Δ\Delta are not defined.

Short answer. Both are now defined before the equation, which is Eq. (7) in the revision.

Details. tt is the log-scale of one ReLU unit; Δ(t)\Delta(t) is the signed value of a fixed affine comparison between two candidates (for example, which of two squared distances is larger).

Changes in the paper. ll. 325–332:

Let t∈Rt\in\mathbb{R} be the log-scale of one ReLU unit: its incoming parameters scale by ete^{t} and outgoing weights by e−te^{-t}. [...] Its signed value along the orbit is Δ(t)=ae−2t+be2t+c\Delta(t)=ae^{-2t}+be^{2t}+c, so a sign change reverses that comparison.

W9. "ReLU-realizable" in Proposition 1 is non-standard.

Short answer. Removed; the statement, now Proposition 2, says directly what is constructed.

Details. These are first-choice existence results, not turnover guarantees (l. 337); constructions are in App. A.8.

Changes in the paper. ll. 333–335:

There exist ReLU classifiers and function-preserving hidden-unit rescalings that change the max-norm first center of BADGE, the first Euclidean facility-location choice of CRAIG, and the first positive-OMP atom in a GradMatch block.

W10. In Sec. 5.4, HSH_S, FLF_L, IxI_x and JRJ_R are undefined, which makes Theorem 2 hard to read.

Short answer. All are now defined before the theorem, which is Theorem 1 in the revision.

Details. sx(y)=∇θBlog⁡pθ(y∣x)s_x(y)=\nabla_{\theta_B}\log p_\theta(y\mid x) is the score; IxI_x the point Fisher information; FLF_L, FUF_U the mean Fisher over labeled and target inputs; HS=FL+∑x∈SIxH_S=F_L+\sum_{x\in S}I_x; RR the prior; and JR(S;FU)=tr[(R+HS)−1FU]\mathcal{J}_R(S;F_U)=\mathrm{tr}[(R+H_S)^{-1}F_U] is the BAIT objective, Eq. (8).

Changes in the paper. ll. 340–347, then Theorem 1 at ll. 348–351.

W11. There are no proper experiment sections.

Short answer. Agreed; experiments now have their own sections, with setup before results.

Details. Protocol and samplers: Sec. 3.3. Vision: Sec. 4 (setup ll. 213–215 and 246–251). LoRA and utility: Sec. 6 (setup ll. 417–429). The BAIT figure moved to App. H.3; Sec. 5.4 keeps only its numbers (ll. 354–359).

Changes in the paper. Secs. 3–6 reorganized as above.

W12. The "balanced and merged" approach for LoRA is not explained.

Short answer. Both are now defined with formulas (ll. 412–416).

Details. Write ΔW=BA=UΣV⊤\Delta W=BA=U\Sigma V^\top. Balanced uses B=UΣ1/2B=U\Sigma^{1/2}, A=Σ1/2V⊤A=\Sigma^{1/2}V^\top, a fixed representative of all factor pairs with the same product, onto which balanced LoRA training projects (App. L.5). Merged uses the gradient with respect to ΔW\Delta W itself, which does not depend on the factorization (sketched to 8×8 for Transformers; App. G).

Changes in the paper. Sec. 6.1, ll. 412–416.

Headline numbers. The revision also changes how the submission's headline numbers should be read (general response, Section 3): they are reproduced to their reported precision, but for BADGE and the encoder experiments their size depends on the reference sampling rule. Under the two standard operations, keyed sampling gives 1–8% and deterministic selectors 2–24%; the CRAIG and GradMatch values of Figure 2 (19–52%) involve no sampling and reflect rescalings whose log-scale spread is about five times that of cross-layer equalization.

If the revision addresses your concerns, we would be grateful if you would consider updating your score. If anything still affects your rating, please tell us which points; we will address them in the discussion.

AuthorsOpenReview ↗

Follow-up: summary of the revision

Dear Reviewer d8jE,

As the author–reviewer discussion closes on November 18 (AoE), we would like to make sure the revision reaches you. Your review described its issues as "mainly on the clarity/presentation aspects but also the empirical aspects". We have addressed all thirteen points (W1–W13 in our replies):

  • Empirical points: the full setup of Sec. 4 is now stated in one paragraph (ll. 213–215 and 246–251; W5), and the quality comparison of the fixes against the baselines that you asked for is now Table 3 in the main text (W13).
  • Introduction: rewritten in four steps (the problem, why it matters and is hard, what prior work does not cover, our approach), followed by bolded contributions; every term is defined at first use, and a new Figure 1 shows the problem at a glance.
  • Section 5: states the setting before any result; Lemma 1 and Theorem 1 are now Remark 1 and an elementary Proposition 1, and all notation in Secs. 5.3–5.4 is defined before use.
  • Experiments: each has its own section (Secs. 3.3, 4 and 6), with the setup before the results.

Our replies (1/3)–(3/3) give the location of every change. We hope the revision resolves your concerns; if so, we would appreciate an updated assessment.

Reviewer_LSgKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether gradient-based data selection is a property of the predictor or its parameter representation. It formalizes selection invariance under function-preserving transformations and shows that equivalent predictors can select substantially different examples, even with candidate data and randomness fixed.

Critical strengths

Controlled empirical evidence: ReLU rescalings produce 18.8–63% query replacement across BADGE, CRAIG, and GradMatch on MNIST, Fashion-MNIST, and CIFAR-10, while the maximum logit discrepancy is only 1.26 × 10⁻¹³.

  • Mechanistic explanation: The paper characterizes conditions for preserving gradient geometry, constructs explicit selection reversals, and identifies BAIT’s fixed identity prior as another source of dependence. Invariant/canonical controls restore identical selections in the evaluated settings.
  • Transformer extension: LoRA basis changes demonstrate the phenomenon in MiniLM and ModernBERT on SNLI, including 70.3% average query replacement in the analytical ModernBERT comparison.
Critical weaknesses

Generality beyond classifiers and encoders remains untested. TIVE-style visual instruction selection is a concrete extension: LoRA gradients determine both example scores and task allocations. Testing decoder MLLMs and image-generation pipelines would establish broader relevance. https://arxiv.org/html/2403.09559v4

Practical impact needs more evidence. After five acquisition rounds, cumulative selected sets differ by 47.9%/63.2%, but terminal accuracy changes by only +0.01/+0.27 percentage points on MNIST/Fashion-MNIST. A compelling extension would evaluate the stability of training examples selected from large generative-model outputs—such as multimodal mixture-of-experts LLMs or coupled image-generation systems—under repeated stochastic sampling. This should distinguish generation randomness from parameterization dependence and measure whether consistency improves data coverage, selection reliability, or training efficiency.
Questions for authors

As mentioned in review

3 comments
AuthorsOpenReview ↗

Response to Reviewer LSgK (1/3): overview and generality (W1)

Thank you for the review, and for highlighting the controlled evidence, the mechanistic explanation (including BAIT's fixed prior) and the Transformer extension. Your review refers to the submitted version. The revised PDF, posted during the discussion, already answers part of both points, and for this discussion we ran four new experiments that address them directly (App. L.16–L.17, Tables 37–39): TIVE's released selection rules on a decoder LM, a vision-language model and a mixture-of-experts LM, and gradient-based selection for a diffusion model, in which generation randomness (the timestep and noise draws of the diffusion loss), selector randomness and the parameterization are varied one at a time. We number your two weaknesses W1 (generality) and W2 (practical impact). W1 is answered in this comment and the next, W2 in comment 3/3.

Your commentShort answerWhere in the revised PDF
W1. Generality beyond classifiers and encoders; TIVE-style visual-instruction selection; decoder MLLMs and image generationTested on four model families beyond classifiers and encoders. Raw factor gradients change the selection in all four; balanced and merged coordinates reproduce it exactly in all fourSec. 6.1, Table 2; App. L.12; App. L.16–L.17, Tables 37–39 (new)
W2. Practical impact; separate generation randomness from parameterization dependenceExport–reload audit with measured costs and benefits; the three sources of variation measured separately in the diffusion modelApp. L.15, L.17 (new); Tables 3, 35–36, 39

On the headline numbers. The revision reports the headline under two standard operations, cross-layer equalization and LoRA rebalancing, and separates the sampling rule from the geometry (General response, Sec. 3). The large BADGE values of the submission depend on the inverse-CDF sampling rule of the official BADGE code: under the standard operations BADGE changes 31–58% with that rule and 1–8% with a candidate-keyed sampler of the same selection law, deterministic selectors change 2–24%, and the analytical invariant controls 0% (Tables 1–2).

W1. Generality beyond classifiers and encoders remains untested; TIVE-style visual instruction selection, decoder MLLMs and image-generation pipelines.

Short answer. We now test TIVE's released rules on a decoder LM, a vision-language model and a sparse mixture-of-experts LM, and gradient-based selection on a diffusion model (App. L.16–L.17). In every model, LoRA rebalancing preserves the model's function up to floating-point error but changes the selection computed from raw factor gradients, and balanced and merged coordinates reproduce every selection exactly.

Results under LoRA rebalancing (five checkpoints × five selector seeds per model; rebalancing applied analytically in float64 to the stored gradients; replacement with the released / keyed / deterministic sampler; "exact" = selection reproduced as a set and in order):

Model and selection ruleRaw factor gradientsBalanced and merged coordinates
Decoder LM (Qwen2.5-0.5B), TIVE rulesbudgets move 1.5–2.5 examples; 5.5 / 1.4 / 11.4% replaced; 0/25 exactbudgets unchanged; 0%; 25/25 exact
Vision-language model (SmolVLM-256M), TIVE rulesabout 19 of 150 budget slots move; 22.3 / 12.7 / 22.1%; 0/25budgets unchanged; 0%; 25/25
Mixture-of-experts LM (Granite-3.0-1B-A400M), TIVE rulesbudgets move 1.5 examples; 5.6 / 1.0 / 18.5%; 0/25budgets unchanged; 0%; 25/25
Diffusion model (DDPM CIFAR-10), gradient cosine to a target class45.9 / 37.4 / 38.0% (inverse-CDF / keyed / deterministic); 0/250%; 25/25

In every model the rebalanced adapter computes the same function: next-token logits agree to within 4e-13 in float64 with no change of greedy prediction in the three LMs, the routed experts are identical for every token in the mixture-of-experts LM, and the predicted noise agrees to 6.5e-6 in float32 in the diffusion model. The invariant merged sketch is also the fastest representation for scoring and sampling in all four models (0.002–0.03 s per selection, against 0.1–1 s for factor gradients).

AuthorsOpenReview ↗

Response to Reviewer LSgK (2/3): details of the new experiments (W1)

W1 (continued): details of the new experiments.

  • TIVE's rules. We apply TIVE's released scoring, allocation and sampling rules (commit 09c2c55: mean cosine within the task, task budgets proportional to the mean gradient norm with floor rounding, softmax(q·v) with q = 0.001, NumPy weighted sampling), with four instruction tasks, a 1,000-example pool and a nominal budget of 150 (15%, as in TIVE). The protocol of each experiment was fixed before any selection. Selection comparisons transport the stored gradients analytically in float64 under rebalancing (as in App. L.12), and an unchanged save–reload control re-extracts them on every checkpoint. Code, protocols and per-run results are in the supplementary material.
  • Vision-language model (App. L.17, Table 38). SmolVLM-256M-Instruct, with rank-8 LoRA on the query and value projections of its language model, the part whose LoRA gradients TIVE uses; the vision encoder is frozen. Four visual-instruction tasks are built from disjoint images: the digit in an MNIST image, the clothing item in a Fashion-MNIST image, the object in a CIFAR-10 image, and whether an image shows a digit, a clothing item or a photo. Rebalancing moves about 19 of the 150 budget slots between tasks in every checkpoint (the CIFAR-10 task gains 19–20 examples), so with raw factor gradients the data mixture itself depends on how the adapter is stored.
  • Mixture-of-experts LM (App. L.17, Table 38). Granite-3.0-1B-A400M-Instruct (1.3B total, 400M active parameters), with rank-8 LoRA on the attention query and value projections of every layer; experts and router are frozen. The routed experts are identical before and after rebalancing for every token of every pool example in all 24 layers, so the change in the selection comes from the gradient representation alone.
  • Diffusion model (App. L.17, Table 39). Rank-8 LoRA on the attention projections of the public DDPM CIFAR-10 UNet. 150 of 1,000 images are selected for fine-tuning toward the cat class by the cosine between each candidate's LoRA gradient and the mean gradient of 100 target images (TracIn/LESS-style), with the timestep and noise of every example's loss held fixed. Rebalancing alone changes 37–46% of the selection, more than a new selector seed (32%); W2 gives the full separation.
  • Decoder LM with BADGE (App. L.12, Table 2). Already in the revision: under LoRA rebalancing, BADGE on Qwen2.5-0.5B changes 50.2% of a query with inverse-CDF sampling and 7.9% with keyed sampling, the deterministic facility-location control changes 15.2%, and balanced and merged coordinates change 0%.

Why the results hold across models. The analysis concerns the LoRA factors, not the architecture around them. Rebalancing B→BGB\to BG, A→G−1AA\to G^{-1}A maps the factor gradients to (gBG−⊤,G⊤gA)(g_B G^{-\top}, G^{\top} g_A), which can change their inner products, cosines and norms unless GG is orthogonal (Sec. 6.1). TIVE's instance values are cosines and its task budgets are norms, so both can move; its within-task softmax draw is a stochastic selector, to which the coupling result of Proposition 3 applies. In balanced or merged coordinates, cosines and norms depend on the adapter only through BABA (Proposition 5), so values and budgets are invariant, and keyed sampling makes the draw reproducible.

Changes in the paper. New App. L.16 (TIVE on the decoder LM, Table 37) and App. L.17 (vision-language, mixture-of-experts and diffusion models, Tables 38–39).

AuthorsOpenReview ↗

Response to Reviewer LSgK (3/3): practical impact and sources of randomness (W2)

W2. Practical impact needs more evidence; five-round sets differ substantially while terminal accuracy barely changes; separate generation randomness from parameterization dependence, and measure coverage, reliability or efficiency.

Short answer. The revision adds a direct measurement of practical impact, an export–reload audit with measured costs and benefits (App. L.15). For this discussion we also measured generation randomness (the timestep and noise draws of the diffusion loss), selector randomness and the parameterization separately in a diffusion model (App. L.17, Table 39), following our protocol (Sec. 3.3): every input except the one under study is held fixed, and each random source is measured separately as a baseline. Unchanged accuracy is consistent with our claim, which concerns which examples are selected.

Details.

  • Generation randomness vs. parameterization (new, Table 39). In the diffusion model of W1 the loss of every example draws a timestep and noise, so the gradient estimate is itself random. We change one source at a time on each checkpoint (replacement, % of 150, raw factor gradients):
SamplerRebalancing (draws fixed)Draws resampledNew selector seed (draws fixed)
inverse-CDF45.982.532.1
keyed37.475.732.3
deterministic38.082.50

Of the three sources, resampling the draws changes the most, in every representation (76–84%); it is held fixed by generating each example's draws from its ID, as keyed sampling does for the selector. With the draws fixed, rebalancing alone still changes more of the selection than a new selector seed, and it changes 38% of the deterministic top-150, which no seed changes. Only invariant coordinates remove this part: in balanced and merged coordinates rebalancing changes 0% and every selection is reproduced exactly (25/25 for each sampler).

  • Selector seed vs. parameterization (Sec. 6.2; Tables 37–38). Over five rounds of acquisition, different seeds change 49–65%, keyed trajectories started from equivalent representations and then retrained on their own labels 32–43%, and the invariant control 0. In TIVE, a new selector seed never changes the task budgets, which are computed from gradient norms; rebalancing moves them in every checkpoint, by about 19 of 150 in the vision-language model. With the released q = 0.001 the within-task draw is close to uniform, so with raw factor gradients a new selector seed changes 76–84% of the examples across the three LMs. When the gradient scores drive the draw, as in a pre-registered sharper variant (q = 1000, with three other choices changed; App. L.16), rebalancing changes about six (decoder LM), eight (vision-language) and four (mixture-of-experts) times as much of the selection as a new seed with the stochastic samplers.
  • Export–reload audit (App. L.15, Tables 35–36). One process logs a query; the checkpoint is equalized with PyTorch's CLE routine and re-saved; a separate process recomputes the query. With raw gradients, the audit fails in every run with the inverse-CDF rule, in 22 of 50 BADGE runs with keyed sampling, and in 19 of 20 for CRAIG and GradMatch. With QuotientGrad every run passes, and end-to-end runtime changes by −1.2% to +4.5% versus raw gradients.
  • Reliability. With the inverse-CDF rule, 21.4 (MNIST) and 27.1 (Fashion) of 50 logged IDs are not reproduced on average, so if labels were obtained for the logged query and work continues from the recomputed one, that many IDs per query still need labels. With the fix, in a handoff using dataset labels as the returned annotations, the recomputed query reuses all 50 logged labels in every run, and weighted training inputs (IDs with coefficients) are reproduced for CRAIG and GradMatch in 20 of 20 cases, against 1 of 20 with raw gradients.
  • Coverage and efficiency. TIVE's task budgets set the data mixture. With raw factor gradients it depends on how the adapter is stored, and in balanced and merged coordinates it stays fixed (Tables 37–38). The invariant merged sketch is the fastest representation for scoring and sampling in all four models of W1.
  • Accuracy. We agree that the five-round sets differ while accuracy hardly changes, and the revision states this directly: turnover does not imply accuracy loss, and the case for invariant selection rests on reproducing the selected examples (Sec. 6.2). In Table 3, QuotientGrad's mean accuracy is within 0.12 points of raw selection, while random selection is 0.4–3.2 points worse.

Changes in the paper. App. L.15 (export–reload audit) and new App. L.16–L.17 (TIVE-style selection and the other model families).

We hope these additions address both points. If so, we would be grateful if you would consider raising your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank the reviewers for their careful reading. Section, table and line numbers refer to the revised PDF. One change affects how the submission's headline numbers should be read; it is set out in full in Section 3.

1. Strengths noted by the reviewers

  • The question. A selected set should depend on the predictor, not on its weight parameterization: "valid" (d8jE); "new and well posed" (ZAuu).
  • Consistency kept separate from utility. "Relevant" (d8jE); "important because data efficiency is a major topic in machine learning right now" (NXSC); "large query turnover leaves accuracy unchanged" (ZAuu).
  • Theory where it is needed. "Uses mathematical theorems to prove points where needed" (NXSC), including "the non-obvious insight that BAIT's sensitivity arises from its fixed regularizer rather than its Fisher terms" (ZAuu).
  • Controlled experiments. "Selector randomness is coupled, model equivalence is verified to 1e-13, and invariant controls return 0% swap" (ZAuu).

2. New or strengthened in the revision

AdditionReviewerWhere
Two operations practitioners already apply: cross-layer equalization (CLE; MLPs and the CIFAR-10 ConvNet) and LoRA rebalancing (Castin et al., 2026), with float32 write-backZAuu W1/Q1; NXSC W3Sec. 4.2, Table 1; Sec. 6.1, Table 2; App. K.1, L.4, L.5, L.13
A decoder-only generative language model (Qwen2.5-0.5B with LoRA)NXSC Q4Sec. 6.1, Table 2; App. L.12
Exact influence functions: Hessian transport, singularity and damping, plus an influence-based selection experimentNXSC W2/Q3Sec. 7, ll. 486–496; App. L.9, L.11
Two samplers with the same selection law on identical checkpoints; a coupling bound (Proposition 3, due to Daliri et al., 2024); a distortion measure η\eta with float32 and matched-noise controlsZAuu W3Secs. 3.3, 4.3, 5.5; Tables 1–2; App. L.1–L.4, L.6
Five-round acquisition under both samplers and under CLEZAuu W2, W3Sec. 6.2; App. L.10
Utility of the invariant control vs. raw and random selection, now in the main textd8jE W13; NXSC W1/Q1Sec. 6.2, Table 3; App. L.8
Practitioner recipe; no default-Fisher recommendationZAuu Q2Sec. 8, ll. 500–511

3. A change in how the main numbers should be read Reviewer ZAuu noted that a new random seed changes a BADGE query more than reparameterization does (W3). Following this up, we found that the size of the same-seed BADGE numbers depends on how the sampler maps random numbers to candidates. The submission used seeded inverse-CDF D² sampling, the rule in the official BADGE code and in scikit-learn's k-means++. We retrained the vision and encoder acquisition models with saved weights and ordered selections, and compared on identical checkpoints: inverse-CDF sampling, a candidate-keyed sampler with the same selection law, and deterministic selectors.

% of a 50-example query replaced, predictions unchanged:

Standard operationBADGE, inverse-CDFBADGE, keyedDeterministicInvariant
CLE (MNIST / Fashion / CIFAR-10)43.8 / 54.9 / 57.81.4 / 4.2 / 2.6CRAIG 4.0 / 4.4 / 12.0; GradMatch 2.4 / 6.4 / 23.60
LoRA rebalancing (MiniLM r=4 / 8 / 16; ModernBERT; Qwen2.5-0.5B)43.9 / 31.4 / 37.0; 56.8; 50.24.2 / 3.2 / 4.2; 1.8; 7.913.2 / 9.2 / 10.0; 3.8; 15.2 (†)0

(†) Facility location on the BADGE Gram with true-label class quotas; true-label CRAIG gives 15.2 / 13.6 / 9.6; 4.5; 14.8.

Submission vs. revision:

SubmissionRevision
Transformations behind the headlinesynthetic unit rescalings; nonorthogonal LoRA basis changestwo standard operations: CLE and LoRA rebalancing
Headline numbers18.8–63.0% (vision); 30.4–43.7% (MiniLM), 70.3% (ModernBERT)BADGE 31–58% (inverse-CDF) vs. 1–8% (keyed); deterministic 2–24%, growing with the distortion (up to 51% under strong LoRA basis changes)
Large BADGE and encoder-LoRA valuesread as the size of a geometric effectdepend on the reference sampling rule
CRAIG and GradMatch in Figure 2 (19–52%)part of the headlineno sampling involved, but synthetic rescalings whose log-scale spread is about five times that of CLE; now a stress test (ll. 252–258)
Selection aggregatesas reportedreproduced to their reported precision, with matching ordered selections (App. B.1, L.4)

The new headline is therefore not a corrected version of the old 19–63% range: it reports a different transformation regime and separates the sampling rule from the geometry.

Keyed sampling is not a complete fix. Over five acquisition rounds with retraining (Sec. 6.2, App. L.10) it keeps the first batch close (3–16% replaced, vs. 76–83% with inverse-CDF), but the trajectories then train on different labels and the difference compounds to 32–43% (different seeds: 49–65%). Only the invariant representation keeps trajectories identical (200/200 runs).

AuthorsOpenReview ↗

General response (2/2): other revisions

4. Other revisions

  • Introduction reorganized (problem, why it matters and is hard, prior work, our approach, contributions) with a new overview figure (Figure 1); Secs. 2–3 rewritten; each term defined at first use (d8jE W1–W4).
  • Figure 2 (query replacement under rescalings) redrawn with both samplers; the invariant controls, which replace nothing, are now marked at zero.
  • Sec. 4.1 states the rescalings, training data, pool construction and what is held fixed (d8jE W5; App. B.1).
  • Sec. 5: the setting precedes all results; Lemma 1 → Remark 1 and Theorem 1 → Proposition 1; the Jacobian is renamed JJ so it no longer clashes with the LoRA factor AA (d8jE W6–W10).
  • Experiments are separated from theory: protocol in Sec. 3.3, vision in Sec. 4, LoRA and utility in Sec. 6; the BAIT figure moved to App. H.3 (d8jE W11).
  • Balanced and merged LoRA representations are defined at ll. 412–416 (d8jE W12).
  • Prior work and how it differs from ours now sits in the introduction (ll. 76–84); Sec. 7 is shorter, and the longer survey is in App. M (d8jE W4).
  • New App. L: per-checkpoint tables, proofs, and an audit of sampling rules in reference implementations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.