acceptodds
Under review as a conference paper at ICLR 2027

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning

Abstract

Gradient-based data selection scores a sample by how it moves the model. Online LLM fine-tuning complicates this: utility depends on the current parameters, the effective update geometry is set by an adaptive optimizer, and per-example gradients are too large to materialize, so online rules score samples independently and weight the selected batch uniformly. We formulate online selection instead as subset-level gradient matching: choose non-negative weights whose composite update aligns with the target under the optimizer-induced geometry. Solved with the coupled greedy solver that gradient matching conventionally uses, this degrades on LLM gradients—the solver refits coefficients while the support is still being built, and the estimation error is comparable to the correlation gaps that decide each choice. Two conditions recover it: coefficient fitting must leave support construction, and the optimizer geometry must enter the alignment term alone, with redundancy measured in the candidates' own metric. Filter-then-Weight meets both. To run it inside the training step we hold per-example gradients as factorized outer products, project the two factors rather than the gradient, and contract the token axis before the feature axes; selection then costs under 9% of a step and carries no quadratic dependence on context length. Across backbones from 0.6B to 8B and two optimizers, it ranks above every online selection baseline we evaluate on both main benchmarks. An ablation shows that the gain comes from filtering and reweighting together.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Data Selection, Supervised Fine-tuning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UGr7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies online data selection and reweighting for LLM fine-tuning. The authors formulate sample selection as subset-level gradient matching under an optimizer-induced geometry and propose Filter-then-Weight, which first constructs a subset using a greedy residual filter and then solves a non-negative least-squares problem to determine sample weights. To make per-example gradient comparisons practical at LLM scale, the method uses factorized gradient representations, random projections, and token-first contractions. Experiments on Llama and Qwen models compare the method against several gradient-based data-selection baselines on MMLU, TyDiQA, and additional downstream tasks.

Critical strengths
  1. The paper addresses an important practical problem. Online data selection is more challenging than offline selection because sample utility changes with the model parameters, and accounting for the actual optimizer state is a meaningful consideration. The subset-level perspective is reasonable. Modeling interactions among jointly selected examples is potentially more expressive than assigning independent scores to individual samples, especially when candidate gradients are redundant.

  2. The implementation is carefully engineered. The factorized representation and token-first contraction substantially reduce the memory cost of computing candidate-target and candidate-candidate gradient statistics, making the approach applicable to models up to 8B parameters. The paper contains extensive ablations. In particular, the authors separately study filtering, reweighting, optimizer-aware preconditioning, projection dimension, candidate-pool size, and the frozen-preconditioner approximation.

Critical weaknesses
  1. The methodological novelty appears somewhat incremental. The main ingredients are gradient matching, greedy residual selection related to OMP, non-negative least squares, optimizer-aware gradient preconditioning, whitening/standardization, and random projection. The central algorithmic modification is to remove coefficient refitting from the greedy support-construction stage and postpone NNLS until after the support is fixed. This is intuitively reasonable and empirically useful, but I am not convinced that this constitutes a sufficiently substantial methodological contribution by itself.

  2. The baseline comparison is difficult to interpret because several prior methods are substantially modified to fit the proposed online setting. In particular, TracIn and LESS normally exploit historical checkpoints or full-corpus information, but the paper removes these components and evaluates only their scoring rules using the current-step gradients. This creates useful controlled baselines, but they are no longer the original algorithms. The evaluation relies heavily on repeated access to a target/validation distribution, and the role of this target data needs clearer characterization. At every selection step, the method computes a target gradient from a validation mini-batch and explicitly chooses training samples to align with it. Thus, the method is not task-agnostic online selection but rather repeated target-supervised selection.

Questions for authors
  1. Since Theorem C.1 does not establish equivalence between the alignment objective and squared-distance matching for nonlinear PtP_t, what theoretical justification supports Eq. (2) for Adam and AdamW, which are used in all experiments? Can the authors provide a bound relating optimization of the practical surrogate to improvement of the original target loss?

  2. The frozen second-moment approximation has high Spearman correlation with the candidate-specific version, but the reported relative ℓ2\ell_2 errors are 0.710.71--0.840.84. Why is ranking agreement sufficient for the subsequent NNLS weighting stage, which depends on the actual magnitudes of the Gram and alignment terms rather than only their ordering?

  3. Is there any theoretical stability or approximation result showing that the proposed unit-weight residual filtering is more robust than coupled OMP under noisy/projected gradients? At present, the motivation from classical OMP instability does not directly imply that the proposed decoupling solves the problem.

  4. Could the authors clarify the novelty of Filter-then-Weight relative to existing gradient matching plus post-selection NNLS/reweighting? Which component of the two-stage optimization procedure is fundamentally new beyond combining existing selection, preconditioning, and reweighting techniques?

  5. TracIn and LESS are modified substantially to operate without their historical checkpoint information. Could the authors discuss how much performance these modifications remove from the original methods? More importantly, could the authors compare against additional methods that are natively designed for online and optimizer-aware data selection?

Reviewer_g27pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Filter-then-Weight, an online framework for data selection and reweighting in LLM finetuning. At each training step, it computes a candidate-target alignment vector and a candidate-candidate pairwise interaction matrix for a candidate pool. It first greedily selects a subset using uniform weights, then solves a regularized non-negative least squares problem to determine the selected samples' weights. The method standardizes candidate gradients and incorporates Adam preconditioning into the alignment vector. It is evaluated against full data SFT and gradient-based baselines, including LESS, GREATS, and GRAD-MATCH.

Critical strengths

The paper improves existing gradient-based data selection methods in several concrete ways. It accounts for Adam preconditioning when computing alignment scores. Unlike GRAD-MATCH, it selects samples using unit weights and computes their final weights only once, aiming to reduce the influence of noise due to repeated weight refitting. It also makes gradient statistics cheaper to compute and store, with clear memory and runtime savings compared to the evaluated ghost inner-product implementation, especially for long token sequences.

Critical weaknesses
  1. Although the paper makes concrete methodological refinements, the downstream gains over the strongest selection baseline are generally modest, and the proposed method has the highest per-step runtime among the selection methods (as shown in Table 11). The practical benefit of the added complexity is therefore not fully clear. The long-sequence experiments show GPU memory and runtime savings, but do not report downstream performance. While Table 4 studies the effect of the projection budget on accuracy, it remains unclear how this budget affects both task performance and computational cost for longer sequences.
  2. The paper is sometimes difficult to follow. Some terms and comparisons need clearer explanations. For example, the runtime discussion uses different baselines without clearly distinguishing them, and it is unclear how the terms from support recovery apply to data selection setting. Simpler language and clearer explanations would help readers better understand the contributions.
Questions for authors
  1. GRAD-MATCH is the closest prior method. The main change is to select samples with unit weights and solve for non-negative weights only after selection, rather than repeatedly refitting weights. The motivation is that intermediate weight refitting may be sensitive to noise. Better performance than GRAD-MATCH supports the proposed design indirectly, but does not establish that reduced noise sensitivity is the reason. For example, does GRAD-MATCH use the same gradient standardization and Adam preconditioning as your method? If not, could you compare the two methods with these settings held fixed?
  2. Why is Adam preconditioning applied only to the alignment vector, but not to the candidate-candidate Gram matrix? Expanding eq. 2 seems to introduce preconditioning into both terms.
  3. GREATS already accounts for interactions among selected samples. Have you compared your method with GREATS followed by the same non-negative reweighting step?
Reviewer_YxtuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Filter-then-Weight (FW), an online, target-guided data-selection and reweighting method for LoRA fine-tuning that uses a small target dataset to guide updates from a larger, mixed-quality training pool. At each step, FW standardizes projected candidate gradients, incorporates a frozen Adam-style preconditioner into target alignment, constructs a subset through unit-weight greedy residual filtering, and then solves a ridge-regularized nonnegative least-squares problem once on the fixed subset. Its computational implementation projects activation and backpropagation factors separately and contracts the token dimension before forming gradient inner products, avoiding quadratic token-pair score tensors. Experiments on four backbones spanning 0.6B–8B parameters and Adam/AdamW report the highest mean performance among the evaluated selectors in all seven main MMLU/TyDiQA configurations under an approximately 5% selected-example budget; for example, best-checkpoint TyDiQA F1 on Llama-3.2-1B increases from 47.50 for GREATS to 48.86 for FW. Additional BBH and IFEval experiments show that benefits depend on the training pool’s relevance to the target, while ablations examine filtering, weighting, preconditioning, and implementation costs.

Critical strengths

The subset-level formulation explicitly considers interactions among jointly selected examples and their continuous weights, addressing redundancy that independent sample rankings cannot directly represent.

The factorized, token-first computation provides a practical way to construct both candidate–target alignment and candidate–candidate Gram statistics without quadratic token-pair tensors, supported by complexity analysis and long-context memory measurements

The empirical study includes multiple backbone sizes, two optimizers, three-seed main comparisons, component ablations, and additional targets with limited or negative gains, providing useful evidence about both effectiveness and operating limitations.

Critical weaknesses

For a frozen diagonal preconditioner DD and candidate gradients gig_i, expanding Eq. (2) requires pairwise terms Gij=⟨Dgi,Dgj⟩G_{ij}=\langle Dg_i,Dg_j\rangle, but Eqs. (4)–(6) put DD only in the alignment vector and use a standardized candidate-only Gram matrix, so the implemented surrogate does not follow from the stated isotropic-Hessian derivation.

  • On 50 candidate pools collected from existing early-, middle-, and late-training Llama-3.2-1B/TyDiQA checkpoints, could you please hold the selected support and standardization fixed and compare target-only preconditioning against a matching objective whose Gram includes D⊤DD^\top D, using the same ridge-tuning budget; replay each proposed update from an identical saved Adam state and report actual held-out loss reduction, update norms, and coefficient disagreement, requiring checkpoint replay rather than retraining.

The claim that repeated coefficient refitting causes the coupled solver’s degradation is confounded by other algorithmic changes, because the documented coupled routine uses unrestricted ridge weights and trial-objective selection, whereas FW uses nonnegative weights and residual-correlation selection.

  • Could you please add a coupled-NNLS control on the existing Llama-3.2-1B/TyDiQA setup that uses exactly FW’s projected gradients, standardization, alignment vector, Gram matrix, relative ridge, and residual-correlation selection rule, changing only whether nonnegative coefficients are refitted after every addition; run the same three seeds and 5% budget, reuse the existing FW results, and compare final F1, weight norms, and support sensitivity to small controlled gradient perturbations.

The evidence for consistent superiority is weaker than the headline significance claims suggest, because best checkpoints are chosen on the evaluation benchmark and Appendix E.5 pools explicitly non-independent conditions without accounting for their dependence, while fixed-budget results cover only four of the seven main settings.

Questions for authors

Please see above.

Comments on the paper

1 comment
~Louis_Dubois1OpenReview ↗

Does the evidence isolate Filter-then-Weight’s contribution?

The revision strengthens the empirical case by reporting fixed-budget results for all seven main configurations. The long-context memory measurements are also valuable. My main concern is whether the reported gains can be attributed to the proposed selection mechanism and whether that mechanism reliably improves the update it is designed to shape.

1. The contribution of residual filtering remains unresolved. The decisive comparison is top-kk on exactly the same standardized alignment vector, followed by the same weight solver. Table 4’s Hard Filter+Reweight variant changes both the ranking scores and the selection rule. Appendix E.6 reports identical subsets on 51 saved states for two nonzero penalty scales, making the zero-penalty control particularly informative.

There is also a failure mode with exact statistics. Consider four standardized candidates along the positive and negative coordinate axes in two dimensions, ordered as e1,e2,−e1,−e2e_1,e_2,-e_1,-e_2 and scaled to have squared norm 2. Let their alignment scores be (0.3,0.2,−0.3,−0.2)(0.3,0.2,-0.3,-0.2) and the budget be two. Equation (5) first selects candidate 1. It then scores candidate 3 at −0.3−(−2)=1.7-0.3-(-2)=1.7, ahead of candidate 2 at 0.20.2. The exact solution to Eq. (6) with positive ridge assigns candidate 3 zero weight. The fixed interim coefficient thus spends half the selection budget on a candidate that the final objective discards, even in the absence of estimation noise.

2. The surrogate and the released solver each have a concrete failure mode. With candidates g=(0.1,1)g=(0.1,1) and −g-g, target gradient h=(1,−1)h=(1,-1), and a scalar frozen preconditioner rescaled to the identity, the first-order alignments of the frozen update are (−0.9,0.9)(-0.9,0.9). Equation (4) gives (9,−9)(9,-9) using population standard deviations. With budget one, the surrogate selects an ascent direction even for a linear target loss. The fixed-subset replay in Appendix E.12 leaves this selection failure untested.

Separately, feeding the released NNLS routine the Gram and alignment vector formed from candidate rows (1,0,0),(0,1,0),(−1,0,1)(1,0,0),(0,1,0),(-1,0,1) and target (1,1,−2)(1,1,-2) yields normalized weights (0,1,0)(0,1,0) at learning rate 0.00010.0001 with the routine’s default ridge. The exact constrained solution, normalized in the same way, is (0.5,0.5,0)(0.5,0.5,0). Before normalization, the first coefficient is zero even though increasing it strictly decreases the objective. The routine removes variables after the unconstrained solve and cannot restore an incorrectly removed variable. This matters because the manuscript uses zero coefficients to interpret redundancy and effective batch use. Could the authors identify the solver version used for the reported runs and check its optimality residuals on saved states? Released solver

3. The TyDiQA comparisons should be verified on the same evaluation items. The released evaluator sets random.seed(42) but draws samples while iterating over a Python set of languages. With the public LESS data, changing only PYTHONHASHSEED from 1 to 2 produced two sets of 1,713 evaluation questions with 1,034 in common. Whether this affected the reported results depends on the runtime configuration. Verifying the question and one-shot demonstration IDs across methods and checkpoints, or reevaluating on a fixed list, would resolve this concern. Evaluator · Public LESS data

GRAD-MATCH already refreshes weighted subsets during training, and LoGra Eq. (6) already projects both gradient factors before summing over tokens. The online fine-tuning integration has value, and these precedents make it especially important to isolate what this particular combination adds. On the current evidence, I recommend rejection. The result most likely to change my assessment is a comparison of residual filtering against matched-score top-kk, using the same verified solver and a fixed evaluation set, with both downstream performance and actual one-step held-out loss changes reported. Each variant should be allowed to select its own subset in the replay. GRAD-MATCH · LoGra

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.