acceptodds
Withdrawn from ICLR 2027

DRESS: Descent-Oriented Data Selection for Token-Efficient Language Model Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Improving token efficiency is increasingly important for training modern large language models. Training tokens differ in their contribution to learning, and their usefulness can change as training progresses, motivating online data selection that adapts to the current model. We propose Descent-oriented Reweighting for Efficient Sample Selection (DRESS), which scores training examples by their predicted contribution to reducing a held-out target loss. From first- and second-order Taylor expansions, we derive two variants, DRESS-FO and DRESS-SO, where DRESS-SO uses a generalized Gauss–Newton (GGN) approximation to capture interactions among candidate updates. The two variants offer a trade-off between scoring cost and the fidelity of the local target-loss approximation. We evaluate DRESS on GPT-2 pretraining across model sizes from 124M to 1.5B parameters and on Qwen3.5-9B-Base LoRA fine-tuning. On GPT-2 XL, both variants outperform the compared baselines in average zero-shot accuracy, and with only 30B update tokens, they even outperform random selection trained on 60B update tokens. When tested on Qwen3.5-9B-Base, both versions of DRESS improve over the standard fine-tuning baseline on evaluated benchmarks, while the second-order DRESS outperforms the first-order version.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, online data selection, token-efficient training, mirror descent.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NK8yOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses a well-defined and practically relevant problem established by closely related work (GREATS, OPUS). The paper's main contribution is to introduce a curvature estimate with retained directional information to address the gap that was left open by related work. Furthermore, the paper introduces second-order methods to stabilize the training process and improve evaluation results. Experimental results cover pre-training and fine-tuning.

Critical strengths

The paper addresses a well-defined and practically relevant problem established by closely related work (GREATS, OPUS). The idea of introducing a curvature estimate with retained directional information can help alleviate what related work left open before. The idea of using second order methods is cheap and scales in practice, which is helpful as model parameter sizes grow. Showing applicability to both pre-training and fine-tuning is helpful for practical guidance.

Critical weaknesses

The paper has several areas of improvement. I list them by impact:

  • None of the experimental results contains a standard deviation or confidence interval. There is no evidence that the results reported in the paper are a product of chance or whether DRESS is reproducible. I understand that pre-training is cost and resource intensive but for a study like this having a standard deviation across at least 3 seeds is a minimum, I think.
  • Almost all pre-training benchmarks show results at near chance (e.g., MMLU reports 25-33%; with random choice being 25%). At such low benchmark scores it tells little about the performance of DRESS. Strangely, ARC Challenge is below chance (without an explanation why). This hints at two things that warrant a more thorough discussion and analysis, potentially involving larger models (GPT-2-style models are almost 8 years old).
  • While the paper discusses some sample-level reweighting approach, it does not provide an experimental comparison. This comparison could provide the reader a more fundamental understanding how the proposed DRESS method compared against existing work.
  • There are several issues on the benchmark and evaluation design: The paper seems to copy baseline numbers from the OPUS paper instead of running the method on their own to ensure an apples-to-apples comparison. Hence, the insights discussed only have limited value. Similarly, the α\alpha value appears to be tuned on the target benchmarks, without providing a discussion on how to tune α\alpha for practical settings. Also for pre-training, it seems the results for Muon are only theoretically derived but never evaluated experimentally (only AdamW on the LM heads). Furthermore, the setup of lora experiments is very narrow (1 dataset only with a base model that performs strongly on the target benchmark).
Questions for authors
  • What would be a practical intuition for tuning α\alpha ?
  • How does DRESS compare to closely related sample-level reweighting approaches like LinUpper or InfoBatch?
  • What is the end-to-end FLOP comparison including the forward passes on the 60B candidate tokens? How do the results compare to the Random baseline at identical wall-clock budget?
Reviewer_VhzrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DRESS, an online data selection framework that prioritizes training examples according to their predicted contribution to reducing a held-out target loss. The method derives first-order and second-order influence scores from a local Taylor approximation of the optimizer update. DRESS-FO uses the alignment between preconditioned candidate gradients and the target gradient, while DRESS-SO incorporates directional curvature through a generalized Gauss–Newton (GGN) approximation. Candidate scores are converted into sampling probabilities using a KL-based mirror-descent update. The authors evaluate DRESS on GPT-2 pretraining at three model scales and Qwen3.5-9B LoRA fine-tuning. DRESS-SO achieves the best reported average zero-shot accuracy among the compared 30B-token methods at the GPT-2 Large and XL scales and improves mathematical reasoning performance in the fine-tuning setting. Additional experiments investigate scoring overhead, LM-head approximations, and the fidelity of second-order target-loss predictions.

Critical strengths
  1. A principled extension of optimizer-aware data selection through directional curvature.

The paper addresses a meaningful limitation of existing online data selection approaches that approximate second-order interactions using isotropic curvature. Replacing this approximation with a GGN-based curvature correction is theoretically motivated and provides a more expressive local estimate of target-loss reduction. The proposed shared curvature-vector product also avoids explicitly materializing the Hessian or all pairwise candidate interactions, making the approach computationally more practical than a naive second-order implementation.

  1. Encouraging empirical results across pretraining and fine-tuning.

The evaluation includes strong existing data selection baselines and multiple model scales. The GPT-2 XL results show improvements in average zero-shot accuracy over the compared methods, while the Qwen3.5-9B experiment provides preliminary evidence that the approach is useful beyond from-scratch pretraining.

Critical weaknesses
  1. Limited evidence that the proposed GGN correction provides a substantial and robust improvement over existing methods.

The first-order scoring criterion is closely related to the optimizer-aware gradient alignment already studied in OPUS. Thus, the main methodological contribution is the GGN-based second-order correction. However, the empirical improvement from DRESS-FO to DRESS-SO is relatively small in GPT-2 pretraining: 0.20 and 0.22 percentage points in average accuracy at the Large and XL scales, respectively. More importantly, the paper does not include an end-to-end comparison between isotropic second-order scoring and GGN-based scoring under an otherwise identical selection pipeline. The target-loss prediction experiment in Table 5 is useful, but it does not establish that improved local approximation fidelity translates into meaningful downstream gains.

  1. Insufficient evidence for the statistical robustness and compute-adjusted significance of the improvements.

Many of the reported downstream improvements are modest, and the main training experiments do not provide variability across independent runs. In particular, the advantage over Random trained on 60B update tokens is only 0.22 and 0.36 percentage points for DRESS-SO at the Large and XL scales. The GPT-2 Large/XL results for competing baselines are reused from OPUS rather than independently rerun, making careful protocol comparability particularly important when interpreting small margins.

Questions for authors

Q1. Contribution of directional curvature.

Could the authors compare DRESS-FO, isotropic second-order scoring, and GGN-based second-order scoring in an otherwise identical end-to-end training experiment? This is important because the central methodological distinction from prior work is the use of directional curvature rather than an identity approximation.

Q2. Statistical robustness.

Can the authors provide independent training-run variability, or other evidence of robustness, for the improvements of DRESS-SO over DRESS-FO and the strongest baselines? This would be particularly useful for interpreting the relatively small average accuracy differences.

Q3. Scope and computational cost of the second-order approximation.

For fixed backbone representations and a linear softmax LM head, the GGN equals the exact Hessian in the restricted head parameter space. Could the authors clarify this relationship and explain the slight differences between the GGN and exact-Hessian results in Table 5? Additionally, what is the end-to-end training-time overhead of DRESS-SO for Qwen3.5-9B LoRA fine-tuning, and how does the method compare under a matched total-compute budget?

Reviewer_5obQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DRESS selects training samples according to how much their parameter updates are predicted to reduce loss on a separate target dataset. DRESS-FO measures gradient alignment; DRESS-SO also approximates how loss changes along the combined update direction. Scores determine which samples are used for training. GPT-2 pretraining and Qwen mathematical fine-tuning improve over random selection.

Critical strengths
  • Sample scores are tied to predicted learning progress on the target dataset, rather than simply treating a high-loss training sample as useful.

  • The second-order correction is computed without storing a full Hessian or individual sample gradients. During pretraining, scoring derivatives are restricted to the output layer, reducing work compared with full-gradient scoring; the model backbone still runs forward.

  • The paper tests both pretraining and mathematical fine-tuning. It also compares predicted versus actual loss changes on small-model updates, showing a more accurate prediction when directional curvature is included.

Critical weaknesses
  • On GPT-2 Large, average zero-shot accuracy is 41.66% for DRESS-SO, 41.61% for OPUS, and 41.46% for DRESS-FO. No repeated-seed results show whether the 0.05-point lead over OPUS is larger than training noise.

  • The Large/XL baseline accuracies are taken from the OPUS paper rather than reproduced alongside DRESS. The authors state that the protocol matches, but the small accuracy gap and separately measured costs would be more convincing under a common implementation and hardware setup.

  • DRESS trains on 30B tokens selected from roughly 60B candidate tokens and pays to score the candidates. It reports extra time at a fixed update-token budget, but not total time compared with random training on 60B tokens. Beating that random baseline in accuracy does not by itself prove equal-compute efficiency.

  • All sample scores aim to improve a fixed target dataset. The pretraining target is general English web text, and the fine-tuning target is mathematics. The paper does not test whether changing the target composition changes which samples are selected or harms performance outside that target domain.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.