acceptodds
Under review as a conference paper at ICLR 2027

GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency

Abstract

Efficient language-model pre-training requires deciding how to divide a fixed token budget across semantic regions and which examples to retain within each region. We introduce **GRIP** (Geometric Refinement and Adaptive Information Potential), which combines model-conditioned allocation with length-aware geometric selection. At the cluster level, static quality and normalized held-out loss reduction determine token budgets. Within clusters, a length-rectified inverse-density sampler increases geometric coverage while retaining long documents. We evaluate **8B and 16B Mixture-of-Experts models** on **eight code benchmarks** with budgets up to **300B tokens**. Across **five paired seeds at 300B tokens**, GRIP exceeds UniGeM by **0.9 and 1.3 average-score points** at 8B and 16B, respectively. Across **three paired seeds at 100B tokens**, GRIP reaches **35.5 ± 0.3**, **1.2 points above UniGeM**. The ablation separates the gains from cluster allocation and document selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DSKrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRIP, which combines model-conditioned allocation with length-aware sampling to improve data efficiency for language model pre-training.

Critical strengths
  1. This paper is well-motivated, it addresses token-budget allocation in language-model pre-training, where informative clusters receive insufficient capacity and cluster budgets are filled with redundant examples.
  2. This paper proposes GRIP, an effective method that combines model-conditioned allocation with length-aware geometric selection.
  3. This paper trains 8B and 16B MoE models from scratch and provides sufficient experimental validation.
Critical weaknesses
  1. I am concerned that the cluster quality scores rely entirely on Qwen3-235B-A22B. Although several outputs from each cluster are manually inspected, the paper does not report the scale of this manual inspection and does not evaluate robustness using alternative judge models. It remains unclear whether a single model’s preference for particular programming languages could bias the final token budget allocation.
  2. I notice that GRIP underperforms UniGeM on several tasks in Table 1. Could the authors explain these degradations and clarify whether they reflect systematic trade-offs introduced by GRIP’s data allocation strategy?
  3. The x-axis label in the left panel of Figure 4 is obscured. Please correct the figure in the main paper.
Questions for authors

See the Weakness.

Reviewer_FKLUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Efficient language-model pre-training requires deciding how to divide a fixed token budget across semantic regions and which examples to keep within each region. GRIP builds on UniGeM by allocating budgets conditioned on a small proxy model’s learning signal and by adding length-aware geometric selection, which corrects the authors' observed tendency of long documents to land in dense embedding regions and be under-selected by inverse-density sampling. The method is validated based on code tasks.

Critical strengths

Rigorous large-scale evaluation across different model sizes: 8B and 16B MoE models, training budgets: up to 300B training tokens, and multiple seeds: up to 5 seeds.

Corrects a length bias in density-based data selection: When selecting code documents for pre-training, long documents (whose chunk embeddings are averaged into one vector) end up in artificially dense embedding regions (Fig. 2a), so inverse-density sampling under-selects them. A simple length factor (Eq. 11) fixes this.

Critical weaknesses
  1. Advantage over UniGeM is still unclear: The advantage over UniGeM depends on a test-tuned length exponent. Without length correction, GRIP scores below UniGeM (Step 5's 33.6 vs. UniGeM's 34.3, Table 1 Panel B). It outperforms UniGeM only once β-dependent step 6 is added (35.5). No separate validation set was used for β selection. Moreover, it is unclear whether the reported result for UniGeM is based on a single seed or the same seeds used to report GRIP results.
  2. Some core equations and results still seem contradictory and unclear. The relation between ∆L and information potential is opposite between equation 7 and Figure 6. The reported gains would be stronger if the transition of the gains from normal to hard tasks is shown for the same dataset, like HumanEval+ (for HumanEval). Moreover, the gain is only based on code domain tasks, and it is unclear how the gains transfer to other domains.
Questions for authors
  1. Can you compare the UniGeM and GRIP results over the same seeds?
  2. How does hyperparameter selection, like β, based on a separate validation set impact the result, instead of selection based on the same test data?
  3. Since ∆L is positive in equation 7, higher ∆L means lower information potential p. However, Figure 6 shows the opposite relation. Can you clarify the reason for this contradiction?
  4. Can you clarify what "HE+" and "pass@1" indicate in Table 1, and how do they differ?
  5. Did you observe similar gains on other domains like commonsense reasoning, math reasoning, and so on?
  6. For the reported results over multiple seeds in the main paper, what is the standard deviation?
Reviewer_fFsrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GRIP, a pretraining data-selection framework. GRIP clusters training data and allocates sampling budgets based on data quality, diversity, and estimated learning potential using lightweight proxy models. The authors pretrained 8B and 16B MoE models from scratch using GRIP and show improvements over other data selection methods (UniGeM) across several coding benchmarks.

Critical strengths
  1. Addressing an important problem: Efficiently selecting pretraining data is increasingly relevant as the cost of training LLMs grows. GRIP offers a principled approach to selecting informative training data via a framework that does both cluster-level allocation and instance-level selection.

  2. Evaluation scale: The authors evaluate GRIP by pretraining 8B and 16B models from scratch, providing evidence of its effectiveness at a meaningful model scale rather than relying solely on smaller models or proxy evaluations.

Critical weaknesses
  1. Generalizability: The paper presents GRIP as a general data-selection framework, but the method is applied only to Stack v2 (code data), and the evaluation is also focused solely on code benchmarks. It is unclear from the paper if this approach is generalizable to other domains, specifically general-purpose language modeling.

  2. Limited interpretability of RAP: The authors explain how RAP estimates cluster-level learning potential but provide limited empirical analysis of its behavior.

  3. Figure quality and presentation: Some figures are difficult to interpret due to presentation issues, including missing or cut-off legends (i.e., Figure 4). Improving figure quality, readability, and layout would make the results easier to understand.

Questions for authors
  1. Is it possible to demonstrate GRIP's effectiveness on data beyond code, such as general natural language, to establish that the framework generalizes beyond code-focused pretraining?

  2. Is per-cluster allocation stable across probe and clustering seeds? How much is the variance across the seeds?

2 comments
AuthorsOpenReview ↗

Final Follow-up on Rebuttal Response

Dear Reviewer fFsr,

Thank you again for your valuable time and constructive feedback.

As today is the final day of the rebuttal discussion period, we would like to kindly follow up on our previous responses. We have carefully addressed your concerns but have not yet received your further feedback.

We would sincerely appreciate it if you could take a moment to review our responses and let us know whether your concerns have been resolved. If you find our clarifications satisfactory, we would be grateful if you could consider updating your evaluation score accordingly.

Thank you very much for your time and consideration.

I appreciate the author's effort in addressing my concerns. I have improved the score accordingly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1)

Dear Area Chair, Senior Area Chairs, and Program Chairs,

We would like to express our sincere gratitude for your time, patience, and efforts throughout the review process.

While our point-by-point rebuttal has addressed the reviewers' individual concerns in detail, we wish to provide a consolidated summary of their assessments and our additional experimental evidence for your convenience, as these responses are important for understanding the contributions and completeness of our work.

We are grateful to all reviewers for their detailed comments and constructive suggestions. We are particularly encouraged that two reviewers increased their scores from 2 to 3 following our rebuttal. We are pleased to see that reviewers recognize the following strengths of GRIP:

Reviewer fFsr (Score: 2 → 3): Recognized that efficient pretraining data selection is an increasingly important problem as LLM training costs grow. The reviewer appreciated GRIP's principled combination of cluster-level allocation and instance-level selection, and highlighted the evaluation using 8B and 16B MoE models pretrained from scratch as meaningful evidence beyond small-scale proxy experiments.

Final response: "I appreciate the author's effort in addressing my concerns. I have improved the score accordingly."

Reviewer FKLU (Score: 2 → 3): Highlighted the rigorous large-scale evaluation across different model sizes, training budgets of up to 300B tokens, and multiple random seeds. The reviewer also recognized GRIP's contribution in identifying and correcting the length bias of density-based data selection, particularly the under-selection of long documents.

Final response: "Thank you for the comprehensive rebuttal and the new experimental results. This effectively addresses almost all of my initial concerns, and I am happy to update my score to reflect this improvement."

Reviewer DSKr (Score: 3): Considered the paper well-motivated and recognized GRIP as an effective approach combining model-conditioned allocation with length-aware geometric selection. The reviewer also acknowledged the substantial experimental validation provided by training 8B and 16B MoE models from scratch.

We sincerely appreciate these positive assessments and the reviewers' recognition of both the methodological contributions and large-scale experimental validation of GRIP.

Summary of Questions Raised in the Reviews

The questions raised in the reviews primarily focused on the following aspects:

  1. Generalizability: Whether GRIP can extend beyond code-specific pretraining to general natural-language data.
  2. RAP and allocation robustness: Whether learning-potential estimation is empirically supported and whether cluster allocations remain stable across random seeds.
  3. Quality-assessment reliability: Whether LLM-based quality scores are consistent across independent judges and human evaluations.
  4. Experimental fairness: Whether comparisons with UniGeM, hyperparameter selection, and task-level performance differences are sufficiently justified.
  5. Methodological clarity and presentation: Clarifications of equations, evaluation metrics, and figure readability.

To address these questions, we conducted additional experiments, performed further analyses, and provided detailed methodological clarifications. The revised PDF marks changes in orange, including selected text edits and the headings and captions of new material. The additional natural-language experiments are reported in Section 4.4 and Appendix G, while Appendix H discusses the questions raised during the review process.

We summarize the additional evidence and our responses below in Part 2.

AuthorsOpenReview ↗

General Response (Part 2)

Point 1: Generalization Beyond Code

(Reviewers fFsr, FKLU)

We extended GRIP to FineWeb natural-language pretraining using an 8B MoE model, 100B training tokens, and three paired seeds. Compared with UniGeM, held-out perplexity decreased from 19.84 to 19.24 (3.02% reduction), while mean accuracy across HellaSwag, ARC-C, PIQA, and MMLU increased from 43.40% to 46.60% (+3.20 pp). GRIP improved on all four benchmarks included in the average. On GSM8K, reported separately, accuracy was 4.4% for GRIP and 3.9% for UniGeM.

These results provide additional evidence that GRIP generalizes beyond code-specific pretraining.

Point 2: RAP Interpretability and Allocation Stability

(Reviewers fFsr, FKLU)

We conducted controlled experiments across 48 clusters to examine the relationship between static quality, adaptation dynamics, and subsequent learning gains. High-quality, low-adaptation-delta clusters achieved the largest additional validation NLL reduction, supporting RAP's joint consideration of data quality and adaptation dynamics.

We also evaluated allocation stability across clustering and probe-sampling seeds. Median allocation-rank Spearman correlations were 0.88 across clustering seeds and 0.96 across probe-sampling seeds, with low budget divergence.

These results provide empirical support for RAP's design and demonstrate substantial allocation stability.

Point 3: Quality-Assessment Reliability

(Reviewer DSKr)

We supplemented the original manual inspection of more than 800 code samples with independent judge and human evaluations.

DeepSeek-V3.2 achieved a 0.86 cluster-level Spearman correlation with Qwen3. Across 400 documents, the overall Qwen–human Spearman correlation was 0.74. These analyses provide additional evidence of assessment reliability and characterize language-specific differences.

Point 4: Experimental Fairness and Performance Analysis

(Reviewers FKLU, DSKr)

We clarified that GRIP and UniGeM use identical paired seeds, candidate corpora, background data, architectures, optimizers, and token budgets.

Across three training settings, GRIP consistently improves aggregate performance, with all paired 95% confidence intervals excluding zero:

  • 8B / 100B: +1.2 pp.
  • 8B / 300B: +0.895 pp.
  • 16B / 300B: +1.298 pp.

We additionally conducted held-out validation for the length-correction parameter. On a second held-out code split, β = 0.3 yielded the lowest relative perplexity (0.987), compared with 0.989 at β = 0.5.

We also clarified task-level trade-offs. At 8B/100B, GRIP trails UniGeM by 1.7 pp on HumanEval+ but leads by 2.1 pp on MBPP+, using identical generated solutions evaluated on matched task sets. These results are detailed in Appendix G.3 and G.6.

Point 5: Methodological and Presentation Clarifications

(All reviewers)

Figures 4(b) and 7 in the revised manuscript show how information potential decreases with adaptation delta at fixed quality under Eq. (7). We also restored the full horizontal-axis label in Figure 4(a).

Furthermore, we clarified benchmark metrics, including HumanEval+, MBPP+, and Pass@1, and addressed the figure readability issues. The corresponding changes are incorporated into the revised manuscript.

Closing Remarks

Overall, these additional experiments and analyses strengthen the evidence for GRIP's effectiveness, generalizability, robustness, and methodological reliability.

We are particularly encouraged that Reviewers fFsr and FKLU both increased their scores from 2 to 3 after considering our additional experimental results and clarifications. We are also grateful to Reviewer DSKr for the constructive feedback, which motivated further investigation into our quality-assessment methodology.

The revised manuscript incorporates the additional experiments, methodological clarifications, and presentation improvements discussed above.

Finally, we sincerely thank the Area Chair, Senior Area Chairs, and Program Chairs for their time, careful consideration, and efforts throughout the review process.

We hope our responses and additional evidence will facilitate the final evaluation of our work.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.