acceptodds
Withdrawn from ICLR 2027

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data selection, LLM continual pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rQTKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Based on what I understand; this paper proposes a Fisher-based method for data selection because there is constraint on the resources and authors use this method for continual pre-training of large language models. For each candidate gradient, they have two different views; an anchor component for high Fisher directions and a frontier component for low Fisher directions. Authors also have a log-determinant objective and Sieve-Streaming for selecting diverse training examples. They tested their idea on TinyLlama-1.1B and Llama-3.1-8B on medical data and showed by using this method we can achieve higher accuracy (or I should say plasticity for learning) while forgetting the previous information less.

Critical strengths
  1. The idea of balancing plasticity (learning new things) and catastrophic forgetting (forgetting previously learnt data) is very interesting and tricky. Moving the Pareto front in this problems is very difficult and based on my own experience once we improve one aspect; the other one start to fail. This paper method looks like have handled both aspects at the same time.
  2. Other thing that I like about this paper is how they handled which documents they want to select and using log-determinant for assign a larger credit to documents that has a high value but also are not repetitive.
Critical weaknesses
  1. I can see the authors maximize a log-determinant objective for data selection. However, the anchor term rewards gradients in high-Fisher directions, which the paper says are important for retaining previous knowledge. I do not understand why maximizing this term should reduce forgetting instead of increasing interference with previous knowledge. Also I am concerned about the bounds of this loss.
  2. The evaluation is very limited. The models are limited, but the bigger problem is that the authors only test adaptation to one target domain (medical), not a sequence of domains. The reason I am concerned about this is because the loss of plasticity usually starts to happen and grows as the number of tasks grows, and we may not see this when we test our methods on a limited number of tasks. Even vanilla learning methods in networks can learn new things without losing plasticity for a few tasks; so I cannot be assured that the effect of frontier is real in a longer continual learning setup.
Questions for authors
  1. Based on the first critical weakness, can you please explain the intuition behind maximizing the anchor term? How does maximizing gradients in high-Fisher directions help reduce forgetting instead of increasing interference with previous knowledge? Do you have any proof for the bounds of this loss? Can it lead to instability because of maximization?
  2. Based on the second critical weakness, can you please test on a higher number of sequential tasks or domains and compare the baseline with anchor and frontier cases to see how they perform over multiple adaptation stages? You do not need to do a long and large experiment; you can use a smaller language model with several sequential text domains.
Reviewer_XQA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a clear and coherent narrative. The writing is carefully developed and communicates the authors' ideas effectively. The methodological analysis is detailed, and the experimental evaluation is substantial.

The paper proposes a Fisher-guided submodular data selection framework for continual pre-training (CPT) of large language models, aiming to improve target-domain adaptation while mitigating catastrophic forgetting. Experiments on TinyLlama-1.1B and Llama-3.1-8B in the medical domain demonstrate improved adaptation performance with substantially less forgetting than random selection, perplexity-based selection, and replay-based baselines.

Critical strengths

S1. Strong performance balancing adaptation and forgetting. The data selection strategy improves the adaptation–retention trade-off. Consistent results across two model sizes and ablations verify its effectiveness, addressing a core continual pre-training challenge.

S2. Practical pipeline combining Fisher-aware selection and streaming submodular optimization. This end-to-end pipeline integrates Fisher-weighted gradients, log-determinant diversity, random projection and streaming subset maximization, accounting for sample redundancy unlike simple ranking.

Critical weaknesses

W1. The objective clashes with its anti-forgetting rationale. Though it intends to constrain updates on high-Fisher sensitive directions, Eq.(3)’s log-determinant objective prioritizes samples with large gradients in those directions. The paper fails to explain why optimizing diversity in the high-Fisher subspace preserves pretrained knowledge. Fisher information cannot directly measure forgetting, and the Fisher-drift evidence for knowledge loss is unjustified, weakening its core conceptual claim.

W2. Theory is disconnected from practical implementation. While the log-determinant objective enjoys submodularity and Sieve-Streaming bounds, multiple practical approximations (LoRA warmup, diagonal Fisher estimation, etc.) distort the original optimization geometry. The theoretical guarantee applies only to the surrogate objective, not downstream performance. Tests are limited to medical CPT, without proven cross-domain generalization. Its claim as a general principled anti-forgetting solution remains unsubstantiated.

Questions for authors

Clarification of the anchor term and forgetting prevention Equation (3) rewards the log-determinant volume of the anchor gradients, which emphasize high-Fisher directions. Since these directions are described as sensitive to changes in pretrained behavior, why does maximizing this term encourage retention rather than potentially destructive updates? Could the authors provide a theoretical argument or a controlled empirical analysis connecting increased anchor utility to reduced forgetting? For example, measuring actual parameter displacement in high-Fisher coordinates, or changes in reference-distribution loss, would help determine whether the proposed objective truly discourages interference.

Reviewer_r6WeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tells a Fisher-guided submodular data selection framework for cpt of large language models, aiming to improve target-domain adaptation while mitigating catastrophic forgetting. The method decomposes candidate gradients into an anchor component, emphasizing high-Fisher directions associated with pretrained capabilities, and a frontier component, emphasizing low-Fisher directions with greater adaptation potential.

Critical strengths

The motivation is good and the exp design is also excellent (except the model choice).

Critical weaknesses
  1. I don't understand why in the year of 2026, the author still choose TinyLlama-1.1B and Llama-3.1-8B for exps? Why not choosing models more modern like Qwen-3.5 2B or 4B or 9B? These are also on par with the size of Llama-3.1-8B.

  2. The experiments focus exclusively on medical-domain CPT with LoRA adaptation, leaving robustness across other distribution shifts, reference distributions, and full-parameter training unclear

  3. The anchor log-determinant term rewards gradients in high-Fisher directions, even though these directions are described as sensitive to destructive updates. Maximizing this term does not explicitly constrain parameter displacement or gradient interference in those directions. Thus, the observed reduction in Fisher drift does not necessarily establish a causal relationship with capability retention

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.