acceptodds
Under review as a conference paper at ICLR 2027

Online Label-Free Gradient Matching for Continual Language Model Training

Abstract

Language models in customer-facing products must be continually updated, since the data they serve shifts over time. As retraining on the full accumulated data is impractical, each update should instead run on a small set of examples carrying the training signal of all data seen so far. The strongest current methods build this set by gradient matching, which synthesizes examples whose training gradients match those of the full data. However, they require ground-truth labels, run once on a fixed snapshot rather than on a growing stream, and are slow on large data. We introduce OLGA, which lifts these limitations. It distills unlabeled data by drawing the gradient matching targets from the model's own next-token predictions. Across five text classification benchmarks with phi-1.5, it improves accuracy by 10 percentage points over the strongest selection baseline using as few as five examples. OLGA also distills online data streams by keeping a running rehearsal memory of synthetic examples, which is re-distilled together with each new batch as it arrives. In the online setting, it outperforms the strongest selection-based continual baseline by 6.6 stream-AUC percentage points. Finally, our method scales to large data by pairing distillation with selection in a hybrid pipeline, letting selection handle the mild stage of compression and distillation the extreme one. It cuts runtime by 30% versus distillation alone at similar accuracy, and beats the best selection-only pipeline by up to 12.6 percentage points.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Dataset Distillation, Gradient Matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yEf5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper works to address the problem of online learning in a constrained setting where we don't want to maintain the full data history, or update the model on a large dataset. The authors propose OLGA (Online Label-free Gradient Matching) as an algorithm that handles the above limitations by first generating synthetic data by matching gradients with the real data, then uses the model to make a best guess on synthetic data labels, and selects a subset through a distillation/core selection step. The authors validate their method in both offline and online settings using 5 classification datasets, comparing against an existing method (GradMM) and a series of ablations on their proposed method.

Critical strengths
  • I appreciate that the ablations and analysis of OLGA are fairly thorough
Critical weaknesses
  • I'm not an expert, but some of the experimental setup appears to be quite out of date (nearest comparison baseline is from 2018, most recent dataset is also from 2018). There has been a lot of advancement in NLP since then, and it would be much more compelling to see methods/data from more recent times
Questions for authors
  • The text in the figures is REALLY small, I struggled to read them. Is it possible to increase the font size?
  • I'm concerned about the lack of modern comparisons here. I can appreciate that things move very quickly right now in NLP, but datasets and methods from 2018 are no longer contemporary works. Can you search to see if there are any more recent classification datasets? Also, are there more modern methods of offline, unsupervised learning than k-center greedy (Sener & Savarese, 2018)?
  • What is the "selection" method in Table 2? Does that mean, select to k=50 and then train on all 50? The definition of that method should be mentioned somewhere
  • Lines 370-371 just have a strange phrasing that should be improved: "Table 2 compares the accuracy of the three routes in the offline setting, where the whole compression happens once over a static pool, and Table 2 compares what each route costs in time."
Reviewer_uvUkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OLGA as a data distillation method for language models, which outperforms other dataset distillation methods on efficiency and text classification tasks. However, the evaluated tasks are very narrow to 5 easy classification tasks, the robustness and scalability of the proposed method are still questionable.

Critical strengths

(1) Targeting at a critical data efficiency topic.

(2) The proposed method outperforms a list of data filtering baselines on 4 out of 5 tasks, while none of the baseline methods use gradient-based methods.

Critical weaknesses

(1) No efficiency gain over standard training: even if the proposed L_{grp} removes the verbalizer (target labels), it still requires computing the batch's gradient along the input prompts. To synthesize the DsynD_{syn} using ADMM needs more extra computation overhead, so the final efficiency is worse than standard training. Can you provide a concrete efficiency computation of your proposed algorithm and other baselines?

(2) Lack of another baseline: since the data distillation is label free, we should compare to a label-free baseline where we don't let the model see the final verbalizer but train on the question stem. It should at least be equivalent to the performance of the proposed distillation method since DsynD_{syn} is up-bounded to this label-free standard gradient DrealD_{real}.

(3) The evaluation tasks are too narrow: the method is evaluated to 5 simple text classification tasks with clear domain features. Also, OLGA fails to outperform other methods in TwitterEmotion benchmark. Whether this method can be generalizable to general LLM pretraining or continual training is questionable.

(4) Confusion on critical methodology: the authors mention in section 3.4 (line 265-266) "a coreset selector first cuts the incoming batch of B examples down to a smaller pool of K " but have not mention what is the corset selector and how it conduct this filtering. Please add some description on this.

Questions for authors

(1) Efficiency: Can you provide a concrete efficiency computation of your proposed algorithm and compare to other baselines?

(2) Extra baseline: we should compare to a label-free baseline where the model is trained on the question stem without verbalizer.

(3) How does the loss LgrpL_{grp} depends on cluster partition? is it robust?

(4) Can you evaluate OLGA on general LLM instruction tuning, mid-training or post-training tasks to show its generalizability?

(5) Please add more descriptions on the corset selector in section 3.4 (line 265-266)

4 comments
AuthorsOpenReview ↗

Response to Reviewer uvUk (1/3)

Thank you for the careful review and the concrete questions. They pushed us to run three new experiments, and here's a quick summary before we go through each point:

  1. Label-free LM baseline (W2/Q2). Training on the 50 real anchors with the next-token loss actually lowers accuracy by 1.3 points compared with zero-shot. OLGA with just 5 synthetic examples raises it by 16.4.
  2. Robustness to the group partition (Q3). Even when 20% of each group is put in the wrong group, average accuracy moves by at most 0.6 points.
  3. A different distilling model (W3/Q4). When Qwen3-1.7B does the distilling, OLGA gets better on every dataset (+7.0 on average) and beats the best selection baseline by 10.6 points. That includes a 9-point lead on TwitterEmotion.

For efficiency (W1/Q1), we restate the runtimes from Table 2 below.

W1 / Q1: Efficiency compared with the baselines and with standard training

We don't claim distillation is free, and the paper reports its cost against every method we compare in Table 2. Here are those numbers again. Each is the wall-clock time of one offline run on SST-2, going from 1,000 examples down to 5, with the average accuracy over the five datasets in parentheses.

RouteGradMM (labeled)OLGA (label-free)Selection
Direct622.1 s (82.7)639.9 s (80.9)4.8 s (66.6)
Double2073.6 s (73.9)2086.4 s (72.2)5.5 s (73.4)
Hybrid (ours)436.1 s (86.0)447.5 s (80.0)—

A few things stand out:

  • Dropping the labels costs nothing extra. OLGA runs within 3% of supervised GradMM.
  • Our hybrid pipeline makes distillation 30% faster with similar accuracy. Like GradMM, we also match only last-layer gradients (Sec. 4.1), which GradMM showed cuts generation time by 2.3× and memory by 2.6× (their Sec. 5.4).
  • Selection is about 100× faster, but it's much less accurate. OLGA-Hybrid beats the best selection route by 6.6 points for a one-time cost of under 8 minutes.

As for standard training, the reviewer is right that if you train a single model once, distilling first costs more than just training on the data. But that isn't where distillation pays off. Its savings come from three places, which the paper motivates in the introduction, the related work, and Sec. 3.3.

  1. The distilled set gets reused. For text, distillation is really a way to generate training text. DiLM (Maekawa et al., 2024) and GradMM (Nguyen et al., 2025) switched from continuous embeddings to readable text exactly so the result isn't tied to the model that made it. GradMM shows this directly in their Sec. 5.2.2 and Table 2. They distill 20 examples with phi-1.5 and use them to fine-tune Llama-3.2-1B and OPT-1.3B, which then beat zero-shot, few-shot, herding and k-center selection. You pay for distillation once, but you save on training every time another model is trained on the result. OLGA's output is plain text too (Sec. 4.1), so it keeps this property.
  2. Continual updates. A deployed model gets updated again and again. Retraining on everything seen so far costs O(tN) at step t, which adds up to O(T²N) over T steps. OLGA's cascaded merge only distills the new block together with the current memory, so each step costs the same (Sec. 3.3) and the total only grows as O(T).
  3. Distillation cost doesn't grow with the data. In the hybrid pipeline (Sec. 3.4), distillation always runs on a pool of K examples, no matter how big the incoming batch is. Training on all the data grows linearly with its size.

Our contribution is to the distillation algorithm itself, namely removing the need for labels and handling a data stream. That's why we measure how accurate the distilled set is. GradMM has already shown that sets from this family of methods can be reused across models.

Paper change: right now these points are spread over the introduction, the related work, Sec. 3.3 and Table 2. We'll pull them into one paragraph that compares distillation directly with standard training, so they're easy to follow for readers outside the distillation literature.

AuthorsOpenReview ↗

Response to Reviewer uvUk (2/3)

W2 / Q2: Label-free baseline without the verbalizer

Thanks for suggesting this baseline. We ran it and will add it to the paper. We fine-tuned phi-1.5 with the standard next-token loss on the same 50 unlabeled anchors that OLGA distills from, with no verbalizer, and kept the best of three learning rates. Each row shows the change from the zero-shot model under the same evaluation.

Change vs. zero-shotSST-2RTIMDBRT-PolTwitEmoAvg
LM loss on 50 real anchors−6.6+1.2+3.3−4.50.0−1.3
OLGA, 5 synthetic+22.9+10.1+25.5+10.3+13.0+16.4
Oracle (all real data, labeled)+28.8+20.3+28.3+18.9+35.9+26.4

Plain LM training sees ten times as many examples, all of them real, and still doesn't beat zero-shot on average. OLGA closes more than 60% of the gap to the labeled oracle with only 5 synthetic examples.

This is what we'd expect given how the two objectives differ. We'll spell it out in the paper, since the review suggests that ℓgrp\ell^{\mathrm{grp}} is upper-bounded by the LM gradient.

  • The two losses aim at different targets. The LM loss predicts the next token of each text, so its gradient follows topic and style. It's basically continued pretraining on the domain. The target qgq_g of ℓgrp\ell^{\mathrm{grp}} (Eq. 3) puts its mass on the k tokens with the highest pointwise mutual information with the group, meaning the tokens that set that group apart from the others. So its gradient points toward the distinction the task actually cares about. It isn't a restricted version of the LM gradient, and neither loss bounds the other.
  • The LM baseline doesn't compress anything. It trains on all the real anchors instead of a 5-example set. So it works as a reference point, not as a competing compression method.

In short, the experiment confirms that OLGA's label-free signal comes from the grouping and not just from seeing in-domain text.

Paper change: we'll add this experiment as an ablation in Sec. 4.5, along with the explanation above.

Q3: Dependence on the group partition

To test this, we moved 1, 5 or 10 anchors per group into the wrong group. We then distilled with OLGA, fine-tuned phi-1.5, averaged over 3 seeds, and compared against a run with the correct groups under the same protocol.

1 swap5 swaps10 swaps
Share of each group that is wrong2%10%20%
Avg. change vs. correct groups (5 datasets)+0.1−0.5−0.6

OLGA holds up well to grouping errors. Even with a fifth of each group wrong, average accuracy shifts by at most 0.6 points. That's because qgq_g averages the model's predictions over the whole group, so a few misplaced examples barely move the target. SST-2 is the most sensitive dataset when the noise is heavy. On the other datasets, the normal variance of fine-tuning on 10 examples is larger than the effect of the swaps. We'd also expect groups built from logged metadata or clustering (Sec. 3.2) to be cleaner than this stress test, where a fifth of each group is wrong.

Paper change: a new appendix section with per-dataset and per-seed results, referenced from Sec. 3.2.

AuthorsOpenReview ↗

Response to Reviewer uvUk (3/3)

W3 / Q4: Scope of the evaluation and generalizability

Why these tasks. We picked this setup so we could run a controlled study and compare cleanly with the work we build on. GradMM, the supervised method we extend, uses the same setting, so our label-free results line up directly with its supervised ones (Table 5). Classification also gives a clear measure of what a 5-example set teaches the model. We need that to fairly compare synthesis with selection at such extreme compression.

New result: generalization across models. OLGA outputs plain text, so the model that distills doesn't have to be the one that trains. To check that OLGA doesn't depend on phi-1.5 as the distiller, we had Qwen3-1.7B do the distilling and fine-tuned phi-1.5 on what it produced. These numbers are preliminary, from a single run with the best of 5 values of k and 3 learning rates.

SST-2RTRT-PolTwitEmoAvg
OLGA, Qwen3-1.7B distills86.778.784.166.579.0
OLGA, phi-1.5 distills (Table 5)82.374.574.656.572.0
Best selection baseline (Table 5)75.370.470.457.568.4

With a stronger distiller, OLGA improves on every dataset, by 7.0 points on average, and beats the best selection baseline by 10.6. So the method isn't tied to phi-1.5, and it gets better as the distilling model gets stronger.

TwitterEmotion. We agree that with phi-1.5 as the distiller, OLGA only ties the best baseline here (−1.0 offline, −0.1 online), even though it leads by 10.2 and 6.6 points on average. The weak spot is the label-free starting signal, not distillation itself. The zero-shot model scores 43.5, so its next-token predictions barely tell sadness apart from joy. With labels, the same distillation machinery reaches 78.6 (GradMM). The new result above backs this up. With a stronger distiller, OLGA reaches 66.5 on TwitterEmotion, 10.0 points higher than with phi-1.5 and 9 points above the best baseline.

Instruction tuning and mid-training. We agree these settings matter. We can't run them within the rebuttal period, but nothing in OLGA is specific to classification. ℓgrp\ell^{\mathrm{grp}} only needs a grouping, and instruction data already provides one through task or domain metadata (Sec. 3.2). We'll add this to the paper as a limitation and a direction for future work.

W4 / Q5: Which coreset selector is used

The reviewer is right that Sec. 3.4 doesn't name the selector where it first comes up. It's uniform random sampling. Sec. 4.4 says so, and the Alternative selectors ablation (Sec. 4.5, Table 3, with per-dataset results in Appendix A7, Table 11) compares it with k-center. Uniform sampling is more accurate on all five datasets (86.0 vs. 82.1 on average) and also faster (436 s vs. 565 s). We think this is because sampling keeps the original data distribution, which distillation then compresses better. Paper change: Sec. 3.4 will read "a coreset selector (uniform random sampling by default; Sec. 4.5 compares it with k-center) first cuts …".

Summary of revisions

  1. One paragraph comparing distillation directly with standard training (W1).
  2. The label-free LM baseline as an ablation in Sec. 4.5 (W2).
  3. A group-robustness appendix (Q3).
  4. A cross-model distillation appendix (W3).
  5. The selector clarification in Sec. 3.4 (W4).
  6. A scope and limitations paragraph (W3).

We hope these results address the concerns behind the soundness rating. The baseline the reviewer suggested falls well short of OLGA, the method holds up under grouping errors, it doesn't depend on a single distilling model, and its cost is laid out in concrete numbers. We'd be grateful if the reviewer could take the new experiments into account and, if they resolve the concerns, consider raising the score. We're happy to answer any further questions during the discussion.

Thanks for the author's careful response! some of my concerns are resolved (corset definition and group partition ablation), however the main concerns on generalizability above simple text classification tasks have not been resolved.

I agree with reviewer yEf5 that this paper still needs improvement and does not reach an accept score. But i am willing to raise the score from strong rejection to weak rejection.

Reviewer_SbwBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OLGA for compressing unlabeled text streams into a small rehearsal memory. Its unlabeled objective is constructed from the next-word predictions of a frozen language model, creating a group-specific soft objective. Tokens are sorted by a probability ratio, and the selected token defines a soft cross-entropy objective. The method maintains the online memory through cascaded redistillation. Experiments with phi-1.5 show SOTA performance of the proposed method.

Critical strengths
  1. Useful unlabeled objective. The group distribution objective provides a practical way to extend text gradient matching methods without supervised labels. Ablation experiments also demonstrate that the proposed objective outperforms the evaluated pseudo-token and self-information objectives.

  2. Strong performance under compression. Using only five synthetic examples, OLGA significantly outperforms the evaluated selection baseline on multiple datasets, improving the average online stream-AUC by 6.6 percentage points compared to k-center.

  3. Practical accuracy-efficiency trade-off. Combining selection with distillation reduces offline distillation runtime by approximately 30% while sacrificing only 0.9 percentage points in average unlabeled accuracy.

Critical weaknesses
  1. The unlabeled protocol is unclear. Grouping is an essential novelty of the method, but the paper does not explain how the grouping for each dataset is constructed. Moreover, the accessibility of labels throughout the process is also not clearly explained. This makes the core claim of unsupervised training unclear.

  2. Evidence for continuous learning is limited. The flow order, distribution shift, and model weights inheritance are not adequately explained. Moreover, the ability of cascaded unions to progressively reduce the weights of older batches while preserving complete historical information remains unclear.

  3. Theoretical guarantees are problematic. Theorem 3 defines the absolute difference between the true and synthetic loss values ​​based on gradient proximity, but its assumption does not control for additive loss bias. For example, losses Lreal(θ)=θ2/2L_{\mathrm{real}}(\theta)=\theta^{2}/2 and Lsyn(θ)=θ2/2+CL_{\mathrm{syn}}(\theta)=\theta^{2}/2+C have the same gradient and optimization trajectory, yet their loss difference can be arbitrarily large. The recursion in the proof does not establish the stated bound, which affects subsequent claims to this conclusion. Corollary 11 also assumes that the reference trajectory minimizes the current cumulative loss, which is not established by Theorem 10.

  4. Experimental reliability and breadth are insufficient. The evaluation uses only phi-1.5 and closely related classification datasets, making it difficult to support the generalization over more tasks and base models.

Questions for authors

How are groups constructed for each dataset? How does the accessibility of labels change over the process?

Can the authors explain my problems of the theory parts? (See weaknesses)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.