- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper works to address the problem of online learning in a constrained setting where we don't want to maintain the full data history, or update the model on a large dataset. The authors propose OLGA (Online Label-free Gradient Matching) as an algorithm that handles the above limitations by first generating synthetic data by matching gradients with the real data, then uses the model to make a best guess on synthetic data labels, and selects a subset through a distillation/core selection step. The authors validate their method in both offline and online settings using 5 classification datasets, comparing against an existing method (GradMM) and a series of ablations on their proposed method.
Critical strengths
- I appreciate that the ablations and analysis of OLGA are fairly thorough
Critical weaknesses
- I'm not an expert, but some of the experimental setup appears to be quite out of date (nearest comparison baseline is from 2018, most recent dataset is also from 2018). There has been a lot of advancement in NLP since then, and it would be much more compelling to see methods/data from more recent times
Questions for authors
- The text in the figures is REALLY small, I struggled to read them. Is it possible to increase the font size?
- I'm concerned about the lack of modern comparisons here. I can appreciate that things move very quickly right now in NLP, but datasets and methods from 2018 are no longer contemporary works. Can you search to see if there are any more recent classification datasets? Also, are there more modern methods of offline, unsupervised learning than k-center greedy (Sener & Savarese, 2018)?
- What is the "selection" method in Table 2? Does that mean, select to k=50 and then train on all 50? The definition of that method should be mentioned somewhere
- Lines 370-371 just have a strange phrasing that should be improved: "Table 2 compares the accuracy of the three routes in the offline setting, where the whole compression happens once over a static pool, and Table 2 compares what each route costs in time."
Response to Reviewer uvUk (1/3)
Thank you for the careful review and the concrete questions. They pushed us to run three new experiments, and here's a quick summary before we go through each point:
For efficiency (W1/Q1), we restate the runtimes from Table 2 below.
W1 / Q1: Efficiency compared with the baselines and with standard training
We don't claim distillation is free, and the paper reports its cost against every method we compare in Table 2. Here are those numbers again. Each is the wall-clock time of one offline run on SST-2, going from 1,000 examples down to 5, with the average accuracy over the five datasets in parentheses.
A few things stand out:
As for standard training, the reviewer is right that if you train a single model once, distilling first costs more than just training on the data. But that isn't where distillation pays off. Its savings come from three places, which the paper motivates in the introduction, the related work, and Sec. 3.3.
Our contribution is to the distillation algorithm itself, namely removing the need for labels and handling a data stream. That's why we measure how accurate the distilled set is. GradMM has already shown that sets from this family of methods can be reused across models.
Paper change: right now these points are spread over the introduction, the related work, Sec. 3.3 and Table 2. We'll pull them into one paragraph that compares distillation directly with standard training, so they're easy to follow for readers outside the distillation literature.