- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Dynamic Sketching (DySk), a dynamic weighted coreset method for efficient neural network training. DySk periodically constructs a compact representation of the full dataset using random Fourier features, selects representative samples through greedy residual matching, and optimizes their sampling weights to approximate the full-data feature distribution. The selection process requires only forward passes, avoiding expensive per-example gradient computations. The authors provide a conditional theoretical analysis connecting sketch approximation to gradient approximation. Experiments on CIFAR-10/100 demonstrate improved accuracy and efficiency over existing coreset methods at matched subset budgets, while additional ImageNet-1k and Transformer pretraining experiments suggest broader applicability.
Critical strengths
-
The proposed method combines kernel mean approximation with dynamic subset selection, preserving representation-space coverage without requiring per-example backward gradients. This provides a computationally motivated alternative to gradient-based coreset methods while adapting to evolving representations during training.
-
DySk consistently outperforms the evaluated coreset baselines across multiple subset budgets on CIFAR-10/100, with particularly substantial gains at small budgets. These results provide meaningful empirical support for the proposed selection strategy.
Critical weaknesses
-
Figure 1, which presents the main experimental results, reports accuracy against nominal coreset fractions rather than actual computational cost. Since DySk dynamically refreshes its subset during training, the overhead of repeated full-dataset forward passes and subset selection is essential to evaluating its efficiency. Although Table 17 provides end-to-end runtimes, it reveals less favorable comparisons at matched computational budgets: on CIFAR-10, Random 20% outperforms DySk 5% at nearly identical runtime (92.07% vs. 90.74%), while on CIFAR-100, Random 20% is both faster and slightly more accurate than DySk 10% (71.29% vs. 70.99%). A more rigorous accuracy–runtime comparison across different subset budgets should be presented in the main paper. Furthermore, complete training-time measurements are provided only for CIFAR, leaving the claimed efficiency benefits on ImageNet and LLM pretraining insufficiently validated.
-
On ImageNet-1k, DySk-Core applies the proposed sketch-based selection to only 256 examples, while the remaining subset is constructed through class-balanced uniform sampling and trained with uniform weights. Despite this substantial simplification, DySk-Core slightly outperforms DySk-Direct, which implements the full weighted selection procedure. This raises concerns about the proposed greedy selection and weighting mechanisms at scale. In particular, the experiments do not sufficiently isolate how much of the improvement comes from kernel sketch matching versus class-balanced random sampling. Additional controlled ablations are needed to establish the contribution of the proposed components.
-
The LLM experiment evaluates only a relatively small 286M-parameter Transformer, without reporting full-data training performance or end-to-end runtime. Moreover, several essential implementation details are missing, including the training corpus, total token count, Transformer layer used to extract representations, and aggregation of token-level hidden states into sequence-level features. The results are also reported from a single run per configuration. These omissions limit reproducibility and make it difficult to assess whether the proposed approach generalizes to realistic large-scale LLM pretraining.
-
The theoretical analysis relies on the assumption that full parameter gradients can be approximated by a linear transformation of random Fourier features derived from penultimate-layer representations. However, the paper provides limited intuition or empirical evidence supporting this relationship for deep neural networks, where gradients also depend on labels and the Jacobians of earlier layers. Without characterizing the gradient-prediction residual or directly measuring gradient approximation errors, it remains unclear whether the theoretical bound provides a meaningful explanation for the observed performance.
Questions for authors
-
Could the authors provide accuracy–runtime curves comparing DySk with baseline methods across different subset budgets, including all warm-up, forward-pass, and selection overhead? In particular, how does DySk compare against random selection at matched wall-clock budgets? Could the authors also report complete end-to-end training times for ImageNet and LLM experiments?
-
Since DySk-Core selects only 256 examples through sketch matching on ImageNet and fills the remaining subset using class-balanced uniform sampling, how much of its improvement is attributable to the proposed sketch-based selection? Could the authors compare against a class-balanced random baseline under the same training protocol? Additionally, why does DySk-Core consistently outperform the full DySk-Direct method?
-
What training corpus, total number of tokens, sequence length, and validation dataset are used in the LLM experiment? Which Transformer layer provides the feature representations, and how are token-level hidden states aggregated? Could the authors provide end-to-end training-time measurements to substantiate the efficiency and scalability claims?
-
What motivates the assumption that random Fourier features of penultimate-layer representations can linearly approximate full parameter gradients? Could the authors empirically measure the gradient approximation error and investigate whether reducing sketch mismatch actually leads to better gradient matching throughout training?