- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to choose among four existing data-selection objectives during training. Building on OnlineSubmod, DriftBandit replaces historical reward estimates with gradient–validation alignment computed at the current model state. It selects an objective once per epoch and uses that objective to construct subsets throughout the epoch. A Page–Hinkley detector temporarily increases exploration when validation accuracy deteriorates. Experiments report consistent improvements over OnlineSubmod on image-classification tasks.
Critical strengths
None
Critical weaknesses
- The incremental contribution is limited, and the practical reason for using this procedure remains unclear. The four selection objectives already exist, and OnlineSubmod already considers dynamically choosing among them. The main change is to choose using current gradient alignment rather than historical rewards, with an additional exploration adjustment after validation deterioration. This is a fairly direct modification. A simple modification can still be valuable, but here I would need either a compelling practical benefit or a clearer new finding to justify the contribution. The practical benefit is not established by the comparison to other selection methods alone. In Table 4, full-information DriftBandit reaches 74.81% accuracy on CIFAR-100 and takes 50.1 seconds per epoch, while ordinary full-data training reaches 76.04% and takes 45.2 seconds. The two-arm variant is faster at 41.5 seconds, but its accuracy is 74.60%. These results leave open why one should use the proposed selection procedure instead of ordinary training, especially under an equal wall-clock budget.
- The meaning of the data budget also matters. Selecting 30% of each batch for updates does not mean that only 30% of the data need to be collected or labeled. The method still uses labeled candidate data to construct and score subsets. I therefore do not see how the reported update-fraction experiments establish the claimed annotation-efficiency benefit. The paper needs a clearer account of the actual resource constraint this procedure addresses.
- The paper does not sufficiently establish a transferable explanation for the gains. The potentially interesting question is whether different selection objectives have different useful operating regimes as a model learns. If this relationship can be characterized and transferred, it could be a more substantial contribution than the selection rule itself. The paper describes an early Graph Cut, middle Facility Location, and late Log Determinant pattern. However, the supporting stage analysis is concentrated on CIFAR-100 at a 30% update fraction. Moreover, the “oracle” objective is defined using gradient alignment, rather than by measuring which objective actually produces better subsequent training. Agreement with this oracle shows agreement with the scoring criterion. it does not establish that each objective is genuinely preferable in the corresponding stage.
- I acknowledge the results on TinyImageNet and medical-imaging datasets, as well as the five-backbone comparison. These provide evidence beyond a single experiment. However, the key fixed-objective comparisons, ablations, and stage analyses remain concentrated on CIFAR-100. Final-accuracy improvements elsewhere do not show that the same mechanism explains those improvements. The medical experiments also involve fine-tuning pretrained models, which is a different setting from learning representations from scratch. I am therefore unsure whether the paper has identified a general property of training, a broadly useful adaptive heuristic with task-specific behavior, or a benefit limited to the evaluated selection setups. The current analysis does not distinguish these possibilities.
Questions for authors
-
What practical constraint makes a fixed per-batch update fraction the appropriate budget here? Under equal total training time, how does DriftBandit compare with ordinary full-data training? Please also clarify how the method reduces annotation requirements when labeled candidate examples are needed for selection.
-
Does the reported stage-dependent ordering reflect actual training utility? For example, starting from the same early, middle, and late checkpoints, does continuing with each objective for an equal budget reproduce the ordering suggested by gradient alignment?
-
How much of the stage pattern transfers across datasets and architectures? A useful comparison would be a predetermined switching schedule derived from one setting and evaluated in another. This would help distinguish a transferable stage pattern from the benefit of making task-specific online decisions.