- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies stochastic optimization with a small amount of ground-truth supervision and additional labels from a fixed ML predictor. Starting from a prediction-powered gradient estimator, it proposes scalar and coordinate-wise Follow-the-Leader updates that minimize historical squared estimator norms, allowing correction weights outside [0,1]. The construction is also extended to stochastic zeroth-order optimization using a shared Gaussian perturbation. The analysis separates potential variance reduction from the cost of learning the correction weights. Experiments cover linear regression, tabular and frozen-feature image classification, and RoBERTa-large sentiment classification.
Critical strengths
-
The proposed tuning rules are simple and well motivated: minimizing the historical quadratic objective yields closed-form updates without an additional learning rate for weight tuning. The analysis connects the potential variance reduction to the correlation between true-label and ML-label gradients, providing useful insight into when the correction can help.
-
The extension to zeroth-order optimization is a useful contribution, supported by convergence analysis and experiments across several tasks. The repeated runs, batch-size comparisons, and tuning trajectories also provide informative evidence about the stability trade-off between scalar and coordinatewise tuning.
Critical weaknesses
-
Can the authors provide a time complexity analysis of the proposed methods relative to standard SGD and zeroth-order optimization, accounting for the labeled and unlabeled batch computations and the scalar/vector tuning updates. This would clarify the trade-off between the iteration savings and the additional computation per iteration.
-
Have the authors conducted ablations that separately evaluate replacing the online tuning rule in PP-SSL with FTL and allowing weights outside [0,1]? For example, comparing constrained and unconstrained FTL under the same student optimizer would help isolate the effect of the weight range. I wonder how these two changes individually contribute to the final results.
Questions for authors
Please refer to the Weaknesses above.
We thank the reviewer for the positive evaluation and thoughtful comments.
Here, n is the size of the mini-batch of data points with true labels, while N is the size of the batch of samples with ML labels. We note that since our work concerns leveraging ML-labeled data, computing stochastic gradients from the ML-labeled data follows naturally from our problem setup.
We thank the reviewer again for the constructive comments, which helped us clarify the computational costs of our methods and our experimental setups.