acceptodds
Under review as a conference paper at ICLR 2027

Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization

Abstract

Training a machine learning (ML) model is an optimization process that typically requires a set of examples with ground-truth labels beforehand. Obtaining these ground truths usually requires laborious human effort and can be costly. The scalability of human labeling becomes a concern when there are a large number of data points that need to be labeled. To address this practical issue, we consider an optimization setting where a subset of samples has ground-truth labels, while the remaining data points were labeled by an ML predictor, which could potentially assign incorrect labels to those points. We propose both first-order and zeroth-order stochastic optimization methods for training on datasets consisting of a mix of true-labeled and ML-labeled samples. We further conduct comprehensive empirical studies from training a linear regression model to fine-tuning a large language model (LLM) to showcase the effectiveness of the proposed algorithms for training with ML-labeled data.

Primary area:
optimization
Keywords:
stochastic optimization, zeroth-order optimization, variance reduction, online learning, prediction-powered gradients, semi-supervised learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_p8C1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies stochastic optimization with a small amount of ground-truth supervision and additional labels from a fixed ML predictor. Starting from a prediction-powered gradient estimator, it proposes scalar and coordinate-wise Follow-the-Leader updates that minimize historical squared estimator norms, allowing correction weights outside [0,1]. The construction is also extended to stochastic zeroth-order optimization using a shared Gaussian perturbation. The analysis separates potential variance reduction from the cost of learning the correction weights. Experiments cover linear regression, tabular and frozen-feature image classification, and RoBERTa-large sentiment classification.

Critical strengths
  • The proposed tuning rules are simple and well motivated: minimizing the historical quadratic objective yields closed-form updates without an additional learning rate for weight tuning. The analysis connects the potential variance reduction to the correlation between true-label and ML-label gradients, providing useful insight into when the correction can help.

  • The extension to zeroth-order optimization is a useful contribution, supported by convergence analysis and experiments across several tasks. The repeated runs, batch-size comparisons, and tuning trajectories also provide informative evidence about the stability trade-off between scalar and coordinatewise tuning.

Critical weaknesses
  • Can the authors provide a time complexity analysis of the proposed methods relative to standard SGD and zeroth-order optimization, accounting for the labeled and unlabeled batch computations and the scalar/vector tuning updates. This would clarify the trade-off between the iteration savings and the additional computation per iteration.

  • Have the authors conducted ablations that separately evaluate replacing the online tuning rule in PP-SSL with FTL and allowing weights outside [0,1]? For example, comparing constrained and unconstrained FTL under the same student optimizer would help isolate the effect of the weight range. I wonder how these two changes individually contribute to the final results.

Questions for authors

Please refer to the Weaknesses above.

1 comment

We thank the reviewer for the positive evaluation and thoughtful comments.

  • Per-iteration complexity: To clarify the computational requirements, we summarize the per-iteration costs of SGD and our four proposed methods in the following table. In all cases, the ML-labeler's predictions can be precomputed once before training and cached. The first-order methods require two backward passes per iteration: one to compute the standard labeled gradient and another to compute the difference between the ML-labeled gradients on unlabeled and labeled examples. The zeroth-order methods use two-point function evaluations without backpropagation. Both scalar and vector FTL tuning admit closed-form updates involving only O(d)O(d) elementary arithmetic operations, where dd denotes the number of trainable parameters. | Method | Forward evaluations | Backward passes | Tuning cost | |:-------|:-------------------:|:---------------:|:-----------:| | SGD | nn | 1 | — | | FO Scalar FTL | n+Nn+N | 2 | O(d)O(d) | | FO Vector FTL | n+Nn+N | 2 | O(d)O(d) | | ZO | 2n2n | 0 | — | | ZO Scalar FTL | 2(n+N)2(n+N) | 0 | O(d)O(d) | | ZO Vector FTL | 2(n+N)2(n+N) | 0 | O(d)O(d) |

Here, nn is the size of the mini-batch of data points with true labels, while NN is the size of the batch of samples with ML labels. We note that since our work concerns leveraging ML-labeled data, computing stochastic gradients from the ML-labeled data follows naturally from our problem setup.

  • Effect of tuning scheme: To ensure a fair comparison and isolate the effect of the power-tuning schemes, we use the same type of learning-rate schedule across all methods: a constant learning rate in most experiments and a linear learning-rate decay schedule for the first-order RoBERTa-large experiments. Specifically, Shoham et al. propose AdaGrad-Norm with AdaGrad-based online power tuning, whereas our first-order methods combine SGD with FTL-based tuning. In our experiments, we therefore implement the power-tuning procedure of Shoham et al. without AdaGrad-Norm learning-rate adaptation, using the same learning-rate scheduling strategy as the other methods. We note that Shoham et al. also consider constant learning rates in the empirical evaluation of their algorithm.

We thank the reviewer again for the constructive comments, which helped us clarify the computational costs of our methods and our experimental setups.

Reviewer_V8RqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies stochastic optimization with a combination of ground-truth-labeled and machine-labeled data. The authors propose first-order and zeroth-order optimization methods that incorporate machine-generated labels through a prediction-powered gradient estimator. Two online tuning strategies, namely scalar and vector Follow-the-Leader (FTL), are developed to control the contribution of machine-labeled data. The paper further establishes convergence guarantees and evaluates the proposed methods on regression, image classification, and text classification tasks.

Critical strengths
  1. The paper addresses a practically relevant problem: how to leverage inexpensive machine-generated labels to improve the efficiency of model training.
  2. Unlike conventional pseudo-labeling approaches, the proposed framework uses machine-generated labels to construct an unbiased gradient estimator rather than directly treating them as additional supervisory targets. This provides a meaningful optimization perspective on learning with imperfect labels.
  3. The extension to zeroth-order stochastic optimization, together with the corresponding convergence analysis, is potentially valuable, particularly for memory-constrained model fine-tuning.
Critical weaknesses
  1. The novelty over existing semi-supervised learning and prediction-powered optimization methods is not sufficiently convincing. The problem setting is closely related to conventional semi-supervised learning, where a small amount of ground-truth-labeled data and a large amount of pseudo-labeled data are jointly utilized for model training. The authors distinguish their approach from conventional pseudo-labeling methods by using machine-generated labels for variance reduction rather than as direct supervisory targets. While this distinction is theoretically meaningful, the underlying prediction-powered gradient estimator and the idea of dynamically adjusting its contribution have already been introduced by Shoham et al. (2025). Consequently, the main methodological novelty appears to lie in the specific FTL-based tuning strategies, coordinatewise weighting, and the extension to zeroth-order optimization. The paper should better clarify the conceptual and technical advances of these contributions beyond the existing prediction-powered semi-supervised learning framework.
  2. The experimental comparisons with conventional semi-supervised learning methods are insufficient. Although the paper discusses conventional SSL methods in the related work, the experimental evaluation does not include representative pseudo-labeling or knowledge distillation baselines. This omission makes it difficult to assess whether the proposed optimization framework provides practical advantages over more conventional ways of exploiting machine-generated labels. I recommend including comparisons with simple pseudo-labeling, supervised training with an additional pseudo-label loss, and knowledge distillation under matched data and computational budgets.
  3. The claimed optimization acceleration is not adequately supported by the experimental evaluation. The paper emphasizes faster stochastic optimization, but the main experiments primarily report test accuracy against training epochs or optimization steps. Since the proposed methods require additional gradient or function-value evaluations on machine-labeled data, fewer optimization steps do not necessarily imply lower computational costs. A more convincing evaluation should report wall-clock training time, computational overhead, and the cost required to reach a given training loss or predictive performance.
  4. The influence of machine-label quality is insufficiently investigated. The theoretical motivation relies on the relationship between gradients computed from true labels and machine-generated labels. However, the experiments largely use fixed machine-labeling models without systematically varying their predictive quality. Experiments involving machine-labelers with different accuracy levels, calibration properties, and systematic errors would help establish when the proposed variance reduction mechanism is effective.
Questions for authors

Please see the Weaknesses.

1 comment

We thank the reviewer for the careful evaluation and constructive feedback.

  • Comparison with Shoham et al.: While our work is inspired by Shoham et al., as we explicitly acknowledged in our introduction, our proposed algorithms differ from their approach and introduce several extensions. Specifically, our first-order method combines SGD with an FTL-based power-tuning scheme, whereas Shoham et al. employ AdaGrad-Norm with an AdaGrad-based online tuning procedure. Furthermore, we introduce vector-valued tuning and extend the framework to zeroth-order stochastic optimization, resulting in four algorithmic variants with corresponding theoretical guarantees. Additionally, while Shoham et al. primarily evaluate the predictive performance of the trained models, our work focuses on characterizing the potential acceleration in the optimization process. In particular, our convergence guarantees are directly comparable to those of standard SGD and the gradient-free methods and precisely quantify the potential improvement through variance reduction.

  • Comparison with SSL methods: As discussed in our paper, our proposed methods use ML-labels as auxiliary information for constructing a variance-reduced stochastic gradient estimator, whereas classical SSL methods use them as direct supervisory targets in the training objective. Our work also provides explicit convergence guarantees for nonconvex optimization that quantify the potential improvement over standard SGD due to variance reduction. We refer the reviewer to our response to Reviewer Dbbd for a more detailed discussion of this distinction and the empirical evidence.

  • Acceleration: We would like to clarify that we use the term acceleration in an optimization sense, referring to improvement in the convergence rate or iteration complexity, without necessarily implying an improvement in computational cost to achieve a prescribed accuracy. Our methods can incur additional computational overhead at each iteration because they use gradient information from unlabeled data. To assess this overhead in practice, we compare test accuracy under equal wall-clock budgets in our RoBERTa-large experiments. We refer the reviewer to our response to Reviewer p8C1 for a table of per-iteration costs and to our response to Reviewer Dbbd for the detailed wall-clock results.

  • ML-labeler quality: We thank the reviewer for this suggestion. Corollary 1 of our paper already provides a theoretical characterization of how the benefits of incorporating ML-labeled data depend on the quality of the ML-labeler. The variance-reduction term is expressed in terms of the covariance between the stochastic gradients computed using true labels and those computed using ML-labels, relative to the variance of the ML-labeled gradient correction. This explicitly quantifies how these quantities influence the potential improvement in convergence.
    To complement this theoretical characterization, we have also added a synthetic linear regression experiment to empirically investigate how the quality of the ML-labeler affects the performance of our methods. Specifically, we vary the magnitude of the prediction error and consider different combinations of systematic model misspecification and additive noise. We then evaluate how these variations affect the performance of our proposed methods relative to SGD. Furthermore, we compute the gradient covariance and variance along the optimization trajectories to empirically quantify the potential variance reduction predicted by Corollary 1. This allows us to investigate how ML-labeler quality influences both the theoretical variance-reduction benefit and the observed optimization performance. The experimental setup and detailed results are provided in Appendix E.5.

We again thank the reviewer for the useful feedback, which help us to further contextualize our results and make them clearer. The reviewer's suggestion of additional empirical investigations into ML-labeler quality has also helped strengthen our work and is reflected in the updated version of our paper.

Reviewer_EpgoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies stochastic optimization with a small amount of ground-truth-labeled data and additional ML-labeled data. Building on prediction-powered gradient estimation, it proposes scalar and coordinatewise weighting schemes based on Follow-the-Leader and extends them to zeroth-order optimization. The authors derive nonconvex convergence bounds that characterize potential variance reduction and evaluate the methods on regression, image classification, and language model fine-tuning tasks.

Critical strengths
  1. The paper addresses a practically relevant problem.

  2. The paper studies both first- and zeroth-order optimization with theoretical analysis.

  3. The paper includes experiments across several model classes and tasks.

Critical weaknesses
  1. The stated convergence rates depend on quantities whose growth is not controlled.

    • The authors describe the additional tuning term as O(log⁡T/T3/2)O(\log T/T^{3/2}) and therefore negligible relative to the T−1/2T^{-1/2} terms. However, C0,C1C_0,C_1 in Theorem 1 and D0,D1D_0,D_1 in Theorem 2 are defined through maxima over the first TT iterations and thus depend on TT. The analysis does not establish bounds on their growth that justify treating them as constants.

    • More fundamentally, the stated assumptions do not ensure that these quantities are finite. For example, the instantaneous optimal weight contains the ratio λt∗=−⟨gt,dt⟩/∣∣dt∣∣2\lambda_t^*=-\langle g_t,d_t\rangle/||d_t||^2 (Eq. (18)), which can have divergent moments when the denominator approaches zero.

  2. The experimental evidence does not sufficiently substantiate the claimed optimization acceleration.

    • The paper motivates its methods as accelerating optimization, but the experiments primarily report test accuracy or test MSE against epochs or optimization steps. These metrics do not directly establish faster convergence of the training objective or the stationarity measure analyzed in the theory.

    • Each iteration processes additional ML-labeled samples, yet no wall-clock time or matched-compute comparison is reported. The gains over PP-SSL are also modest in several configurations; for example, many CIFAR-10 configurations improve by approximately 0.1–0.2 percentage points.

    • The image experiments train only a linear classifier on cached features from a frozen ImageNet-pretrained ResNet-50, providing limited evidence for general nonconvex deep-network optimization. The text experiments do involve RoBERTa-large fine-tuning, but the trainable parameter subset is not clearly specified. Together, these limitations make it difficult to assess the scope and practical significance of the claimed acceleration.

Questions for authors
  1. Theoretical clarification: What conditions ensure that C0,C1,D0,D1C_0,C_1,D_0,D_1 are finite and grow sufficiently slowly with TT to justify the stated rates? In particular, how is the potential instability caused by near-zero denominators controlled?

  2. Implementation clarification: In the text classification experiments, which RoBERTa-large parameters are updated? Is this full-parameter fine-tuning, classifier-only training, or a parameter-efficient approach?

2 comments

We appreciate the reviewer’s thorough reading of our paper and the thoughtful feedback provided.

  • Constants: We thank the reviewer for the comment.
    To address the reviewer's comment, we have refined our convergence analysis. Under the additional assumption that the stochastic gradients are bounded (Assumption 4), the constant in our convergence-rate bound now depends on the expected maximum squared magnitude of the tuning parameters, E[max⁡1≤t≤T+1∣λt∣2]\mathbb{E}\left[\max_{1 \leq t \leq T+1} |\lambda_t|^2\right].
    We note that in first-order optimization, the proposed Follow-the-Leader (FTL) for scalar tuning chooses λt\lambda_t to minimize the cumulative loss:
\qquad F_{t-1}(\lambda) := \sum_{s=1}^{t-1} \phi_s(\lambda), \qquad \phi_s(\lambda) := \\| \mathbf{g}_s + \lambda \mathbf{d}_s\\|^2.$$ Each $\phi_s$ is a one-dimensional quadratic function that is strongly convex whenever $\\|\mathbf{d}_s\\| > 0$. Furthermore, whenever the denominator is positive, we have $$|\lambda_t| = \frac{ \lvert \sum_{s=1}^{t-1} \langle \mathbf{g}_s, \mathbf{d}_s \rangle \rvert }{ \sum_{s=1}^{t-1} \\| \mathbf{d}_s \\|^2 },$$ c.f. (2) in the paper. The sum in the denominator, i.e., $\sum_{s=1}^{t-1} \\|\mathbf{d}_s\\|^2$ is nondecreasing in $t$ and increases strictly whenever a new round contributes a nonzero term. We also recall that when $\sum_{s=1}^{t-1} \\|\mathbf{d}_s\\|^2 = 0$, the FTL objective is constant in $\lambda$, in which case we set $\lambda_t$ to zero. Thus, $|\lambda_t|$ go towads a small value whenever the absolute cumulative inner product grows more slowly than the cumulative squared norm. This condition still allows for fluctuations between rounds. We also obtain an analogous result for zeroth-order optimization.\ Empirically, we observed that $|\lambda_t|$ is relatively large in the early iterations and then gradually decreases, eventually settling near or fluctuating around a small value (see Figures 5, 7, 9, 11, 13, 15, 17, and19).\ We additionally propose an alternative online tuning scheme based on Follow-the-Regularized-Leader (FTRL) in Appendix D. For example, in first-order optimization with scalar tuning, FTRL with a quadratic regularizer yields the update $$\lambda_t = \underset{\lambda \in \mathbb{R}}{\arg\min} \left\\{ \sum_{s=1}^{t-1} \\|\mathbf{g}_s + \lambda \mathbf{d}_s\\|^2 + \epsilon\lambda^2 \right\\} = -\frac{ \sum_{s=1}^{t-1} \langle \mathbf{g}_s, \mathbf{d}_s \rangle }{\epsilon + \sum_{s=1}^{t-1} \\|\mathbf{d}_s\\|^2}.$$ Here, $\epsilon > 0$ keeps the denominator positive even when all previous corrections vanish. The other three variants follow the same regularized update, using zeroth-order estimators in the zeroth-order setting and minimizing the regularized loss separately for each coordinate in the vector setting. - **Acceleration**: We would like to clarify that, in the optimization literature, the term **acceleration** refers to an improvement in the convergence rate or iteration complexity, without necessarily implying an improvement in computational cost to achieve a prescribed accuracy. Since our methods use additional gradient information from unlabeled data, each iteration can require additional computations. To assess the practical impact of this overhead, we compare test accuracy at matched wall-clock budgets in the RoBERTa-large experiments. We refer the reviewer to our response to Reviewer p8C1 for the per-iteration cost table and to our response to Reviewer Dbbd for the detailed wall-clock time results.
  • CIFAR-10 Experiments: Our proposed Scalar FTL method consistently matches or outperforms PP-SSL across all ten batch-size configurations, achieving higher mean test accuracy in nine configurations, with improvements ranging from 0.08 to 0.26 percentage points (0.16 percentage points on average). Importantly, both methods have comparable per-iteration computational complexity, yet our Scalar FTL method consistently achieves comparable or better predictive performance, highlighting the benefits of the proposed FTL-based tuning scheme.

  • RoBERTa-large Experiments: In the RoBERTa-large experiments, we perform full-parameter fine-tuning, updating all model parameters, including the embedding layers, transformer blocks, and masked-language-model head. Thus, our experiments also evaluate the proposed methods in a large-scale, nonconvex deep-network optimization setting. We have explicitly clarified the trainable parameter configuration in the revised paper.

We hope that our clarifications of the experimental setups, the additional wall-clock comparisons, and the new FTRL convergence guarantees, which relax the boundedness assumptions on the tuning coefficients, address the reviewer’s concerns. We have updated our paper based on the discussion here accordingly. We again thank the reviewer for the constructive feedback, which has helped strengthen our contributions.

Reviewer_DbbdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about how to do training when there are a limited set of ground truth labels and a larger set of labels drawn from some machine learning predictor. In particular, it takes the approach of using the ground truth to produce a learning update and then using the machine learning labeled batch to correct for the noise in our initial learning update. This allows variance reduction without biasing towards the machine learning predictor. In particular, the paper suggests a way of choosing this correction rule and extends it to training without backpropagation.

Critical strengths

The correction weight is found in a relatively straightforward and principled way since the objective directly follows from the optimization problem and ensures that it remains conditionally unbiased. The extension of the zeroth-order method is also interesting and builds on the growing interest in moving away from purely gradient-based training.

Critical weaknesses

The core problem is that the experiments do not actually prove that there is a speed up as a result of this method. While the reported curves do show that test accuracy benefits measured against the number of steps, each step requires more data to be processed in order to generate the gradient and so being able to produce better performance after the same number of steps or equal performance after fewer steps does not imply that when considering all the computation that is done, the target result can actually be achieved faster. In particular, it would be useful to have a simple baseline to compare against that trains using the true labels plus some standard techniques from the weak supervision literature.

Questions for authors

Does the method actually reduce the optimization work? How does it compare with a simpler use of the labels that were generated by the machine learning model as opposed to the ground truth? From a theoretical perspective, are there any conditions required to ensure that the constants grow slowly enough to ensure the bound holds?

2 comments

We appreciate the reviewer’s positive assessment of our work and helpful suggestions.

  • Optimization speedup: We would like to clarify that in the optimization literature acceleration refers to an improved convergence rate or iteration complexity, which does not necessarily imply a corresponding reduction in computational cost to achieve a prescribed accuracy. Indeed, our methods leverage additional gradient information from unlabeled data, which may introduce some additional computations. Similar computational overhead is also required in the closely-related method of Shoham et al. [1]. More broadly, additional computations are common in SSL methods that leverage unlabeled data, often involving extra forward and backward passes, see e.g., [2], [3].
    To further clarify the computational requirements, we refer the reviewer to our response to Reviewer p8C1, where we summarize the per-iteration costs of SGD and our four proposed methods in a table.
    To assess computational efficiency, we additionally conduct a wall-clock time comparison in the RoBERTa-large experiment, evaluating the methods at common time budgets corresponding to the time required for SGD (or ZO method, respectively) to complete 25%, 50%, 75%, and 100% of the prescribed TT iterations. Because test accuracy was recorded every 25 iterations, we linearly interpolate between adjacent checkpoints to estimate accuracy at each time budget. The alignment is performed separately for each seed using its measured optimization time, excluding evaluation overhead, and the resulting accuracies are then averaged across the five seeds.
    SST-2, first-order methods — Test accuracy (%)
SGD progressSGDScalar FTLVector FTLPP-SSL
25%92.3291.9191.2090.86
50%92.3692.5292.0892.01
75%92.4192.6392.5092.28
100%92.3492.8292.5192.56

SST-5, first-order methods — Test accuracy (%)

SGD progressSGDScalar FTLVector FTLPP-SSL
25%51.5449.2747.5249.44
50%50.4850.8850.4451.40
75%51.0251.1351.0451.47
100%50.9651.3751.7451.09

SST-2, zeroth-order methods — Test accuracy (%)

ZO progressZOZO Scalar FTLZO Vector FTL
25%90.5787.3986.00
50%91.1089.8887.60
75%91.5691.0889.21
100%91.8191.6290.53

SST-5, zeroth-order methods — Test accuracy (%)

ZO progressZOZO Scalar FTLZO Vector FTL
25%47.4045.8645.45
50%48.7848.2446.55
75%49.1448.5147.55
100%49.3049.6548.66

As the tables above show, Scalar FTL achieves higher mean test accuracy than the corresponding baseline in three of the four settings at the final wall-clock budget. In the first-order experiments, Scalar FTL outperforms SGD by 0.48 percentage points on SST-2 and 0.41 percentage points on SST-5. Vector FTL also outperforms SGD on both datasets. In the zeroth-order experiments, ZO Scalar FTL exceeds the ZO baseline by 0.35 percentage points on SST-5 and finishes 0.19 percentage points below it on SST-2 (91.62% versus 91.81%)

References:

[1] Noa Shoham, Ron Dorfman, Shalev Shaer, Kfir Y. Levy, Yaniv Romano (2025), Prediction-Powered Semi-Supervised Learning with Online Power Tuning

[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel (2019), MixMatch: A Holistic Approach to Semi-Supervised Learning

[3] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, Colin Raffel (2020), FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

  • Comparison with SSL methods: As discussed in our paper, our methods differ from classical SSL methods in that ML-labels are not used as direct supervisory targets in the training objective, but as auxiliary information for constructing a variance-reduced stochastic gradient estimator. Our approach is particularly useful when an external labeler makes systematic prediction errors but its gradients remain correlated with the true-label gradients. In this setting, a pseudo-labeling method that retains incorrect predictions as training targets can introduce bias into the training update. Some SSL filtering methods can mitigate this risk, but they can also discard predictions whose gradients would be useful for variance reduction. Our method uses ML-labels to construct a correction direction whose conditional mean is zero. The tuning parameter controls the weight of this correction, allowing the method to exploit informative gradient correlations. Additionally, while widely used SSL methods primarily establish their effectiveness empirically, our framework provides rigorous convergence guarantees for nonconvex optimization that are directly comparable to those of standard SGD, explicitly quantifying the improvement achieved through variance reduction.
    Furthermore, Shoham et al. [1] already compare their prediction-powered method against a baseline that directly uses ML-labeled data for training (denoted as "SSL") and demonstrate improved performance over this baseline. Since our experiments include direct comparisons with the method of Shoham et al. [1], and our proposed methods achieve better performance in several settings, these results also provide indirect evidence of their advantages over the conventional SSL baseline considered in [1].

  • Constants: We thank the reviewer for this comment. As detailed in our response to Reviewer Epgo, we have refined our convergence analysis under the assumption of bounded stochastic gradients. We also propose an alternative online tuning scheme based on Follow-the-Regularized-Leader (FTRL), whose quadratic regularizer keeps the denominator in the tuning update bounded away from zero even when the correction norms are small. We refer the reviewer to the same response for details.

We have included the new results and revised our paper based on the reviewer's comment in the updated version of our paper. We appreciate the constructive feedback from the reviewer, which has helped strengthen our work

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.