- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when grouped score function gradient estimates become unreliable under discrete downstream evaluation. It identifies two failure modes, boundary starvation and tangential noise, and introduces measures estimated from pilot samples to determine whether a fixed query budget can reveal informative loss differences and accurately recover the smoothed gradient. The proposed analysis and diagnostic are evaluated on controlled shortest path problems, SEMO Energy Knapsack, and UCI HAR feature selection.
Critical strengths
The paper provides a rigorous geometric characterization of two distinct sources of finite budget failure in score function learning. Separating the scarcity of informative loss comparisons from the variance contributed by irrelevant directions offers a useful framework for understanding when an unbiased estimator may still be practically unreliable.
The empirical evaluation also uses independent pilot, reference, and future samples across controlled and application based settings. This prospective design, together with explicit abstention when the pilot evidence is insufficient, makes the reported diagnostic results more credible.
Critical weaknesses
-
High diagnostic overhead. The primary experiments use 256 pilot groups, corresponding to 2048 evaluator calls for SEMO and 2048 or 4096 calls for HAR. These costs are comparable to or larger than the future budgets being diagnosed, which range from 128 to 4096 calls. This overhead may eliminate the claimed query savings (Section 5.2, Appendix D.4 and D.5, Tables 26 and 38).
-
Missing end to end validation. The experiments measure how well the pilot predicts future gradient error and diagnostic states, but do not show that using the diagnosis improves training convergence or downstream task performance under an equal total query budget (Section 5 and Section 6).
-
Restricted operational scope. The diagnosis applies only to a fixed sampling configuration and assumes Gaussian perturbations with deterministic piecewise constant feedback. The authors also state that reusing a pilot after updating the sampling distribution is not covered, limiting the applicability of the method to iterative learning (Section 2 and Section 6).
Questions for authors
-
When reporting reductions in the selected future budget, how are the pilot evaluations accounted for? Can pilot samples be incorporated into the final gradient estimate or reused after an update, and if not, how does the method achieve net query savings during iterative learning?
-
Under an equal total query budget that includes pilot evaluations, does using the proposed diagnosis improve training convergence or downstream task performance relative to a fixed budget strategy?
-
How well do the nominal confidence levels match empirical coverage across repeated pilot draws at the same configuration, particularly in rare activation regimes?
6 comments
Response to Reviewer sHVS (2/2)
2. Does diagnosis improve convergence or downstream performance under equal total budgets including pilot costs?
Response. The current experiments do not establish this benefit for the pilot-based budget strategy. We revised the practical-benefit interpretation and investigated a key missing link: does improved gradient estimation necessarily translate into optimization progress?
The revision provides new theory and direct experiments. In the BPM, under shared perturbations, a fixed smoothing scale, and common scalar step sizes, full and normal-projected estimates can have different mean-squared errors yet produce identical ordinary-SGD objective trajectories. Norm clipping instead couples tangential noise to the useful normal step, allowing estimation improvements to affect objective decrease.
We added paired experiments with equal query budgets: 32 evaluations per update, 120 updates, and 3,840 evaluations per trajectory. Each setting uses 256 independent trajectory pairs, with dimension and clipping-threshold sweeps. Projection uses the known BPM normal to isolate tangential noise.
The findings match the analysis:
- Under ordinary SGD, full and projected updates have identical normal coordinates and objectives at every recorded step.
- At and clipping threshold 0.1, a separate same-state comparison shows that the full update retains only about 28.3% of the projected normal step. In the equal-budget trajectory experiment, final mean losses are 0.0111570 and 0.00233768, respectively.
These results establish how estimation noise affects optimization; they do not establish an end-to-end advantage for pilot-based budget selection. They explain why accurate prediction of future gradient error alone cannot imply better training: the effect also depends on noise direction and the update rule.
The revision thus strengthens the mechanism analysis you recognized with new budget and trajectory results, aligning its central contributions with direct evidence.
Changes in the manuscript:
- Sections 4.1–4.2: adds joint query budgets and cost-dominance regimes.
- Section 4.3 and Appendix B.6: adds ordinary-SGD trajectory equivalence and norm-clipping analysis.
- Section 5.3, Figure 4, and Appendix C.4: adds equal-budget trajectories and same-state update controls.
- Introduction, experiment organization, and Conclusion: distinguishes mechanism validation from pilot-policy benefits.
3. How does empirical coverage compare with nominal levels, especially under rare activation?
Response. We added repeated independent pilots at fixed configurations with known population quantities. The experiment covers weak-signal, rare-activation, and near-threshold settings: 21 configurations, each with 500 independent pilots and 799 multiplier draws per pilot.
Coverage varies with configuration and sample size, with undercoverage in some settings:
- At the gradient-resolvability threshold, coverage increases from 85.6% at to 94.2% at .
- A two-interface weak-signal configuration still has 91.0% coverage at .
- Under rare activation, unconditional coverage at is 100%, but interval availability is only 1.2%. Unavailable inference returns the full parameter space, so this is not evidence of precise, reliable diagnosis. At , availability reaches 71.6%, unconditional coverage is 94.2%, and coverage conditional on availability is 91.9%.
We therefore report both coverage and interval availability, separately from diagnostic output rates and label agreement.
The added experiment evaluates two-sided simultaneous moment bounds, whereas the application diagnostics use direction-specific margin bounds. We distinguish these constructions and do not interpret the joint-interval experiment as direct coverage validation of the application rule. We also correct confidence statements to report the nominal levels actually used and retain the undercoverage results.
These experiments reveal a practical difficulty beyond the population budget relation: rare feedback increases gradient-recovery costs and can also prevent a finite pilot from providing informative statistical assessments.
Changes in the manuscript:
- Appendix C.7, “Repeated-pilot interval coverage,” and Table 9: adds repeated-pilot experiments, reporting coverage, availability, and conditional coverage under rare activation.
- Appendix C.7, “Moment estimates and operational labels,” and Table 7: clarifies the application construction and distinguishes nominal levels, output rates, and label agreement.
We hope these revisions address your concerns and clarify the paper's contribution to understanding query costs and optimization mechanisms in discrete-feedback SFL.
Thank you for the substantial revision, which addresses much of my original concern. I share Reviewer PZ7N’s remaining concern about directly validating the application diagnostic, and I am still unclear whether the pilot procedure retains an actionable methodological role after being repositioned as supplementary. Given the extensive reframing and new results, I am keeping my rating unchanged for now but remain open to revising it.
Response to Reviewer sHVS (1/2)
Thank you for your careful review and for recognizing the geometric analysis and independent-sampling design. Your concerns about pilot overhead, end-to-end benefits, and reuse prompted us to distinguish how many queries gradient recovery requires from whether diagnosing that requirement provides practical benefits. We strengthen the former through budget and optimization analysis and reassess the latter through total-cost accounting and repeated-pilot coverage experiments.
Revisions primarily addressing your comments are highlighted in blue in the revised manuscript. Locations of shared revisions are also listed below. All references refer to the revised manuscript.
1. How are pilot costs counted? Can pilot samples enter the final gradient estimate or be reused after an update?
Response. Pilot queries must count toward the total; reducing the subsequent budget alone does not establish net savings. The revision reports pilot, subsequent-update, and total costs:
These results do not establish net savings for a single update. We correct this interpretation and use the pilot study to examine what diagnostic information is obtainable and how much obtaining it costs.
Regarding sample use, the current experiments keep pilot, reference, and future samples independent; pilot samples are not incorporated into the evaluated future gradient estimate. Under a fixed distribution, pilot samples also contain gradient information. However, incorporating the same samples used to select the budget would change the current independence structure and require analysis of the resulting estimation error and statistical properties. We have not counted such pooling as a demonstrated source of savings.
Regarding reuse, two cases must be distinguished:
Diagnostic value and cost-effectiveness therefore require separate evaluation. The revision retains evidence relating pilot predictions to independent future gradient quality while explicitly reporting total costs and the fixed-state scope.
Changes in the manuscript:
Questions 2 and 3 are addressed in the next comment.