- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether conventional policy-gradient statistics, particularly means and covariances, are sufficient to determine the probability that a finite-batch gradient points in the wrong direction. It establishes theoretical non-identification results, characterizes the resulting ambiguity, and demonstrates its persistence in sequential reinforcement learning environments. The authors also propose a finite-sample reliability certificate based on confidence regions and moment-generating-function information. Experiments show that Gaussian approximations can be reliable in standard settings but may substantially underestimate directional risk under rare-event uncertainty, although the proposed certificate can be computationally demanding and conservative.
Critical strengths
1- The paper goes beyond the classical observation that the mean and variance cannot determine tail probabilities. It establishes rigorous non-identification results even when extensive statistical descriptors and complete first- and second-order policy-gradient functions are matched. The sharp ambiguity characterization further strengthens the theoretical contribution.
2- The authors construct fixed pairs of sequential REINFORCE MDPs exhibiting distinct directional large-deviation exponents despite matching key policy-gradient and environmental characteristics. The robustness results demonstrate that this phenomenon is not restricted to simple scalar or symmetric constructions.
3- Rather than presenting only impossibility results, the paper proposes a non-oracle finite-sample reliability certificate. The experiments employ independent validation, prespecified statistical tests, and transparent reporting of negative and inconclusive findings, strengthening the credibility of the empirical evidence.
Critical weaknesses
1- The certification procedure requires substantial independent pilot data and known gradient bounds. Its effectiveness deteriorates sharply when these bounds are conservative, and it becomes vacuous in several evaluated environments. This raises concerns about its feasibility in practical policy-gradient training.
2- Although the theoretical results establish important identification limitations, the strongest empirical failures arise in deliberately constructed rare-event settings. Most standard control experiments show adequate Gaussian calibration or remain inconclusive, limiting evidence that the identified problem is widespread in real-world RL.
3- The paper focuses on the probability of a gradient pointing in the wrong direction, but does not sufficiently demonstrate how differences in this probability affect policy improvement, training stability, or convergence. Thus, the operational consequences of the theoretical findings remain unclear.
4- The proposed finite-sample certificate is not comprehensively compared against alternative statistically valid confidence-based approaches under equivalent sampling budgets, making its practical advantage difficult to assess.
Questions for authors
N/A
2 comments
Thank you for your time and consideration. I believe I will keep my score at 3.
Response to Reviewer 5esd
Thank you for recognizing the non-identification results, sequential construction, and non-oracle certificate. We have substantially revised the entire manuscript, including the Abstract, Introduction, related work, theory, experiments, conclusion, and technical appendix. We also added targeted experiments addressing your concerns about practical utility, empirical breadth, and downstream learning consequences.
1. Practical utility and certificate comparisons (Section 4 and appendix).
We now distinguish finite-sample validity from practical informativeness and explicitly report pilot costs, conservative support bounds, and certificate abstention.
A new matched-budget benchmark evaluates 36,000 configurations across 60 MDPs, three gradient estimators, multiple pilot budgets, and support-envelope settings. With 20k pilot trajectories, MGF produces a valid recommendation in 18.4% of cases versus 38.8% for Bernstein. When both certify, the median Bernstein/MGF recommended-batch-size ratio is 1.054.
The matched H=4 construction demonstrates a more substantial conditional advantage: median sufficient future batch sizes of 70/69 (MGF), 150/151 (Bernstein), and 872/879 (Cantelli) for its two endpoints.
These findings establish a regime-dependent trade-off: MGF can yield tighter guarantees but abstains more frequently. We explicitly discuss total pilot cost and do not claim universal superiority or an established end-to-end advantage over adaptive safe-PG methods.
2. Stronger empirical confirmation (Section 4; Grid2Op appendix).
We completed a full independent Grid2Op replication, comprising 20,000 reference episodes, 50,000 calibration episodes, and 100,000 new confirmation batches of 16 trajectories.
The original experiment observed 1.258% wrong-direction risk; the replication observed 1.243%. Their exact one-sided 95% lower confidence bounds are 1.2006% and 1.1859%, respectively, both exceeding the prespecified 1% threshold.
The replicated Gaussian prediction is 0.9841% (95% bootstrap CI [0.8715%, 1.1077%]). The independently observed-minus-Gaussian risk gap is 0.2589 percentage points, with a 95% bootstrap CI of [0.1181, 0.3921] percentage points.
This provides independently replicated evidence of Gaussian underestimation in the specified configuration. We explicitly acknowledge that the Gaussian interval itself crosses 1%, and that the evaluation concerns a simplified Grid2Op sandbox rather than production grids. The supplement includes the implementation, raw data, bootstrap draws, and reproducibility checks.
3. Connection to actual return degradation (Section 3).
We strengthened the operational interpretation of the sequential result. In the matched construction, the true objective is strictly increasing in the policy parameter. Consequently, for every positive step size, a strict wrong-direction update is exactly equivalent to a decrease in expected return. This identity does not rely on a local approximation.
4. Direct learning-curve evidence (Section 4; matched-MDP appendix).
We added the requested training comparison across 18 configurations: horizons H∈{2,4,8}, batch sizes n∈{8,16,32}, two learning rates, 512 paired seeds, and 120 REINFORCE updates.
No final-return difference was statistically significant under simultaneous Bonferroni-adjusted 95% confidence intervals. We report this negative result explicitly: directional-risk separation does not imply a detectable difference in average final learning performance under these budgets.
A separate controlled Windy LineWorld family demonstrates that exactly moment-matched transition laws can also exhibit different finite-batch risks and training-level success rates, without establishing that such effects are prevalent in general RL.
5. Scope and reproducibility.
The revised Introduction and related work clarify the distinction between exact risk identification, conservative moment-based certification, and safe policy improvement. The appendix now specifies the complete rare-event trajectory-gradient sampling law and statistical confirmation protocol. We also explicitly distinguish the independent Grid2Op replication from the original implementation.
Overall, the revised manuscript now combines the theoretical identification results with an exact return-degradation interpretation, matched-learning experiments, a large certificate benchmark, independently replicated Grid2Op evidence, and empirical uncertainty intervals. These additions strengthen the practical interpretation while transparently identifying where the proposed certificates are informative and where their limitations remain.