acceptodds
Under review as a conference paper at ICLR 2027

When Is a Policy-Gradient Batch Reliable? Identification Limits Under Severe Uncertainty

Abstract

In reinforcement learning (RL), an agent learns a parameterized policy for sequential decision making by maximizing the expected cumulative return . Policy-gradient algorithms act on gradients estimated from finite trajectory batches, so a basic operational question is whether the resulting update points with high probability in the population-gradient direction. Existing work reduces gradient variance, adapts batch sizes, or controls policy improvement under explicit concentration assumptions, but these approaches do not answer a prior identification question: does the information conventionally retained about a one-trajectory gradient estimator determine its finite-batch wrong-direction probability? We show that, in general, it does not. Bounded atom-free REINFORCE problems can match any fixed jointly finitely generated menu of conventional one-rollout descriptors together with complete first- and second-order policy-gradient functions while having arbitrarily different strict directional risks. Under fixed Chernoff information and a bounded log-likelihood-ratio radius, we derive a sharp ambiguity law; a fixed pair of finite-horizon sequential REINFORCE Markov decision processes (MDPs) retains distinct directional large-deviation exponents despite exact matching of the stated policy-gradient and environment descriptors. We then give a non-oracle finite-sample certificate (without supplying the true population-gradient direction) using an independent confidence cone and held-out one-sided moment-generating-function information. Experiments show a regime-dependent picture: standard control settings can be close to Gaussian, whereas a controlled rare-critical family exhibits familywise-confirmed failures and a separately selected Grid2Op configuration survives an independent predeclared confirmation. The results position finite-batch policy-gradient reliability as an identification problem before it is a calibration problem.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, policy gradients, REINFORCE, finite-sample analysis, uncertainty quantification, large deviations, rare events, statistical certification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5esdOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether conventional policy-gradient statistics, particularly means and covariances, are sufficient to determine the probability that a finite-batch gradient points in the wrong direction. It establishes theoretical non-identification results, characterizes the resulting ambiguity, and demonstrates its persistence in sequential reinforcement learning environments. The authors also propose a finite-sample reliability certificate based on confidence regions and moment-generating-function information. Experiments show that Gaussian approximations can be reliable in standard settings but may substantially underestimate directional risk under rare-event uncertainty, although the proposed certificate can be computationally demanding and conservative.

Critical strengths

1- The paper goes beyond the classical observation that the mean and variance cannot determine tail probabilities. It establishes rigorous non-identification results even when extensive statistical descriptors and complete first- and second-order policy-gradient functions are matched. The sharp ambiguity characterization further strengthens the theoretical contribution.

2- The authors construct fixed pairs of sequential REINFORCE MDPs exhibiting distinct directional large-deviation exponents despite matching key policy-gradient and environmental characteristics. The robustness results demonstrate that this phenomenon is not restricted to simple scalar or symmetric constructions.

3- Rather than presenting only impossibility results, the paper proposes a non-oracle finite-sample reliability certificate. The experiments employ independent validation, prespecified statistical tests, and transparent reporting of negative and inconclusive findings, strengthening the credibility of the empirical evidence.

Critical weaknesses

1- The certification procedure requires substantial independent pilot data and known gradient bounds. Its effectiveness deteriorates sharply when these bounds are conservative, and it becomes vacuous in several evaluated environments. This raises concerns about its feasibility in practical policy-gradient training.

2- Although the theoretical results establish important identification limitations, the strongest empirical failures arise in deliberately constructed rare-event settings. Most standard control experiments show adequate Gaussian calibration or remain inconclusive, limiting evidence that the identified problem is widespread in real-world RL.

3- The paper focuses on the probability of a gradient pointing in the wrong direction, but does not sufficiently demonstrate how differences in this probability affect policy improvement, training stability, or convergence. Thus, the operational consequences of the theoretical findings remain unclear.

4- The proposed finite-sample certificate is not comprehensively compared against alternative statistically valid confidence-based approaches under equivalent sampling budgets, making its practical advantage difficult to assess.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5esd

Thank you for recognizing the non-identification results, sequential construction, and non-oracle certificate. We have substantially revised the entire manuscript, including the Abstract, Introduction, related work, theory, experiments, conclusion, and technical appendix. We also added targeted experiments addressing your concerns about practical utility, empirical breadth, and downstream learning consequences.

1. Practical utility and certificate comparisons (Section 4 and appendix).

We now distinguish finite-sample validity from practical informativeness and explicitly report pilot costs, conservative support bounds, and certificate abstention.

A new matched-budget benchmark evaluates 36,000 configurations across 60 MDPs, three gradient estimators, multiple pilot budgets, and support-envelope settings. With 20k pilot trajectories, MGF produces a valid recommendation in 18.4% of cases versus 38.8% for Bernstein. When both certify, the median Bernstein/MGF recommended-batch-size ratio is 1.054.

The matched H=4H=4 construction demonstrates a more substantial conditional advantage: median sufficient future batch sizes of 70/69 (MGF), 150/151 (Bernstein), and 872/879 (Cantelli) for its two endpoints.

These findings establish a regime-dependent trade-off: MGF can yield tighter guarantees but abstains more frequently. We explicitly discuss total pilot cost and do not claim universal superiority or an established end-to-end advantage over adaptive safe-PG methods.

2. Stronger empirical confirmation (Section 4; Grid2Op appendix).

We completed a full independent Grid2Op replication, comprising 20,000 reference episodes, 50,000 calibration episodes, and 100,000 new confirmation batches of 16 trajectories.

The original experiment observed 1.258% wrong-direction risk; the replication observed 1.243%. Their exact one-sided 95% lower confidence bounds are 1.2006% and 1.1859%, respectively, both exceeding the prespecified 1% threshold.

The replicated Gaussian prediction is 0.9841% (95% bootstrap CI [0.8715%, 1.1077%]). The independently observed-minus-Gaussian risk gap is 0.2589 percentage points, with a 95% bootstrap CI of [0.1181, 0.3921] percentage points.

This provides independently replicated evidence of Gaussian underestimation in the specified configuration. We explicitly acknowledge that the Gaussian interval itself crosses 1%, and that the evaluation concerns a simplified Grid2Op sandbox rather than production grids. The supplement includes the implementation, raw data, bootstrap draws, and reproducibility checks.

3. Connection to actual return degradation (Section 3).

We strengthened the operational interpretation of the sequential result. In the matched construction, the true objective is strictly increasing in the policy parameter. Consequently, for every positive step size, a strict wrong-direction update is exactly equivalent to a decrease in expected return. This identity does not rely on a local approximation.

4. Direct learning-curve evidence (Section 4; matched-MDP appendix).

We added the requested training comparison across 18 configurations: horizons H∈{2,4,8}H\in\{2,4,8\}, batch sizes n∈{8,16,32}n\in\{8,16,32\}, two learning rates, 512 paired seeds, and 120 REINFORCE updates.

No final-return difference was statistically significant under simultaneous Bonferroni-adjusted 95% confidence intervals. We report this negative result explicitly: directional-risk separation does not imply a detectable difference in average final learning performance under these budgets.

A separate controlled Windy LineWorld family demonstrates that exactly moment-matched transition laws can also exhibit different finite-batch risks and training-level success rates, without establishing that such effects are prevalent in general RL.

5. Scope and reproducibility.

The revised Introduction and related work clarify the distinction between exact risk identification, conservative moment-based certification, and safe policy improvement. The appendix now specifies the complete rare-event trajectory-gradient sampling law and statistical confirmation protocol. We also explicitly distinguish the independent Grid2Op replication from the original implementation.

Overall, the revised manuscript now combines the theoretical identification results with an exact return-degradation interpretation, matched-learning experiments, a large certificate benchmark, independently replicated Grid2Op evidence, and empirical uncertainty intervals. These additions strengthen the practical interpretation while transparently identifying where the proposed certificates are informative and where their limitations remain.

Thank you for your time and consideration. I believe I will keep my score at 3.

Reviewer_Up78OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether the mean and covariance of a REINFORCE gradient estimate determine how often a batch gradient "points downhill", i.e., how often a small step along it would decrease expected return. The author shows that the answer is no, by providing a counterexample of two MDPs with the same mean and variance, but one has a higher probability of pointing downhill than the other. The paper also shows synthetic contextual bandit experiments that empirically validates this result. The authors also show how to use rollouts to upper-bound this probability for future batches.

Critical strengths

The paper thoroughly studies whether mean and variance are sufficient statistics of the PG to predict the reliability of a batch update. Theorem 1 shows a contrastive pair of MDPs where every policy in the class has the same expected return and same mean/variance of its REINFORCE estimate, yet the probability taht the batch gadient points against the true gradient can differ by an arbitrarily large factor.

Critical weaknesses

Firstly, I found the paper's writing quite hard to read and parse which is why I give a presentation score of 1. For example, the Abstract is very long and begins by defining the RL problem. I found even the abstract very difficult to parse and understand, as it reads very dense and most of the information should not be in the abstract. I found this to generally be the case while reading the rest of the paper as well. I don't believe the paper is in a publishable state without significant rewriting of a lot of the text.

My second concern is the practical significance of Theorem 3. The two MDPs differ in how often a single REINFORCE batch "points downhill". However, for training, we typically care more about how quickly the policy approaches optimal return, rather than ensuring that every single gradient update is pointing downhill, which is impossible to ensure with stochastic gradient updates. Starting from the same policy and using the same batch size and step size, do the two MDPs produce meaningfully different learning curves at the same rollout budget?

Questions for authors
  1. Starting from the same policy and using the same batch size and step size, do the two MDPs produce meaningfully different learning curves at the same rollout budget?
  2. Section 2 discusses prior work on high-probability policy improvement, including Papini et al. In the Theorem 3 example, a downhill update actually lowers return. For the same target probability of improvement, would your certificate lead to a different choice of batch size than an existing safe-PG guarantee?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Up78

Thank you for recognizing the significance of the exact non-identification result and for identifying two important weaknesses: the manuscript's presentation and the relationship between directional risk and actual policy learning.

1. Substantial rewriting throughout the manuscript.

The manuscript has been substantially rewritten, rather than merely supplemented with clarifying remarks. The Abstract is more focused; the Introduction now begins with the central identification question and a concrete numerical example; Section 2 more clearly distinguishes our contribution from variance reduction and safe policy improvement; and Section 3 separates the main theorems, their interpretations, and their operational consequences. We also revised the experimental narrative, conclusion, and technical appendix to make assumptions, quantifiers, and limitations more explicit.

In particular, the revised Introduction illustrates the matched sequential environments using an H=4H=4, n=16n=16 example: their objectives and first two gradient moments agree exactly, while strict downhill-update probabilities are approximately 0.923% and 0.710%.

2. Do the matched MDPs produce different learning curves under equal budgets?

We now answer this question with a new matched-learning experiment, reported in Section 4, “Do the matched MDPs learn differently?”, and the supplementary subsection Matched-MDP learning and certificate comparison.

We trained vanilla REINFORCE on both endpoints from identical initial policies, using matched rollout budgets and paired random seeds. The experiment covers 18 configurations:

  • Horizons H∈{2,4,8}H\in\{2,4,8\};
  • Batch sizes n∈{8,16,32}n\in\{8,16,32\};
  • Learning rates {0.5,2}\{0.5,2\};
  • 512 paired seeds and 120 updates per configuration.

No final-return difference was statistically significant under the simultaneous Bonferroni-adjusted 95% confidence intervals.

This is an important distinction: the theoretical separation in directional-error probabilities does not automatically produce a detectable difference in average final learning performance under these budgets. The revised manuscript now states this explicitly.

We also established a stronger, exact one-update interpretation of Theorem 3. In the matched positive-gradient construction, the common objective is strictly increasing in the policy parameter. Consequently, for every positive step size, the event that the stochastic update decreases expected return coincides exactly with the strict wrong-direction event (revised Section 3, equation labeled actual-degradation). This is an exact return-degradation identity, not a local Taylor approximation. It explains the operational significance of the risk difference without claiming different convergence rates.

3. How does the certificate compare with existing safe-PG guarantees?

We added quantitative, matched-pilot comparisons against valid Cantelli, Hoeffding, and Bernstein bounds in Section 4, “Comparison with valid concentration bounds,” and the corresponding appendix.

For the matched H=4H=4 pair, with the same 20,000-trajectory pilot and 95% pilot-confidence budget, the median sufficient future batch sizes for a 1% directional-risk target are:

CertificateEndpoint AEndpoint B
Held-out MGF7069
Bernstein150151
Cantelli872879
Hoeffding1,1341,144

In this construction, an upper bound on wrong-direction risk is also an upper bound on the probability of an actual return-decreasing update. Thus, the matched example provides a concrete setting where different valid bounds recommend different batch sizes for the same one-update reliability target.

We further added a 60-MDP benchmark that reveals a more nuanced picture: the MGF certificate can be tighter when informative, but it abstains more frequently than Bernstein. We therefore distinguish future-batch tightness from total pilot cost and do not claim an end-to-end advantage over the complete adaptive safe-PG algorithms of Papini et al.

Summary of revisions. The rewritten paper now provides a clearer identification question, an exact link to one-update return degradation, a directly requested learning-curve experiment, and statistically valid matched-budget certificate comparisons. The result is a more precise account of both the contribution and its practical scope.

Reviewer_pWkPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the probability that a finite-batch policy-gradient estimate points against the population gradient: $ R_n^{\mathrm{str}} = \Pr!\left(\langle \bar g_n,\nabla J(\theta)\rangle<0\right). $ Its central question is whether commonly retained information about a trajectory-gradient estimator determines this probability.

The main theoretical contributions are:

  • Exact non-identification: For any fixed even batch size and an admissible finite descriptor menu, bounded, atom-free REINFORCE problems can match those descriptors and their complete objective, gradient, and gradient-second-moment functions while exhibiting arbitrarily large directional-risk ratios.
  • A sharp ambiguity law: Within a sign-symmetric likelihood-ratio class, fixing Chernoff information C0C_0 and an LR-radius bound BB leaves a risk-envelope ratio of order min⁡{n,eB/4}\min\{\sqrt n,e^{B/4}\}.
  • Sequential persistence: A fixed pair of transition laws yields matched finite-horizon MDPs with distinct directional large-deviation exponents at every finite horizon. Additional constructions address symmetry, continuous transition noise, and full-rank policy geometry.
  • Constructive certification: An independent direction-confidence sample and held-out MGF validation sample provide a finite-sample upper bound on the wrong-direction risk of future independent batches. The experiments combine exactly enumerable and deliberately constructed examples with control-task audits and a separately confirmed Grid2Op configuration. They support a regime-dependent conclusion: Gaussian approximations can work well, but higher-order distributional information can materially affect directional reliability.
Critical strengths
  1. Theorem 1 preserves a broad, precisely defined collection of descriptors together with the complete Js(η)J_s(\eta), ∇Js(η)\nabla J_s(\eta), and second-moment functions. The continuous-microcell and Carathéodory construction gives a concrete explanation for how substantial distributional freedom survives these constraints. The use of atom-free laws avoids making the main conclusion depend on how zero-gradient batches are classified.

  2. The likelihood-conditioned ambiguity law is a substantive quantitative contribution. Theorem 2 identifies both growth and saturation of the compatible-risk ratio. The fixed-BB saturation is important: the result does not imply indefinitely increasing ambiguity with batch size under fixed likelihood constraints. The paper also correctly restricts the matched-PG corollary to the branch it actually establishes.

  3. The fixed sequential construction addresses a meaningful quantifier issue. Theorem 1 permits its witnesses to depend on the requested batch size. Theorem 3 supplies a different construction whose endpoint laws remain fixed as batch size grows. Matching the objective, gradient geometry, and specified environmental information while changing the directional exponent is a stronger and more interesting statement than merely finding different finite-sample tails. Reporting the 1/(H−1)1/(H-1) decay of the exponent gap appropriately limits its interpretation.

  4. Theorem 5 propagates direction uncertainty through a confidence cone and validates a selected MGF value on independent observations. The event-containment argument is transparent, and the paper clearly states the necessity of a valid deterministic envelope. The vector confidence ball is appropriately credited to existing empirical-Bernstein work; the paper’s contribution is its use within the directional-risk certificate.

  5. The authors distinguish controlled mechanisms from prevalence evidence, discovery from confirmation, and failed reference gates from negative findings. The CartPole and Acrobot results help prevent an overly broad claim of Gaussian failure. The SRE examples preserve substantial population signal, while the independent Grid2Op confirmation provides an external sequential example. The certificate’s envelope sensitivity and complete abstention on the prospective MountainCar/Acrobot panel are valuable limitations to report.

Critical weaknesses
  1. Exact non-identification should be distinguished more sharply from the possibility of conservative certification. Some operational language, especially in the introduction, risks suggesting that first- and second-order information cannot support valid reliability guarantees. The negative theorems establish that such information does not determine the exact risk. To make the distinction concrete, let $ Z=\langle \widehat g,g/|g|\rangle, m=\mathbb E Z=|g|>0, \operatorname{Var}(Z)=\sigma^2. $ Applying the one-sided Chebyshev/Cantelli inequality to the batch mean gives $ R_n^{\mathrm{str}} \leq \frac{\sigma^2}{\sigma^2+n m^2}. $ Thus, known population first and second moments already yield a valid, potentially conservative sufficient batch size. This is consistent with classical moment-based probability bounds. flora.insead.edu The operational contribution should therefore be framed around risk identification, tightness, and the conservatism of summary-based guarantees. Uncertainty in estimated moments is a further statistical issue. Quantifying how much the proposed certificate improves over valid moment-based bounds would strengthen the practical argument.

  2. The all-horizon perturbation argument needs a more explicit uniform justification. Appendix G.3 invokes continuity and positive margins to preserve the derivative inequalities under one perturbation that works for every horizon. However, the relevant unnormalized quantities vanish near zero: the matched tangent satisfies h(z)=O(z3)h(z)=O(z^3), and the symmetric combination in Equation (39) vanishes at fourth order. Consequently, strict inequalities on a punctured interval do not by themselves provide a uniform positive margin. The proof should explicitly control normalized quantities such as h(z)/z3h(z)/z^3 and the corresponding fourth-order expression, including their continuous extensions at zero and preservation of the required cancellations under perturbation. The preceding algebra suggests this may be repairable, but the current continuity argument is too compressed for the strength of Theorem 4.

  3. Practical usefulness of the certificate remains limited, and its comparative advantage is unclear. The main tabular result uses 20,000 pilot trajectories to obtain a median sufficient future batch size of 200 among positive certificates. Increasing the pilot helps, but Table 6 shows substantial sensitivity to the supplied envelope: doubling the tight bound reduces positivity from 59.4% to 18.9%, and multiplying it by five leaves 1.1%. The paper acknowledges these costs, but it does not establish when they are preferable to simpler rigorous alternatives. The most useful comparison would use the same information, confidence budget, and total trajectory budget for the proposed certificate and concentration-based future-risk bounds. Existing safe-PG work (https://proceedings.neurips.cc/paper_files/paper/2017/file/ea6b2efbdd4255a9f1b3bbc6399b58f4-Paper.pdf) already studies adaptive batching under concentration assumptions, so a qualitative guarantee table alone does not establish an operational advantage.

  4. The external confirmation supports a narrower claim than a confirmed failure of Gaussian calibration. Grid2Op reports 1,258 wrong-direction batches out of 100,000, with a one-sided lower bound of approximately 1.2006%. This supports risk exceeding the nominal 1% threshold, subject to identifying the population-gradient sign correctly. The independent Gaussian estimate is approximately 0.9978%, only slightly below that threshold, and no uncertainty interval is supplied for that estimate. The paper appropriately labels it descriptive; this qualification is essential. The result should not be presented as a joint statistical demonstration that the population Gaussian prediction is below 1% while the true risk is above it.

  5. The sequential theoretical family remains highly structured, even after its useful robustness extensions. The 65-dimensional experiment trains a random-feature output layer under designed noise laws. The PPO-trained CartPole audit evaluates fresh frozen-policy score-function gradients, rather than the full multi-epoch PPO update. Eleven of twelve prospective MountainCar/Acrobot estimator cases are inconclusive, and the broader MuJoCo panel does not produce a stable reference. These limitations do not undermine the existence theorems. They do mean that the paper’s strongest contribution is currently theoretical and diagnostic. It provides limited evidence about how often the mechanism affects ordinary deep actor–critic training or whether the certificate improves end-to-end learning.

  6. Section 4 states that confirmatory threshold tests use exact Clopper–Pearson bounds, whereas Appendix Q.3 specifies Hoeffding lower bounds for SRE confirmation. Both can be valid, but the description must identify which procedure produced each result. The SRE appendix also needs a self-contained specification of the trajectory-gradient law, its dependence on horizon, numerical choices for qq and bordb_{\mathrm{ord}}, the candidate batch-size grid, and screening/confirmation sample counts. These details matter for reproducing the principal controlled failures.

Questions for authors

Can you explicitly distinguish failure to identify RnR_n from the existence of valid conservative upper bounds?

Can you provide the normalized derivative inequalities and explicit uniform margins needed in Appendix G.3, including their behavior at zero and preservation under asymmetric smoothing?

In which concrete fixed-policy setting can a 20k–30k pilot be amortized sufficiently to improve total sample cost?

What is the uncertainty interval for the independent Gaussian risk estimate?

Were the reported familywise bounds computed using Hoeffding or Clopper–Pearson? Please provide the complete estimator definition, batch-selection rule, confirmation counts, and confidence-budget allocation.

Which parts of the sequential separation argument fail for state-dependent baselines or return-to-go estimators?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pWkP

Thank you for the detailed technical review. We have substantially revised the entire manuscript, including the Abstract, Introduction, related work, theory exposition, experiments, conclusion, and appendix. The principal changes directly address your concerns:

1. Identification versus certification (Introduction and Section 3). We now explicitly distinguish non-identification of exact wrong-direction risk from valid conservative moment-based guarantees. In particular, Cantelli provides a valid upper bound; our negative results establish that matching moments need not identify the actual risk. The non-oracle MGF certificate addresses a separate finite-sample certification problem.

2. All-horizon proof (Appendix G.3). We repaired the perturbation argument using continuously extended normalized quantities h(z)/z3h(z)/z^3, the symmetric derivative expression normalized by a4a^4, and minimizer-bracket expressions normalized by aa. Exact-rational margins establish a common perturbation neighborhood preserving the required inequalities uniformly across finite horizons, including near zero imbalance. This directly addresses the previous unjustified inference of uniformity from vanishing unnormalized quantities.

3. Fair certificate comparisons (Section 4 and appendix). Under matched pilot observations and confidence allocation, the H=4H=4 construction gives median sufficient future batch sizes of 70/69 (MGF), 150/151 (Bernstein), and 872/879 (Cantelli). We additionally evaluated 36,000 configurations across 60 MDPs. With 20k pilots, MGF certifies in 18.4% versus 38.8% for Bernstein; among jointly certified cases, the median Bernstein/MGF batch-size ratio is 1.054. We now explicitly report abstention, pilot cost, and the absence of a demonstrated end-to-end advantage over adaptive safe-PG algorithms.

4. Grid2Op uncertainty and full replication (Section 4; Grid2Op appendix). The original confirmation observed 1,258/100,000 wrong-direction batches (1.258%; exact one-sided 95% lower bound 1.2006%). A separate full replication—20k reference episodes, 50k calibration episodes, and 100k new batches—observed 1,243/100,000 (1.243%; lower bound 1.1859%). Its Gaussian prediction is 0.9841% (95% bootstrap CI [0.8715%, 1.1077%]). The directly observed risk exceeds that prediction by 0.2589 percentage points (95% bootstrap CI [0.1181, 0.3921]). The Gaussian CI itself crosses 1%; our conclusions reflect this distinction. Original and replicated samples are analyzed separately. The supplement provides raw observations, code, bootstrap draws, and provenance.

5. SRE reproducibility (SRE appendix). We now specify the complete trajectory-gradient sampling law, including the episode-level rare-event draw, Gaussian noise, horizon-dependent gradient and variance, and the frozen batch grid 20,…,2202^0,\ldots,2^{20}. All 280 archived case–threshold records were checked. We explicitly distinguish Bonferroni-adjusted Hoeffding confirmation for SRE from exact Clopper–Pearson confirmation for Grid2Op.

6. Estimator scope (sequential appendix). We clarify that terminal-only rewards make undiscounted return-to-go identical to full return, and analyze deterministic time-indexed baselines, including the variance-minimizing baseline. We do not extend these results to arbitrary learned state-dependent baselines.

These revisions strengthen the mathematical justification, empirical reproducibility, and matched-budget evaluation while maintaining the distinction between identification, valid certification, and downstream learning performance.

Thank you for addressing my comments and questions. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.