- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to estimate the expected cumulative return of a fixed target strategy by using the complete trajectory collected by the unknown behavior strategy. The author proposes a reverse recursive regression algorithm, which adds the correction based on the advantage estimation to the actual observed follow-up return, and shrinks the correction to zero according to the confidence interval. When the confidence interval is valid, the conditional mean of the modified label still corresponds to the value of a replacement strategy. Based on this property, the paper gives a finite sample error guarantee without knowing the action probability of the behavior strategy under the condition that the value of the replacement strategy family meets the linear realizability and the data coverage target strategy. The author also proves by constructing an example that the realizability of the replacement strategy and the target coverage can still be satisfied when the corresponding full strategy conditions are not tenable, and reports the main statistical guarantee and the lean formal verification of the separation results.
Critical strengths
One of the key ideas of this paper is to link the return correction with the strategy replacement. After conservatively shrinking the estimated advantage, as long as the confidence interval is valid, the conditional mean of the corrected return still corresponds to the value of a replacement strategy. In this way, the regression needs to represent the value of the intermediate strategy, but the data only needs to cover the target strategy, not all the places that the intermediate strategy will access. Using this difference, the author gives a clear estimation algorithm and sample guarantee without knowing the probability of behavior strategy. The separation examples in this paper also show that the relaxation of these conditions does make the theorem applicable to the situation that the previous relevant guarantees cannot cover.
Critical weaknesses
The experimental part of this paper is relatively simple, which can supplement more experiments and conduct more comprehensive empirical analysis.
Questions for authors
no question
2 comments
We also studied weak support (Appendix I.2, Table 5). A second protocol sweeps the logging probability of the critical action from to on 20 further models. At the bias-floor correction with the smaller leverage cap stays between and over the whole sweep, as predicted; with the default cap it rises to at probability under the band logger. Under the same logger, FQE and the correction without shrinkage improve only gradually, from and at probability to and at and and at .
Two further families are reported in Appendix I.3 (Table 6). RMSE at the largest budget:
| Setting | Bias-floor (ours) | Without shrinkage | FQE | WIS, count logger | DR, count logger |
|---|---|---|---|---|---|
| Three-stage example of Appendix D.4, unlogged action | .667 | .667 | |||
| Same example, logger support | |||||
| Random linear MDPs, | .0083 | .0083 | .0056 | .0184 | .0124 |
| .0086 | .0086 | .0069 | .0178 | .0106 | |
| .0157 | .0157 | .0099 | .0935 | .0486 | |
| .0258 | .0259 | .0164 | .493 | .349 |
The random family has a full-support logger, the setting most favorable to FQE. There our rule stays within a factor – of FQE and is far more accurate than importance weighting at longer horizons.
On the three-stage example the verified rule itself converges: Algorithm 1 with the theorem's radius follows roughly the rate, with RMSE , , , at , , , . On the random linear MDPs it remains conservative at the tested budgets (RMSE –).
On protocol, the pre-specified part of the study uses 9,380 independent test datasets from 80 test models and one fixed example. Each protocol and its predictions were fixed before its test data were drawn, and all methods share each dataset. The convergence run is counted separately, and the feature-dimension diagnostic of Appendix I.4 is descriptive, without a frozen protocol. The order of events matters for the leverage cap. The four-regime and benchmark studies were run first, with the default cap . The smaller cap was then chosen on separate development models, after those test data had been inspected, and the support sweep tested it on new models. In Tables 4 and 6, which report the largest budgets, the two caps give the same values. The smaller cap costs accuracy when a block is small relative to the feature dimension (Appendix I.3). The code documentation gives protocols, seeds and commands, and every table is regenerated by one script.
For reference, the revision adds Proposition A.1 (split radius) and Corollary A.2 (zero correction above the leverage cap), with proofs and Lean verification (Appendices A.5 and F), and the numerical study: rules and baselines (I.1), the four-regime study and support sweep (I.2, Tables 4–5, Figure 2), two benchmark families and the convergence study (I.3, Table 6), and a feature-dimension study of Assumption 1 (I.4).
We hope the new study addresses your concern about the experimental part, and we would be glad to discuss any of it.
Dear Reviewer 6ZwH,
Thank you for the careful summary of the replacement mechanism and for asking for a more comprehensive empirical analysis. The submission contained only small numerical diagnostics. The revision adds a numerical study, Appendix I, organized around one question: where does each class of estimator fail, and does the proposed correction avoid those failures?
The main study uses 20 independently drawn models (Appendix I.2). Each has four decisions, three supplied features and 65,536 states per middle layer, and is evaluated under four unknown loggers in a two-by-two design. One target action, in a region the target never visits, is either logged or never logged ("missing support"). The logger's action probabilities are either constant or a non-monotone "band" function of a public feature. Target coverage is finite in all four cases.
The construction says where particular baselines should lose accuracy. Missing support makes regression extrapolate, which should break fitted Q evaluation (FQE) with a minimum-norm or ridge fit and corrections applied without shrinkage. The band logger should break weighted importance sampling (WIS) with a logistic logger model, and with per-state counts when support is also missing. After a pilot on two development models, and before drawing any test model, we fixed the protocol and 44 predictions of which methods would be accurate (RMSE at most .02) or fail (at least .05). 43 held. The exception is count-logger WIS in the last regime, whose RMSE of .042 is just below the failure threshold.
The table below gives RMSE at N=262,144 trajectories, with 200 datasets per column (Table 4 in the paper).
Each baseline that does not use the true logger is accurate in some columns and loses accuracy in others: its worst-column RMSE is between .034 and .40, against .0055 for the bias-floor correction. That correction is accurate in all four and has the smallest worst-case error at every budget (Figure 2). The 95% bootstrap interval of every entry lies within ±18% of it. With full support, FQE is slightly more accurate.
The rule comes from Proposition A.1, which is new in the revision and formally verified in Lean. It shows that the main theorem holds with a radius split into a deterministic part, which accounts for directions the data cannot determine, and a noise part. The bias-floor rule keeps only the deterministic part. Corollary A.2, also verified in Lean, guarantees on every dataset that it makes no correction at a query whose target actions all have leverage above a cap, which is the case for the never-logged action here. Its accuracy is an empirical finding. The finite-sample accuracy guarantee applies to the theorem's radius, which makes no correction at these budgets (RMSE .60–.62, Table 4) and converges at larger ones, as shown in our second comment.