- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes pairwise Bellman validation as a way to select fitted Q-functions and occupancy-ratio estimators without an extra round of critic fitting. The main additions are penalties that tolerate misspecified candidate classes, an adjoint selector for occupancy ratios, and finite-sample oracle inequalities. The framework is coherent and the theory is a sensible extension.
Critical strengths
1 PBV constructs critics directly from candidate differences and evaluates observable Bellman moments, avoiding auxiliary critic fitting or candidate-specific validation regressions.
2 The paper treats both Q-function and occupancy-ratio selection. The misspecification guarantees and the adjoint construction extend the scope beyond selecting among realizable Q-function classes.
3 The experiments include LSTD-Tournament, penalty ablations, broad-offline and controlled-overlap settings, and temperature sensitivity analyses.
Critical weaknesses
- The novelty relative to existing work is not clearly articulated.
- Monte Carlo rollout reuse is insufficiently documented.
- The connection between the guarantees and the tempered weights remains unclear.
- The assumptions are restrictive, with unclear applicability to the experiments.
- Empirical gains over existing methods are inconsistent
Questions for authors
-
Novelty relative to LSTD-Tournament. My main difficulty is that the paper is close enough to its closest comparison, LSTD-Tournament [1], that the marginal contribution is hard to judge. [1] already constructs critics from normalized candidate differences and avoids auxiliary critic fitting, and the paper itself acknowledges that the two-candidate LSTDQ normal equations are essentially an unpenalized unweighted PBV in form. That leaves the misspecification analysis and the adjoint (occupancy-ratio) selector as the only apparent deltas. I would like the authors to spell out precisely what these two additions contribute over [1] — in the guarantees and in practice, and at what assumption cost — because as written the gap is too thin to assess whether the contribution clears the bar.
-
Rollout implementation. Appendix D.2 (p. 22, lines 1153–1156) says broad Hopper uses the Monte Carlo candidates from [1], but it never spells out how the rollout estimates get assigned across the critic, residual, and penalty products, or whether the split-rollout treatment from [1] (Appendix E.3, p. 28) carries over. Reusing a noisy estimate inside a product can add variance, and the independent continuation returns on p. 23 are about final error evaluation, so they don't settle this.
-
Tempering and the scope of the guarantees. The paper explicitly describes tempering and training-sample normalization (pp. 9 and 22), and Table S2 (p. 25) already provides temperature sensitivity results. Given that Theorem S5 (p. 19) accommodates imperfect fitted weights, could the authors clarify how its relative weight-error term is controlled after tempering and how this connects to Theorem 5.1? Tempering generally changes the weighting distribution even when the original ratio is exact.
-
Assumptions. Under what benchmark conditions do the positive, floored neural ratio candidates actually satisfy A2's support and uniform comparability requirements (p. 8; implementation on p. 22)? Could the authors also clarify how the trajectory-mixture coverage bounds correspond to the actual validation distribution ν, including time sampling and termination (pp. 22–23), and how the dependence assumptions apply to this sampling procedure?
-
Evidence for the distinctive components. The one clearly positive headline is that TA-PBV attains the lowest mean value error under controlled overlap (14.5 vs. 25.5 for unweighted PBV and 15.9 for LSTD-Tournament on Hopper; 69.3 vs. 83.2 and 74.7 on Deep OPE). But Table 1 is otherwise mixed for the penalty itself: under controlled Hopper overlap, penalized unweighted PBV has mean value error 25.5 versus 15.1 without the penalty. So the ablations matter. I would like to see when the penalty helps rather than hurts, a direct evaluation of A-PBV's ratio-selection quality, and paired uncertainty estimates for the selector comparisons. Since the penalty and the adjoint selector are the paper's most distinctive components, pinning down which of them drives the controlled-overlap win would substantially strengthen the case.
References
[1] Pai Liu, Lingfeng Zhao, Shivangi Agarwal, Jinghan Liu, Audrey Huang, Philip Amortila, and Nan Jiang. Model selection for off-policy evaluation: New algorithms and experimental protocol. In Advances in Neural Information Processing Systems, volume 38, 2025.
9 comments
Response to Reviewer EdGk (Part 2 of 3)
Relation to LSTD-Tournament.
Thank you for this feedback. We have substantially revised the positioning and related-work discussion to clarify the similarities and differences between PBV and LSTD-Tournament.
We agree that both approaches use candidate value functions to construct Bellman-based comparisons without fitting auxiliary critics. Indeed, the two-candidate LSTDQ normal equations are closely related to unpenalized unweighted PBV. However, the criteria and their theoretical motivations differ. Liu et al. (2025) derive LSTD-Tournament from pairwise LSTDQ, using the candidate -functions themselves as critics (Eq. 10). We instead show, through an oracle-critic analysis, that pairwise differences between candidates suffice as critics, and introduce quadratic penalties to obtain guarantees under misspecification (revised Section 3.2).
The closest resemblance is to the modified LSTD-Tournament criterion used in their experiments, rather than the criterion analyzed theoretically. Their experimental implementation transforms the LSTDQ features to include normalized pairwise differences, primarily to improve matrix conditioning (Appendix E.2). In contrast, our use of pairwise differences follows directly from the oracle-critic analysis, and our criterion requires neither normalization of the critics nor matrix conditioning and inversion assumptions. Their theoretical guarantee does not directly cover the data-dependent normalization used in their experiments.
These differences matter for the guarantees. For the theoretically analyzed criterion, LSTD-Tournament assumes exact realizability and invertible pairwise LSTD matrices, with bounds that deteriorate under poor conditioning (Theorem 2). This can be restrictive in model selection, where candidates trained with similar hyperparameters often produce nearly collinear -functions. Unweighted PBV instead allows misspecification through its quadratic penalties and requires only absolute-value coercivity of a scalar Bellman quadratic form. We now show that LSTD-Tournament's conditioning constant can become arbitrarily small while PBV's coercivity constant remains bounded away from zero (revised Appendix B). For the normalized LSTD-Tournament criterion, we also show through counterexamples and finite-sample analysis that dividing by estimated standard deviations can introduce substantial estimation error and instability when these standard deviations are small (revised Appendices M–N, Proposition S35).
Beyond unweighted selection, we develop two methods that LSTD-Tournament does not address. TA-PBV uses target-occupancy weighting to remove the coercivity requirement with exact weights, trading this assumption for accurate ratio estimation and sufficient coverage. A-PBV selects occupancy ratios without coercivity or matrix-invertibility assumptions, requiring instead bounded log ratios between candidates and one-step coverage. Combining the two gives an end-to-end guarantee without coercivity or matrix invertibility, while allowing approximation error in both the ratio and -function libraries (revised Section 5.2, Theorem 5.2). We have also added penalty ablations and direct A-PBV ratio-selection experiments to evaluate these contributions separately.
When does penalization help?
Our original experiments showed mixed benefits from penalization in unweighted PBV. We investigated this and found that large negative values of the estimated Bellman quadratic penalty undermined performance. By taking its absolute value, we weaken our theoretical coercivity assumption to allow the Bellman quadratic form to have either sign, while substantially improving empirical performance.
Our revised ablations now show clear benefits from penalization. With 3,200 validation transitions, removing the penalty increases mean absolute policy-value error by 21% and 52% for unweighted PBV on model-free and model-based Hopper, respectively, and by 33% and 43% for TA-PBV at . Initial-state value RMSE also increases in both settings. These comparisons hold candidates, weights, and validation draws fixed (revised Appendix F.7, Figure S4).
The penalty also matters theoretically under misspecification, when even the best candidate can have a nonzero Bellman residual. It prevents large critics from dominating the Bellman moments. We constructed a counterexample showing that without the penalty, the selected candidate can be arbitrarily worse than the best available candidate, even with perfect coverage (revised Appendix D.2, Proposition S7).
Response to Reviewer EdGk (Part 3 of 3)
Direct evaluation of A-PBV.
We now evaluate A-PBV directly as an occupancy-ratio selector, independently of downstream -function selection. For comparison, we extended LSTD-Tournament to occupancy ratios using the adjoint Bellman equation (revised Appendix O). A-PBV avoids the matrix-conditioning assumptions of this baseline and achieves lower Hellinger error across the FORE (van der Laan and Kallus, 2026), DualDICE (Nachum et al., 2019), and combined candidate libraries on controlled Hopper. On the combined library, Hellinger error decreases from 0.580 to 0.537. Generalized KL, ratio RMSE, and importance-weighted reward error also improve across all three libraries (revised Appendix G.2, Table S6).
We have also weakened A-PBV's theoretical assumptions. Rather than requiring bounded log ratios relative to the unknown true occupancy ratio, our revised guarantee requires only bounded log ratios between candidates and standard one-step coverage. The candidate bounds can be enforced when constructing the library. Combined with TA-PBV, this gives an end-to-end guarantee that allows approximation error in both the ratio and -function libraries (revised Section 5.2, Condition A3 and Theorem 5.2).
Tempering and theoretical guarantees.
We adopted tempering from Sinha et al. (2022, Eq. 8), who study importance-weighted TD learning and recommend to manage the bias-variance tradeoff. Their theoretical guarantees use exact target-policy weights, while their experiments use tempering to improve performance. We find similar benefits: holding estimated ratios, -functions, and validation draws fixed, increasing from 1 to 5 reduces mean absolute policy-value error from 5.50 to 2.31 on Hopper and from 0.121 to 0.088 on controlled-overlap DeepOPE (revised Appendix G.3, Table S7).
Tempering interpolates between TA-PBV () and unweighted PBV (). Our original TA-PBV selection guarantee remains valid, but its relative weight-error term must be evaluated after tempering and normalization (revised Appendix D.1, Theorem S5). Thus, even exact occupancy ratios can introduce weight error after tempering, potentially making this bound loose.
To address this, we establish a complementary -function error guarantee under the tempered weighting distribution that does not require accurate occupancy-ratio estimation. Instead, it requires coercivity of the Bellman quadratic form under that distribution (revised Appendix D.3, Theorem S11). Thus, even imperfect weights can be useful if they improve alignment with the target-policy dynamics, allowing coercivity to hold where it fails under unweighted sampling.
Rollout reuse and validation sampling.
For the released Hopper benchmark, we use LSTD-Tournament's logged transitions, cached Monte Carlo predictions, and reference values. We reuse the same cached predictions in the pairwise critics, TD residuals, and quadratic penalties, rather than splitting continuation rollouts across these terms. All selectors use identical caches and validation draws, while independent continuation returns are used for final evaluation. Controlled Hopper and DeepOPE instead use fitted neural -functions, which we also hold fixed across terms (revised Appendix F.1–F.3).
We also clarify how the validation distribution accounts for trajectory lengths, time sampling, and termination. In our benchmark comparisons, we pool transitions from complete trajectories. Sampling logging policies uniformly at trajectory start does not necessarily yield a uniform transition mixture when episode lengths differ. In contrast, our direct ratio experiment uses a known mixture of normalized discounted occupancies. Our concentration bounds assume that the validation process satisfies the stated spectral-gap condition, yielding effective sample size (revised Section 5, Condition A1). We also preserve complete trajectories in training-validation splits and DeepOPE uncertainty calculations (revised Appendices F.1, F.4, and G.2).
References.
- Fu et al. (2021). Benchmarks for Deep Off-Policy Evaluation. ICLR.
- Liu et al. (2025). Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol. NeurIPS.
- Nachum et al. (2019). DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections. NeurIPS.
- Sinha et al. (2022). Experience Replay with Likelihood-free Importance Weights. L4DC.
- van der Laan and Kallus (2026). Fitted Occupancy-Ratio Evaluation without Bellman Completeness. arXiv.
Response to Reviewer EdGk (Part 1 of 3)
Revised manuscript (anonymous PDF; key changes in blue): PDF
Thank you for your positive assessment of the problem, theory, and experiments. We have revised our methods, extended the theoretical guarantees, and added experiments to address your concerns.
Since submission, a few simple changes to our PBV criteria have substantially improved empirical performance, with both unweighted PBV and TA-PBV now often outperforming LSTD-Tournament on Hopper and remaining competitive on DeepOPE. These changes also strengthen our theoretical guarantees. For unweighted PBV, the original criterion required a Bellman quadratic form to be nonnegative and coercive. By taking absolute values, we now require only coercivity in absolute value, allowing the form to have either sign (revised Section 5.1, Condition A2). This substantially weakens the assumption. Our experiments also showed that large negative values of the signed Bellman moment or penalty largely explained PBV's previously mixed performance relative to LSTD-Tournament.
For TA-PBV, which uses occupancy weighting to remove the coercivity requirement under exact target-occupancy weights, we made the same absolute-value change and replaced the L2 penalty with an importance-weighted version of PBV's quadratic penalty. TA-PBV is now obtained simply by importance weighting each expectation in unweighted PBV (revised Sections 4.1–4.2). Although absolute values are unnecessary for the guarantee with exact weights, the revised criterion substantially improves performance with imperfect or tempered weights.
We also establish new Q-function error guarantees for TA-PBV with imperfect or tempered weights under the induced weighting distribution, provided coercivity holds under that distribution (revised Appendix D.3, Theorem S11). These guarantees complement our original selection guarantees, which can become uninformative when weight-estimation error is substantial.
We have also expanded our theoretical and empirical comparisons with LSTD-Tournament, added penalty ablations and direct A-PBV ratio-selection experiments, reported paired uncertainty estimates, and clarified the Monte Carlo rollout implementation and sampling assumptions.
New experimental results.
In the revision, we reproduce the original LSTD-Tournament benchmarks, spanning 340 environment–policy combinations across model-free Hopper (120) and DeepOPE/D4RL (220), plus 36 model-based Hopper combinations. PBV and TA-PBV now achieve the lowest mean absolute policy-value error and initial-state value RMSE across the Hopper groups and in controlled-overlap DeepOPE, with comparable performance on the broader DeepOPE benchmark (Liu et al., 2025; Fu et al., 2021; revised Table 1, Figure 1, and Appendix G.3). We also report paired uncertainty estimates in Table S1 of revised Appendix F.4.
Table 1. Selection at nQ=3,200. VE is mean absolute policy-value error; RMSE is initial-state value RMSE. MF, poor MF, and MB are the main model-free, poor-coverage model-free, and model-based Hopper groups. Bold values mark the smallest mean in each column. Hopper errors use original return units; DeepOPE errors are normalized by each environment's reference policy-value range.
On main and poor-coverage Hopper, unweighted PBV reduces mean absolute policy-value error by 42% and 35%, respectively, relative to normalized LSTD-Tournament.