acceptodds
Under review as a conference paper at ICLR 2027

Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation

Abstract

Model selection is a central challenge in off-policy evaluation (OPE) because fitted -functions and occupancy ratios lack supervised validation losses. Standard approaches based on Bellman-residual estimation or minimax criteria often require auxiliary regressions or critic classes that must themselves be learned and tuned. We introduce pairwise Bellman validation (PBV), a tuning-free selection procedure among a finite collection of fitted -functions and occupancy-ratio estimators that uses only held-out transitions and requires no auxiliary validation models. The construction is motivated by an oracle critic given by a candidate's estimation error, which, when used to test the Bellman residual, controls the error of interest. Because this oracle critic is unknown, we approximate it using pairwise differences between candidates. Under misspecification, these pairwise approximations are imperfect, so we add penalties that account for the resulting critic-approximation error. For -function selection, we introduce target-aligned PBV, which uses marginalized importance weighting to evaluate the pairwise tests under the discounted target-occupancy distribution. When occupancy-ratio estimation is undesirable, we propose an unweighted variant closely related to LSTD-Tournament, but it requires an additional conditioning assumption linking offline Bellman tests to -function error. For occupancy-ratio selection, we introduce adjoint PBV, which applies the same pairwise construction to the adjoint Bellman equation. For all three procedures, we establish population and finite-sample oracle inequalities relative to the best candidates in possibly misspecified classes.

Primary area:
reinforcement learning
Keywords:
off-policy evaluation, model selection, hyperparameter tuning, Bellman validation, minimax estimation, action-value functions, occupancy ratios, offline reinforcement learning.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EdGkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes pairwise Bellman validation as a way to select fitted Q-functions and occupancy-ratio estimators without an extra round of critic fitting. The main additions are penalties that tolerate misspecified candidate classes, an adjoint selector for occupancy ratios, and finite-sample oracle inequalities. The framework is coherent and the theory is a sensible extension.

Critical strengths

1 PBV constructs critics directly from candidate differences and evaluates observable Bellman moments, avoiding auxiliary critic fitting or candidate-specific validation regressions.

2 The paper treats both Q-function and occupancy-ratio selection. The misspecification guarantees and the adjoint construction extend the scope beyond selecting among realizable Q-function classes.

3 The experiments include LSTD-Tournament, penalty ablations, broad-offline and controlled-overlap settings, and temperature sensitivity analyses.

Critical weaknesses
  1. The novelty relative to existing work is not clearly articulated.
  2. Monte Carlo rollout reuse is insufficiently documented.
  3. The connection between the guarantees and the tempered weights remains unclear.
  4. The assumptions are restrictive, with unclear applicability to the experiments.
  5. Empirical gains over existing methods are inconsistent
Questions for authors
  1. Novelty relative to LSTD-Tournament. My main difficulty is that the paper is close enough to its closest comparison, LSTD-Tournament [1], that the marginal contribution is hard to judge. [1] already constructs critics from normalized candidate differences and avoids auxiliary critic fitting, and the paper itself acknowledges that the two-candidate LSTDQ normal equations are essentially an unpenalized unweighted PBV in form. That leaves the misspecification analysis and the adjoint (occupancy-ratio) selector as the only apparent deltas. I would like the authors to spell out precisely what these two additions contribute over [1] — in the guarantees and in practice, and at what assumption cost — because as written the gap is too thin to assess whether the contribution clears the bar.

  2. Rollout implementation. Appendix D.2 (p. 22, lines 1153–1156) says broad Hopper uses the Monte Carlo candidates from [1], but it never spells out how the rollout estimates get assigned across the critic, residual, and penalty products, or whether the split-rollout treatment from [1] (Appendix E.3, p. 28) carries over. Reusing a noisy estimate inside a product can add variance, and the independent continuation returns on p. 23 are about final error evaluation, so they don't settle this.

  3. Tempering and the scope of the guarantees. The paper explicitly describes tempering and training-sample normalization (pp. 9 and 22), and Table S2 (p. 25) already provides temperature sensitivity results. Given that Theorem S5 (p. 19) accommodates imperfect fitted weights, could the authors clarify how its relative weight-error term is controlled after tempering and how this connects to Theorem 5.1? Tempering generally changes the weighting distribution even when the original ratio is exact.

  4. Assumptions. Under what benchmark conditions do the positive, floored neural ratio candidates actually satisfy A2's support and uniform comparability requirements (p. 8; implementation on p. 22)? Could the authors also clarify how the trajectory-mixture coverage bounds correspond to the actual validation distribution ν, including time sampling and termination (pp. 22–23), and how the dependence assumptions apply to this sampling procedure?

  5. Evidence for the distinctive components. The one clearly positive headline is that TA-PBV attains the lowest mean value error under controlled overlap (14.5 vs. 25.5 for unweighted PBV and 15.9 for LSTD-Tournament on Hopper; 69.3 vs. 83.2 and 74.7 on Deep OPE). But Table 1 is otherwise mixed for the penalty itself: under controlled Hopper overlap, penalized unweighted PBV has mean value error 25.5 versus 15.1 without the penalty. So the ablations matter. I would like to see when the penalty helps rather than hurts, a direct evaluation of A-PBV's ratio-selection quality, and paired uncertainty estimates for the selector comparisons. Since the penalty and the adjoint selector are the paper's most distinctive components, pinning down which of them drives the controlled-overlap win would substantially strengthen the case.

References

[1] Pai Liu, Lingfeng Zhao, Shivangi Agarwal, Jinghan Liu, Audrey Huang, Philip Amortila, and Nan Jiang. Model selection for off-policy evaluation: New algorithms and experimental protocol. In Advances in Neural Information Processing Systems, volume 38, 2025.

9 comments
AuthorsOpenReview ↗

Response to Reviewer EdGk (Part 1 of 3)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for your positive assessment of the problem, theory, and experiments. We have revised our methods, extended the theoretical guarantees, and added experiments to address your concerns.

Since submission, a few simple changes to our PBV criteria have substantially improved empirical performance, with both unweighted PBV and TA-PBV now often outperforming LSTD-Tournament on Hopper and remaining competitive on DeepOPE. These changes also strengthen our theoretical guarantees. For unweighted PBV, the original criterion required a Bellman quadratic form to be nonnegative and coercive. By taking absolute values, we now require only coercivity in absolute value, allowing the form to have either sign (revised Section 5.1, Condition A2). This substantially weakens the assumption. Our experiments also showed that large negative values of the signed Bellman moment or penalty largely explained PBV's previously mixed performance relative to LSTD-Tournament.

For TA-PBV, which uses occupancy weighting to remove the coercivity requirement under exact target-occupancy weights, we made the same absolute-value change and replaced the L2L^2 penalty with an importance-weighted version of PBV's quadratic penalty. TA-PBV is now obtained simply by importance weighting each expectation in unweighted PBV (revised Sections 4.1–4.2). Although absolute values are unnecessary for the guarantee with exact weights, the revised criterion substantially improves performance with imperfect or tempered weights.

We also establish new QQ-function error guarantees for TA-PBV with imperfect or tempered weights under the induced weighting distribution, provided coercivity holds under that distribution (revised Appendix D.3, Theorem S11). These guarantees complement our original selection guarantees, which can become uninformative when weight-estimation error is substantial.

We have also expanded our theoretical and empirical comparisons with LSTD-Tournament, added penalty ablations and direct A-PBV ratio-selection experiments, reported paired uncertainty estimates, and clarified the Monte Carlo rollout implementation and sampling assumptions.

New experimental results.

In the revision, we reproduce the original LSTD-Tournament benchmarks, spanning 340 environment–policy combinations across model-free Hopper (120) and DeepOPE/D4RL (220), plus 36 model-based Hopper combinations. PBV and TA-PBV now achieve the lowest mean absolute policy-value error and initial-state value RMSE across the Hopper groups and in controlled-overlap DeepOPE, with comparable performance on the broader DeepOPE benchmark (Liu et al., 2025; Fu et al., 2021; revised Table 1, Figure 1, and Appendix G.3). We also report paired uncertainty estimates in Table S1 of revised Appendix F.4.

Table 1. Selection at nQ=3,200n_Q=3{,}200. VE is mean absolute policy-value error; RMSE is initial-state value RMSE. MF, poor MF, and MB are the main model-free, poor-coverage model-free, and model-based Hopper groups. Bold values mark the smallest mean in each column. Hopper errors use original return units; DeepOPE errors are normalized by each environment's reference policy-value range.

SelectorMF VEMF RMSEPoor MF VEPoor MF RMSEMB VEMB RMSEDeepOPE VEDeepOPE RMSE
Unweighted PBV1.784.225.2610.281.253.090.5990.704
TA-PBV (T=5T=5)2.315.045.2510.241.613.760.5980.705
LSTD-Tournament (normalized)3.097.458.0414.281.363.430.5970.702
LSTD-Tournament (unnormalized)2.465.906.5012.201.894.340.5870.691
Squared TD error6.7514.4012.0220.252.565.350.7330.826

On main and poor-coverage Hopper, unweighted PBV reduces mean absolute policy-value error by 42% and 35%, respectively, relative to normalized LSTD-Tournament.

AuthorsOpenReview ↗

Response to Reviewer EdGk (Part 2 of 3)

Relation to LSTD-Tournament.

Thank you for this feedback. We have substantially revised the positioning and related-work discussion to clarify the similarities and differences between PBV and LSTD-Tournament.

We agree that both approaches use candidate value functions to construct Bellman-based comparisons without fitting auxiliary critics. Indeed, the two-candidate LSTDQ normal equations are closely related to unpenalized unweighted PBV. However, the criteria and their theoretical motivations differ. Liu et al. (2025) derive LSTD-Tournament from pairwise LSTDQ, using the candidate QQ-functions themselves as critics (Eq. 10). We instead show, through an oracle-critic analysis, that pairwise differences between candidates suffice as critics, and introduce quadratic penalties to obtain guarantees under misspecification (revised Section 3.2).

The closest resemblance is to the modified LSTD-Tournament criterion used in their experiments, rather than the criterion analyzed theoretically. Their experimental implementation transforms the LSTDQ features to include normalized pairwise differences, primarily to improve matrix conditioning (Appendix E.2). In contrast, our use of pairwise differences follows directly from the oracle-critic analysis, and our criterion requires neither normalization of the critics nor matrix conditioning and inversion assumptions. Their theoretical guarantee does not directly cover the data-dependent normalization used in their experiments.

These differences matter for the guarantees. For the theoretically analyzed criterion, LSTD-Tournament assumes exact realizability and invertible pairwise LSTD matrices, with bounds that deteriorate under poor conditioning (Theorem 2). This can be restrictive in model selection, where candidates trained with similar hyperparameters often produce nearly collinear QQ-functions. Unweighted PBV instead allows misspecification through its quadratic penalties and requires only absolute-value coercivity of a scalar Bellman quadratic form. We now show that LSTD-Tournament's conditioning constant can become arbitrarily small while PBV's coercivity constant remains bounded away from zero (revised Appendix B). For the normalized LSTD-Tournament criterion, we also show through counterexamples and finite-sample analysis that dividing by estimated standard deviations can introduce substantial estimation error and instability when these standard deviations are small (revised Appendices M–N, Proposition S35).

Beyond unweighted selection, we develop two methods that LSTD-Tournament does not address. TA-PBV uses target-occupancy weighting to remove the coercivity requirement with exact weights, trading this assumption for accurate ratio estimation and sufficient coverage. A-PBV selects occupancy ratios without coercivity or matrix-invertibility assumptions, requiring instead bounded log ratios between candidates and one-step coverage. Combining the two gives an end-to-end guarantee without coercivity or matrix invertibility, while allowing approximation error in both the ratio and QQ-function libraries (revised Section 5.2, Theorem 5.2). We have also added penalty ablations and direct A-PBV ratio-selection experiments to evaluate these contributions separately.

When does penalization help?

Our original experiments showed mixed benefits from penalization in unweighted PBV. We investigated this and found that large negative values of the estimated Bellman quadratic penalty undermined performance. By taking its absolute value, we weaken our theoretical coercivity assumption to allow the Bellman quadratic form to have either sign, while substantially improving empirical performance.

Our revised ablations now show clear benefits from penalization. With 3,200 validation transitions, removing the penalty increases mean absolute policy-value error by 21% and 52% for unweighted PBV on model-free and model-based Hopper, respectively, and by 33% and 43% for TA-PBV at T=5T=5. Initial-state value RMSE also increases in both settings. These comparisons hold candidates, weights, and validation draws fixed (revised Appendix F.7, Figure S4).

The penalty also matters theoretically under misspecification, when even the best candidate can have a nonzero Bellman residual. It prevents large critics from dominating the Bellman moments. We constructed a counterexample showing that without the penalty, the selected candidate can be arbitrarily worse than the best available candidate, even with perfect coverage (revised Appendix D.2, Proposition S7).

AuthorsOpenReview ↗

Response to Reviewer EdGk (Part 3 of 3)

Direct evaluation of A-PBV.

We now evaluate A-PBV directly as an occupancy-ratio selector, independently of downstream QQ-function selection. For comparison, we extended LSTD-Tournament to occupancy ratios using the adjoint Bellman equation (revised Appendix O). A-PBV avoids the matrix-conditioning assumptions of this baseline and achieves lower Hellinger error across the FORE (van der Laan and Kallus, 2026), DualDICE (Nachum et al., 2019), and combined candidate libraries on controlled Hopper. On the combined library, Hellinger error decreases from 0.580 to 0.537. Generalized KL, ratio RMSE, and importance-weighted reward error also improve across all three libraries (revised Appendix G.2, Table S6).

We have also weakened A-PBV's theoretical assumptions. Rather than requiring bounded log ratios relative to the unknown true occupancy ratio, our revised guarantee requires only bounded log ratios between candidates and standard one-step coverage. The candidate bounds can be enforced when constructing the library. Combined with TA-PBV, this gives an end-to-end guarantee that allows approximation error in both the ratio and QQ-function libraries (revised Section 5.2, Condition A3 and Theorem 5.2).

Tempering and theoretical guarantees.

We adopted tempering from Sinha et al. (2022, Eq. 8), who study importance-weighted TD learning and recommend T=5T=5 to manage the bias-variance tradeoff. Their theoretical guarantees use exact target-policy weights, while their experiments use tempering to improve performance. We find similar benefits: holding estimated ratios, QQ-functions, and validation draws fixed, increasing TT from 1 to 5 reduces mean absolute policy-value error from 5.50 to 2.31 on Hopper and from 0.121 to 0.088 on controlled-overlap DeepOPE (revised Appendix G.3, Table S7).

Tempering interpolates between TA-PBV (T=1T=1) and unweighted PBV (T→∞T\to\infty). Our original TA-PBV selection guarantee remains valid, but its relative weight-error term must be evaluated after tempering and normalization (revised Appendix D.1, Theorem S5). Thus, even exact occupancy ratios can introduce weight error after tempering, potentially making this bound loose.

To address this, we establish a complementary QQ-function error guarantee under the tempered weighting distribution that does not require accurate occupancy-ratio estimation. Instead, it requires coercivity of the Bellman quadratic form under that distribution (revised Appendix D.3, Theorem S11). Thus, even imperfect weights can be useful if they improve alignment with the target-policy dynamics, allowing coercivity to hold where it fails under unweighted sampling.

Rollout reuse and validation sampling.

For the released Hopper benchmark, we use LSTD-Tournament's logged transitions, cached Monte Carlo Q/VQ/V predictions, and reference values. We reuse the same cached predictions in the pairwise critics, TD residuals, and quadratic penalties, rather than splitting continuation rollouts across these terms. All selectors use identical caches and validation draws, while independent continuation returns are used for final evaluation. Controlled Hopper and DeepOPE instead use fitted neural QQ-functions, which we also hold fixed across terms (revised Appendix F.1–F.3).

We also clarify how the validation distribution ν\nu accounts for trajectory lengths, time sampling, and termination. In our benchmark comparisons, we pool transitions from complete trajectories. Sampling logging policies uniformly at trajectory start does not necessarily yield a uniform transition mixture when episode lengths differ. In contrast, our direct ratio experiment uses a known mixture of normalized discounted occupancies. Our concentration bounds assume that the validation process satisfies the stated spectral-gap condition, yielding effective sample size ngbng_b (revised Section 5, Condition A1). We also preserve complete trajectories in training-validation splits and DeepOPE uncertainty calculations (revised Appendices F.1, F.4, and G.2).

References.

Reviewer_YDBgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses model selection in OPE, where fitted Q-functions and occupancy-ratio estimators lack directly observable supervised validation losses. -The authors introduce PBV, a tuning-free procedure that selects among finite candidate classes using held-out transitions, without fitting auxiliary validation models or critics.
  • PBV approximates oracle critics using pairwise differences between candidates. Each candidate is scored by its worst-case penalized Bellman moment, with penalties accounting for approximation errors under misspecification.
  • Three variants are developed: TA-PBV uses estimated occupancy ratios to evaluate Q-function candidates under the discounted target-occupancy distribution; unweighted PBV avoids weighting but requires a coercivity assumption; and A-PBV selects occupancy-ratio estimators using the adjoint Bellman equation.
  • The authors establish population and finite-sample oracle inequalities relative to the best candidate, allowing misspecified classes and dependent validation trajectories under the stated assumptions. -Experiments on Hopper and Deep OPE benchmarks show competitive but inconsistent performance relative to LSTD-Tournament. TA-PBV achieves the lowest mean value error under controlled overlap but provides no consistent improvement on standard offline benchmarks.
Critical strengths

Strengths:

  • The oracle inequalities hold for misspecified classes, which is the realistic case. This is a clear theoretical advance over LSTD-Tournament, whose guarantees require the true Q-function to be among the candidates. The proofs seem correct.

  • Innovative new setup adressing a fairly new problem afaik: Selecting among already-fitted occupancy-ratio estimators has received little attention, and adjoint PBV gives a principled, guaranteed procedure for it.

  • Honest reporting of empircial results and where the method did not win.

Critical weaknesses

a) Limited and inconsistent empirical improvements: Despite its theoretical advantages, PBV does not consistently outperform LSTD-Tournament. Reported uncertainty intervals further limit the strength of some empirical comparisons. Overall, the experiments do not establish a consistent practical advantage over the closest baseline. Limited practical differentiation from LSTD-Tournament.

b) Limited empirical benefit of the proposed penalty. The penalties are central to the oracle guarantees under misspecification, yet their empirical benefits are unclear. Although not contradicting the theory, raises questions about the practical usefulness and calibration of the penalty. Experiments do not clearly identify settings where penalisation is beneficial.

  1. Loose bounds and restrictive theoretical assumptions (not matching the experiments?) . Although the authors note that coercivity can be assessed empirically, they do not report such measurements for the evaluated benchmarks. These assumptions may limit the applicability of the guarantees under severe distribution shift. As gamma approaches to 1 the bounds become deteriorate.
Questions for authors

i) PBV provides stronger theoretical guarantees under misspecification, but its empirical performance is often comparable to LSTD-Tournament. Could the authors identify settings where PBV offers a clear practical advantage? In particular, could they evaluate performance under varying degrees of candidate-class misspecification? ii) The quadratic penalty is central to the theoretical guarantees, yet removing it improves unweighted PBV in several settings. Could the authors explain this behavior and investigate whether the penalty is overly conservative? How sensitive are the results to its coefficient? ii) practical implementation uses ratio tempering with exponent 1/5, which is not directly covered by the theoretical guarantees. How sensitive are the results to this choice? Can the theoretical analysis be extended to account for tempered occupancy ratios? iv) A-PBV is one of the main contributions, but its effectiveness as an occupancy-ratio selector is not directly established in the main experiments. Could the authors evaluate its selection quality independently, for example using generalized-KL error relative to the best candidate, and compare it against alternative selection procedures? v) The guarantees rely on strong assumptions, particularly uniform comparability of occupancy-ratio candidates with the true ratio and pairwise coercivity for unweighted PBV. Could the authors assess whether these assumptions approximately hold in the evaluated benchmarks? In particular, can the pairwise coercivity constant be estimated empirically, and does it correlate with selection performance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YDBg (Part 1 of 3)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for your positive assessment of our misspecification guarantees, occupancy-ratio selection method, and experimental design. Since submission, we have revised both PBV criteria, substantially improved empirical performance, and added new theoretical analyses and experiments. We address your questions below.

(i) Practical advantages over LSTD-Tournament and misspecification

Since submission, a few simple changes to our PBV criteria have substantially improved empirical performance. Both unweighted PBV and TA-PBV now often outperform LSTD-Tournament on Hopper and remain competitive on DeepOPE. We reproduce the original LSTD-Tournament benchmarks, spanning 340 environment–policy combinations across model-free Hopper (120) and DeepOPE/D4RL (220), plus 36 model-based Hopper combinations. Our revised criteria achieve the lowest mean absolute policy-value error and initial-state value RMSE across the Hopper groups and in controlled-overlap DeepOPE, with comparable performance on the broader DeepOPE benchmark (revised Table 1, Figure 1, and Appendix G.3). We also report paired uncertainty estimates (revised Appendix F.4, Table S1).

Table 1. Selection at nQ=3,200n_Q=3{,}200. VE is mean absolute policy-value error; RMSE is initial-state value RMSE. MF, poor MF, and MB denote the main model-free, poor-coverage model-free, and model-based Hopper groups. Bold values mark the smallest mean in each column. Hopper errors use original return units; DeepOPE errors are normalized by each environment's reference policy-value range.

SelectorMF VEMF RMSEPoor MF VEPoor MF RMSEMB VEMB RMSEDeepOPE VEDeepOPE RMSE
Unweighted PBV1.784.225.2610.281.253.090.5990.704
TA-PBV (T=5T=5)2.315.045.2510.241.613.760.5980.705
LSTD-Tournament (normalized)3.097.458.0414.281.363.430.5970.702
LSTD-Tournament (unnormalized)2.465.906.5012.201.894.340.5870.691
Squared TD error6.7514.4012.0220.252.565.350.7330.826

On main and poor-coverage Hopper, unweighted PBV reduces mean absolute policy-value error by 42% and 35%, respectively, relative to normalized LSTD-Tournament.

We have also expanded our theoretical comparison with LSTD-Tournament. Its original guarantee requires exact realizability and invertible pairwise LSTD matrices, which can be restrictive when candidates trained with similar hyperparameters are nearly collinear. PBV allows misspecification without requiring matrix invertibility. We now show that LSTD-Tournament's conditioning constant can become arbitrarily small while PBV's coercivity constant remains bounded away from zero (revised Appendix B). We also show that the normalization used in their experiments can amplify estimation error when critic standard deviations are small or underestimated. Our new misspecification analysis of both LSTD variants further clarifies these differences (revised Appendices M–N).

AuthorsOpenReview ↗

Response to Reviewer YDBg (Part 2 of 3)

(ii) When does penalization help?

Our original experiments showed mixed benefits from penalization in unweighted PBV. We investigated this and found that large negative penalty values undermined performance. The penalty estimates a Bellman quadratic form that our original coercivity assumption required to be nonnegative. By taking absolute values in our criterion, we remove this restriction and address the empirical problem.

Our revised ablations now show clear benefits from penalization. At 3,200 validation transitions, removing the penalty increases mean absolute policy-value error by 21% and 52% for unweighted PBV on main model-free and model-based Hopper, respectively, and by 33% and 43% for TA-PBV at T=5T=5. Initial-state value RMSE also increases in both groups. These comparisons hold candidates, occupancy weights, and validation draws fixed (revised Appendix F.7, Figure S4).

The penalty also matters theoretically under misspecification, when even the best candidate can have a nonzero Bellman residual. It prevents large critics from dominating the Bellman moments. We constructed a counterexample showing that without the penalty, the selected candidate can be arbitrarily worse than the best available candidate, even with perfect coverage (revised Appendix D.2, Proposition S7).

(iii) Tempering and theoretical guarantees

We adopted tempering from Sinha et al. (2022, Eq. 8), who study importance-weighted TD learning and recommend T=5T=5 to manage the bias-variance tradeoff. Their theoretical guarantees use exact target-policy weights, while their experiments use tempering to improve performance. We find similar benefits: holding estimated ratios, QQ-functions, and validation draws fixed, increasing TT from 1 to 5 reduces mean absolute policy-value error from 5.50 to 2.31 on Hopper and from 0.121 to 0.088 on controlled-overlap DeepOPE. We found T=5T=5 to be a good default, although T=2T=2 performs better in some settings (revised Appendix G.3, Table S7).

Tempering interpolates between TA-PBV (T=1T=1) and unweighted PBV (T→∞T\to\infty). Our original TA-PBV guarantee remains valid, but its relative weight-error term must be evaluated after tempering and normalization (revised Appendix D.1, Theorem S5). Thus, even exact occupancy ratios can introduce weight error after tempering, potentially making this bound loose.

To address this, we establish a complementary QQ-function error guarantee under the tempered weighting distribution that does not require accurate occupancy-ratio estimation. Instead, it requires coercivity of the Bellman quadratic form under that distribution (revised Appendix D.3, Theorem S11). Thus, even imperfect weights can be useful if they improve alignment with the target-policy dynamics, allowing coercivity to hold where it fails under unweighted sampling.

(iv) Direct evaluation of A-PBV

We now evaluate A-PBV directly as an occupancy-ratio selector, independently of downstream QQ-function selection. For comparison, we extended LSTD-Tournament to occupancy ratios using the adjoint Bellman equation (revised Appendix O).

On controlled Hopper, A-PBV achieves lower Hellinger error than this baseline across FORE (van der Laan and Kallus, 2026), DualDICE (Nachum et al., 2019), and combined candidate libraries. On the combined library, Hellinger error decreases from 0.580 to 0.537. Generalized KL, ratio RMSE, and importance-weighted reward error also improve across all three libraries (revised Appendix G.2, Table S6). Downstream TA-PBV performance remains competitive, although rankings vary across benchmarks (revised Appendix G.3, Table S7).

AuthorsOpenReview ↗

Response to Reviewer YDBg (Part 3 of 3)

(v) Assumptions and their practical implications

The main assumption that is difficult to verify is coercivity, which our original analysis required for unweighted PBV. We have substantially weakened this condition: by taking absolute values of the Bellman moments and quadratic penalties, we now require only coercivity in absolute value, allowing the quadratic form to have either sign (revised Section 5.1, Condition A2). For a finite candidate class, this requires the form to be nonzero on nonzero pairwise differences, with the coercivity constant determining the strength of the guarantee. The condition plays a role similar to matrix-conditioning assumptions in linear TD learning and LSTD-Tournament and holds automatically under stationary on-policy sampling, but not necessarily off-policy (revised Appendix D.2, Corollary S9).

We developed TA-PBV and A-PBV specifically to overcome this limitation. TA-PBV uses target-occupancy weighting to remove the coercivity requirement when the weights are exact, trading this assumption for accurate ratio estimation and sufficient coverage. We then developed A-PBV to select occupancy ratios without coercivity or matrix-invertibility assumptions, exploiting a KL-divergence contraction property of the adjoint Bellman operator under the offline distribution.

We have also weakened the assumptions for A-PBV. Our original analysis required uniformly bounded log ratios between each candidate and the unknown true occupancy ratio. We now require only bounded log ratios between candidates, which holds automatically for a finite class of ratios bounded above and away from zero, along with standard one-step coverage (revised Section 5.2, Condition A3). Combining A-PBV with TA-PBV gives an end-to-end guarantee without coercivity, matrix invertibility, or exact realizability, depending instead on the approximation quality of the ratio and QQ-function libraries under standard coverage conditions (revised Section 5.2, Theorem 5.2).

Our experiments also help explain when these assumptions matter. Unweighted PBV's strong performance on Hopper suggests that our weakened coercivity condition is compatible with good performance in challenging off-policy settings. Our controlled-overlap experiments include a fixed fraction of on-policy data, improving target-policy coverage and making occupancy ratios easier to estimate and less variable. Accordingly, TA-PBV performs more favorably relative to unweighted PBV in these settings than in the broader offline benchmarks, which include settings with limited coverage.

Finally, the deterioration of the bounds as γ→1\gamma\to1 reflects a general challenge in long-horizon OPE. Similar dependence appears in analyses of FQE (Le et al., 2019, Theorem 4.2) and LSTD-Tournament (Liu et al., 2025, Theorem 2). The LSTD-Tournament guarantee also depends on matrix-conditioning constants that may deteriorate as γ\gamma approaches one.

References.

Reviewer_77wCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies model selection in OPE, and proposes pairwise Bellman validation (PBV) as a tuning-free selection procedure over a finite candidate class. The PBV algorithm is introduced in three variants, i.e. adjoint, target-aligned and unweighted PBV, each coming with finite-sample guarantees. Experiments show promising value-error results under controlled overlap, yet less consistent gains are observed in broad offline benchmarks.

Critical strengths
  1. The research problem studied by the paper is important, and the proposed solution is intuitive.
  2. The theoretical analyses is technically sound and easy to follow. The treatment of distribution shift is clear and helps to explain when each variant may or may not work.
  3. The simulations are designed to include relevant comparisons and ablations for better understanding of the empirical behavior of the algorithm.
Critical weaknesses
  1. The experimental results do not show consistent gains. In fact, a significant portion of the best results in Table 1 are not achieved by the proposed algorithm. The authors are urged to provide reasonable explanations.
    • Also, according to the paper, the selection ratio is "tempered" and selected as wT=w1/T/Pn,w[w1/T]w_T = w^{1/T} / \mathbb{P}_{n,w}[w^{1/T}] with T=5T=5, which comes with no justification and seems to be inconsistent with the theoretical analysis. An explanation would also be helpful here.
  2. The assumptions used in theoretical analyses seem hard to verify in practical settings. It would be helpful to connect the assumptions and bounds to the empirical results to promote understanding of the theoretical results.
Questions for authors

See weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 77wC (Part 1 of 3)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for your positive assessment of the problem, theory, and experiments. In response to your concerns, we have improved the empirical performance of both PBV criteria, provided new guarantees for imperfect and tempered weights, and clarified the scope and practical implications of our assumptions.

Since submission, a few simple changes to our PBV criteria have substantially improved empirical performance: both unweighted PBV and TA-PBV now often outperform LSTD-Tournament on Hopper and remain competitive on DeepOPE. These changes also strengthen our theoretical guarantees. For unweighted PBV, the original criterion required a quadratic form induced by the Bellman operator to be nonnegative and coercive on pairwise differences between candidates. By taking absolute values, we remove the sign restriction and require only coercivity in absolute value (revised Section 5.1, Condition A2). For a finite candidate set, this condition only requires the quadratic form to be nonzero on nonzero pairwise differences, with the magnitude of the coercivity constant determining the strength of the guarantee. We therefore expect the condition to hold much more generally. Our experiments also showed that PBV's previously mixed performance relative to LSTD-Tournament was largely driven by extreme negative values of either the signed moment or the signed penalty.

The coercivity requirement applies only to unweighted PBV; TA-PBV uses importance weighting to remove it. However, with imperfect or tempered weights, our original selection guarantees for TA-PBV can become uninformative when weight-estimation error is substantial. To address this limitation and improve robustness, we made the same absolute-value change and replaced the L2L^2 penalty with an importance-weighted version of PBV's quadratic penalty. TA-PBV is now obtained simply by importance weighting each expectation in unweighted PBV (revised Sections 4.1–4.2). We also establish new QQ-function error guarantees for TA-PBV with imperfect or tempered weights, analogous to those for unweighted PBV under the induced weighting distribution, provided coercivity holds under that distribution (revised Appendix D.3, Theorem S11). These changes preserve the original pairwise-validation construction and require only routine adjustments to the existing proofs. Empirically, they substantially improved TA-PBV's performance with imperfect or tempered weights.

New experimental results.

In the revision, we reproduce the original LSTD-Tournament benchmarks, spanning 340 environment–policy combinations across model-free Hopper (120) and DeepOPE/D4RL (220), plus 36 model-based Hopper combinations (Liu et al., 2025; Fu et al., 2021). PBV and TA-PBV now achieve the lowest mean absolute policy-value error and initial-state value RMSE across the Hopper groups and in the controlled-overlap DeepOPE settings, with comparable performance on the broader DeepOPE benchmark (see revised Table 1 and Figure 1 for the main results, and Appendix G.3 for controlled-overlap results). Table S1 in revised Appendix F.4 reports paired uncertainty estimates.

Table 1. Selection at nQ=3,200n_Q=3{,}200. VE is mean absolute policy-value error; RMSE is initial-state value RMSE. MF, poor MF, and MB are the main model-free, poor-coverage model-free, and model-based Hopper groups. Bold values mark the smallest mean in each column. Hopper errors use original return units; DeepOPE errors are normalized by each environment's reference policy-value range.

SelectorMF VEMF RMSEPoor MF VEPoor MF RMSEMB VEMB RMSEDeepOPE VEDeepOPE RMSE
Unweighted PBV1.784.225.2610.281.253.090.5990.704
TA-PBV (T=5T=5)2.315.045.2510.241.613.760.5980.705
LSTD-Tournament (normalized)3.097.458.0414.281.363.430.5970.702
LSTD-Tournament (unnormalized)2.465.906.5012.201.894.340.5870.691
Squared TD error6.7514.4012.0220.252.565.350.7330.826

On main and poor-coverage Hopper, unweighted PBV reduces mean absolute policy-value error by 42% and 35%, respectively, relative to normalized LSTD-Tournament.

AuthorsOpenReview ↗

Response to Reviewer 77wC (Part 2 of 3)

Tempering: motivation and guarantees.

Tempering is a standard way to manage the bias-variance tradeoff in importance weighting. We adopted it from Sinha et al. (2022, Eq. 8), who study importance-weighted TD learning, and followed their recommended default of T=5T=5. We find similar empirical benefits to those reported by Sinha et al.: holding estimated ratios, candidate QQ-functions, and validation draws fixed, increasing TT from 1 to 5 reduces mean absolute policy-value error from 5.50 to 2.31 on Hopper and from 0.121 to 0.088 on controlled-overlap DeepOPE. We found T=5T=5 to be a good default, although T=2T=2 performs better in some settings. In the revision, we explicitly discuss the motivation for tempering and provide additional ablations of the tempering factor (revised Appendix G.3, Table S7).

We also clarify the theoretical role of tempering. Our original TA-PBV selection guarantee remains valid, with ratio-estimation error evaluated after tempering and normalization, but can be loose when tempered weights differ substantially from the true occupancy ratios (revised Appendix D.1, Theorem S5). In contrast, our new guarantee (Theorem S11) does not require accurate occupancy-ratio estimation, but instead requires coercivity under the induced weighting distribution. Thus, even imperfect weights can be useful if they improve alignment with the target-policy dynamics and allow coercivity to hold where it fails under unweighted sampling.

AuthorsOpenReview ↗

Response to Reviewer 77wC (Part 3 of 3)

Interpretation of the assumptions and empirical results.

The main assumption that is difficult to verify is coercivity, which applies to unweighted PBV. Our revised condition only requires a scalar Bellman quadratic form to be nonzero on nonzero pairwise candidate differences, without requiring nonnegativity. This condition plays a role similar to the matrix-conditioning assumptions in linear TD learning and LSTD-Tournament, which are also difficult to verify and reflect the challenges of distribution shift in OPE. It holds automatically under stationary on-policy sampling, but not necessarily off-policy (revised Appendix D.2, Corollary S9).

We developed TA-PBV and A-PBV specifically to overcome this limitation. TA-PBV uses importance weighting in the model-selection criterion to remove the coercivity requirement with exact target-occupancy weights, trading this assumption for accurate ratio estimation under sufficient coverage. To address the resulting ratio-estimation problem, we developed A-PBV, which selects occupancy ratios without coercivity or matrix-invertibility assumptions by exploiting a KL-divergence contraction property of the adjoint Bellman operator under the offline distribution.

We have also weakened the assumptions for A-PBV. Our original analysis required uniformly bounded log ratios between each candidate and the unknown true occupancy ratio. We now require only bounded log ratios between candidates, which holds automatically for a finite class of ratios bounded above and away from zero, along with standard one-step coverage (revised Section 5.2, Condition A3). Combining A-PBV with TA-PBV then gives an end-to-end guarantee without coercivity, matrix invertibility, or exact realizability. Instead, the guarantee depends on the approximation quality of the ratio and QQ-function libraries under standard coverage conditions (revised Section 5.2, Theorem 5.2).

We have also expanded our analysis of LSTD-Tournament to help explain PBV's improved performance. For the unnormalized criterion, Liu et al. (2025) require strong matrix-conditioning assumptions on pairs of candidate value functions, which can be restrictive when candidates are nearly collinear, as often occurs when tuning similar models. In contrast, our revised PBV criterion only requires absolute-value coercivity of a scalar Bellman quadratic form, without pairwise matrix invertibility. For the normalized criterion used in their experiments, LSTD-Tournament divides by estimated standard deviations of pairwise differences to improve matrix conditioning. Through counterexamples and finite-sample analysis, we show that these standard deviations can be small and difficult to estimate, making normalization unstable and weakening the theoretical guarantees. Both limitations provide plausible explanations for PBV's stronger empirical performance.

We have also revised our experimental discussion to better connect the theoretical assumptions to the results. Unweighted PBV's strong performance on Hopper suggests that our weakened coercivity condition is compatible with good performance in challenging off-policy settings. Our controlled-overlap experiments include a fixed fraction of on-policy data, improving target-policy coverage and making occupancy ratios easier to estimate and less variable. Consistent with this, TA-PBV performs more favorably relative to unweighted PBV in these settings than in the broader offline benchmarks, which include settings with limited coverage. Our new theory further shows that imperfect or tempered weights can still be useful: we obtain QQ-function error guarantees under the induced weighting distribution when coercivity holds there, without requiring accurate occupancy-ratio estimation. Unweighted PBV avoids ratio-estimation error and can therefore be preferable when weights are unreliable, provided its coercivity condition holds.

References.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Our paper addresses a practical obstacle in off-policy evaluation: choosing among fitted value functions and occupancy-ratio estimators without an observable validation loss. PBV uses held-out sample averages, requires no auxiliary critic fitting, and provides oracle guarantees even when every candidate is misspecified.

We have expanded the theory and experiments to address the reviewers’ main concerns:

  1. Comparison with LSTD-Tournament. LSTD-Tournament's theory uses candidate Q-functions themselves as critics, whereas PBV uses pairwise differences with a quadratic critic penalty. We now distinguish LSTD-Tournament's theoretical variant from its normalized experimental implementation and analyze both under misspecification. Our finite-sample theory shows that LSTD-Tournament can be sensitive to matrix conditioning and normalization by estimated standard deviations, respectively, neither of which PBV requires. We also provide a counterexample where LSTD-Tournament becomes ill-conditioned while PBV's coercivity remains bounded away from zero. Beyond this comparison, our original submission developed importance-weighted PBV and extended pairwise validation to occupancy ratios.

  2. Experimental performance. We modified both PBV criteria to take absolute values of the Bellman moments and quadratic penalties, weakening the coercivity assumption for unweighted PBV and substantially improving empirical performance. We also expanded our experiments to include the full Hopper benchmark from the LSTD-Tournament paper. Unweighted PBV now outperforms both normalized and unnormalized LSTD-Tournament on both reported metrics in all three released Hopper groups, reducing mean policy-value error by 42% and 35% relative to the normalized variant on the main and poor-coverage groups. Under controlled overlap, TA-PBV achieves the lowest aggregate mean policy-value error among the compared selectors. On the separate DeepOPE benchmark, the methods performed comparably. We also added matched penalty ablations, paired uncertainty estimates, and direct A-PBV evaluations across FORE, DualDICE, and combined candidate libraries to assess the individual components.

  3. Weaker assumptions. We simplified the unweighted criterion so that coercivity need only hold in absolute value, allowing the Bellman quadratic form to have either sign. We also relaxed other assumptions, including the comparability condition between candidate ratios and the true ratio for A-PBV.

  4. Theory for tempered weights. A reviewer asked how the guarantees for estimated occupancy ratios apply to the tempered weights used in our experiments. Tempering is a standard way to stabilize importance weights, and our experiments directly follow Sinha et al. (2022), who also use tempering in their experiments and provide theory for the accurate-weight case. To address this concern, we added finite-sample guarantees for inexact, tempered weights under approximate coercivity. We also evaluated empirical coercivity along candidate differences and found that, in 211 of 220 DeepOPE settings, the typical comparison exceeds the worst-case bound guaranteed under exact discounted target-occupancy weighting. Thus, the tempered weights show stronger empirical coercivity along these candidate comparisons than that worst-case guarantee.

  5. Independent-rollout check. We reproduced the Hopper benchmark released by the authors of LSTD-Tournament, our closest comparator, following their use of cached Monte Carlo predictions. In response to a reviewer’s question about rollout reuse, we added matched comparisons using independent rollout sets. Across 18 prespecified model-based settings, PBV has the lowest aggregate error on both metrics under either construction; splitting reduces its policy-value error by 14.8% and initial-state RMSE by 22.8%. Across 12 prespecified model-free settings, PBV also outperforms the normalized LSTD-Tournament implementation used in our main comparison on both metrics.

These revisions provide a more direct theoretical comparison, weaker assumptions, and substantially expanded experimental evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.