acceptodds
Under review as a conference paper at ICLR 2027

Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations

Abstract

Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias–variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Off-Policy Evaluation, Bellman Residual, Minimax Estimation, Adaptive Critic, Multi-Step Bellman Operator

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_5uX9OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Mixed-AC combines one-step and two-step Bellman residuals in a minimax offline value estimator. Its critic uses current-state features together with predicted future features learned on separate trajectories. The paper gives contraction, identification, and conditional objective-concentration results, then evaluates the estimator in two synthetic environments and four MetaWorld tasks, with ablations of residual mixing and critic representation.

Critical strengths
  1. The predictive critic is a useful construction: it incorporates transition information but remains a function of the current state. At zero mixing, it reduces MSE relative to the current-state critic by about 24%, 83%, and 85% on Drawer, Button, and Faucet (Table 7).

  2. The empirical reporting is careful. The paper includes 100 repetitions, paired comparisons, unsuccessful offline selection rules, and explicit limits on the theory. This makes both the positive results and the method's limitations easier to assess.

Critical weaknesses
  1. Residual mixing adds relatively little in several tasks. Door improves from 0.08361 to 0.08180, while Drawer favors no mixing; Faucet provides stronger evidence. The common-studentized selector in Table 9 is worse on average than zero mixing in all six environments. Thus, the practical case for combining the two residuals remains incomplete. The contraction result alone does not resolve this, as the paper correctly acknowledges.

  2. The baseline comparison does not isolate the proposed components. On Door, neither the adaptive critic nor mixing explains the large gap over FQE. Representation and estimator differences could matter, and the methods use markedly different tuning grids. Moreover, representation and regularization settings are selected using true-value pilot MSE (E.9). The pilots are independent, but the complete tuning procedure still requires information unavailable in a purely offline deployment. I therefore lean toward rejection despite the promising critic results.

Questions for authors
  1. Clarification: Without simulator value truth, how would you choose the complete configuration, including representation, regularization, and mixing? Is there a fixed configuration that retains the main gains?

  2. Targeted check: Could you compare against linear FQE or LSTD using the same value features and fitting data, and check convergence of the existing FQE regressions? This would help identify what accounts for the large Door improvement beyond the small mixing effect.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5uX9 (1/2)

Thank you for the thoughtful review.

W1. Mixed-AC is a family of estimators indexed by α\alpha, and α\alpha = 0 is a valid member of it. At α\alpha = 0 the critic still uses the predicted one-step and two-step features, so transition information enters without two-step importance weights. The point of residual mixing is not that it always helps, but that in some more complex settings it does, and our framework provides a tunable parameter to achieve this. For example, on Faucet, α\alpha = 0.25 lowers MSE by about 17% (paired interval [−0.0472, −0.0026]), and with each α tuned separately, longer horizons lower MSE significantly on Button and Faucet. Where mixing does not help, as in Env I, α = 0 remains available within the family. We suspect α\alpha = 0 is a good choice in most simple settings. What the framework adds is that the best α changes from setting to setting, and in some settings neither end point, α\alpha = 0 or 1, is best. Choosing α offline is discussed under Q1.

W2. The Door gap is studied under Q2, and configuration choice without value truth under Q1. To check the effect of the critic in Table 7, we compared the predictive critic with a current-state critic of the same dimension, built from the same feature family. At α=0.5\alpha = 0.5, the predictive critic lowers MSE by about 10% on Button and Faucet. On Door and Drawer the two critics perform about the same, with the predictive critic slightly worse. We will add this control next to Table 7.

Q1. In simpler situations, we think α=0\alpha = 0 should perform good (although an intermediate α\alpha can be better) as the two step information also enters through the critic. And we understand that practical usability is important. However, selecting hyperparameters for off-policy evaluation is known to be hard and this is not unique to our setting. It is hard because the true value is never observed (Tang & Wiens, 2021; Liu et al., 2025). In Appendix E.16 we study two observable rules for choosing α\alpha. When the critic-scaled rule chooses α\alpha, Mixed-AC still has the lowest scalar policy-value error in five of six environments (Tables 1 and 10). The pointwise MSE also remains lower than that of FQE on three of the four MetaWorld tasks. With α\alpha chosen by this rule and a third of the data held out for validation, the MSE is 0.106 on Door, 0.0041 on Button and 0.222 on Faucet (Table 8), against 0.272, 0.0106 and 1.89 for FQE with an expanded grid (see Q2). On Drawer FQE is lower (0.066 against 0.092). These are means from separate runs and not a paired comparison. Recent work offers approaches that we would like to explore for Mixed-AC, such as pairwise Bellman validation (van der Laan et al., 2026), pairwise tournament selectors that compare candidate value functions two at a time (Xie & Jiang, 2021; Zhang & Jiang, 2021; Liu et al., 2025), and held-out validation based on more accurate Bellman-error estimates (Zitovsky et al., 2023). These selectors choose among a finite set of fitted value functions, which is what the α\alpha family provides. Finally, we would like to stress that our aim is not to show that a single α\alpha works in every setting. Our aim is to provide a framework in which α\alpha can be tuned, since the best α\alpha changes from one setting to another. This gives a tunable family from α=0\alpha = 0 to α=1\alpha = 1 rather than a fixed choice of the one-step or the two-step residual.

A nearly fixed configuration keeps the errors low on three of the four environments. For each task we took every hyperparameter from the pilot choices of the other three tasks (the majority choice, or the median when all three differ), so no value information from the evaluation task is used. This gives 64 random Fourier and 16 fixed neural features, neural-feature scaling 10, coefficient regularization 1, predictor width 32 and α\alpha = 0 on every task; only the predictor weight decay differs (10−410^{-4} for Door and Drawer, 10−510^{-5} for Button and Faucet). Over 100 fresh 85-trajectory subsamples, the MSE is 0.261 on Door, 0.334 on Drawer, 0.00510 on Button and 0.374 on Faucet. This remains below FQE tuned with true values on Door (0.261 against 0.272), Button (0.00510 against 0.0106) and Faucet (0.374 against 1.89). Drawer is the exception. There the transferred configuration differs from the pilot choice only in the coefficient regularization (1 instead of 0.1) and the predictor weight decay, and about three quarters of the MSE is squared bias (0.247 of 0.334). This suggests that Drawer is sensitive to the regularization.

AuthorsOpenReview ↗

Response to Reviewer 5uX9 (2/2)

Q2. We fit importance-weighted state-value LSTD and fitted iteration on Mixed-AC's frozen value features, built on the same 43 trajectories, and the same 42 coefficient-fitting trajectories, with 28 configurations each (24 for Mixed-AC), selected in the same way. LSTD and fitted iteration agree to the displayed precision.

TaskMixed-ACLSTD / fitted iteration95% CI, LSTD − Mixed-AC
Door0.1210.148[−0.0025, 0.0485]
Drawer0.05550.0937[0.0131, 0.0499]
Button0.003750.00475[−0.00068, 0.00225]
Faucet0.2030.223[−0.112, 0.077]

Mixed-AC has lower MSE on all four environments, by about 18% on Door, 41% on Drawer, 21% on Button and 9% on Faucet. Only the Drawer interval excludes zero. LSTD here is the moment estimator with the current-state critic, so this difference is what the predictive critic and residual mixing add under the same representation. The value representation is thus strong even with standard linear estimation, and the Mixed-AC fit improves on it, clearly so on Drawer.

We checked FQE convergence on Door. Across 1,800 frozen-target checks, 2,000 additional regression updates changed the evaluation predictions by at most 1.45% in relative L2 norm, and raising the outer iterations from 100 to 400, or the inner updates fourfold, did not improve mean Door MSE. The stopping point mattered more, so we added iteration 10 to FQE's grid. This lowers its Door MSE from 0.678 to 0.272, against 0.121 for Mixed-AC. The original Door gap therefore reflects mainly the value representation, regularization and FQE's stopping point, not a large mixing or prediction effect.

We hope this addresses your concerns, and we would be grateful if you would reconsider your score.

References

  • Liu, P., Zhao, L., Agarwal, S., Liu, J., Huang, A., Amortila, P., and Jiang, N. Model selection for off-policy evaluation: New algorithms and experimental protocol. NeurIPS 2025.
  • Tang, S., and Wiens, J. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. MLHC 2021, PMLR 149:2–35.
  • van der Laan, L., Syrgkanis, V., and Kallus, N. Model selection for off-policy evaluation via pairwise Bellman validation. arXiv:2610.35748, 2026.
  • Xie, T., and Jiang, N. Batch value-function approximation with only realizability. ICML 2021, PMLR 139:11404–11413.
  • Zhang, S., and Jiang, N. Towards hyperparameter-free policy selection for offline reinforcement learning. NeurIPS 2021.
  • Zitovsky, J. P., de Marchi, D., Agarwal, R., and Kosorok, M. R. Revisiting Bellman errors for offline model selection. ICML 2023, PMLR 202:43369–43406.

Thank you for your comprehensive response. I have raised my score to 4.

Thank you for reading our response and for raising your score. We appreciate your time and constructive comments.

Reviewer_ovCBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses offline policy evaluation (OPE), which aims to estimate the value of a target policy using data collected by a different behavior policy without further environment interaction. The main challenge considered is the trade-off between approximation error in one-step Bellman residuals and increased variance from multi-step importance weighting. To address this, the authors propose Mixed-AC, which combines one-step and two-step Bellman residuals through a mixing parameter and optimizes them using a minimax formulation. The method further introduces an adaptive critic that incorporates predicted future-state features to capture transition dynamics, along with sample splitting to mitigate overfitting.

Critical strengths
  1. Standard minimax OPE methods test the Bellman residual with a fixed kernel on the current state, using the same critic class regardless of the MDP. Mixed-AC instead builds a data-dependent critic from neural predictions of one- and two-step future features under the behavior policy.
  2. The convex combination of one- and two-step Bellman residuals offers a simple, tunable mechanism for trading approximation error against the variance of multi-step importance weighting.
  3. The experiments cover simulated environments and four MetaWorld manipulation tasks, with 100 repetitions, paired confidence intervals, and both pointwise and scalar comparisons against FQE, Kernel BRM/MQL, and three DICE variants.
Critical weaknesses
  1. Although mixing one-step and two-step Bellman residuals is presented as a central contribution, most improvements over the baselines are already achieved with α=0\alpha=0 (one-step residual with adaptive critic). Additional gains from intermediate α\alpha are modest, often comparable to the reported standard errors, and sometimes negative. This raises questions about the practical benefit of residual mixing beyond the adaptive critic.

  2. The experimental protocol selects hyperparameters using ground-truth target-policy values from pilot replications, which are generally unavailable in practical offline settings. Moreover, the Mixed-AC tuning grid is substantially larger than those of the baselines, raising concerns about comparison fairness. Meanwhile, the proposed offline validation criteria do not reliably identify a suitable α\alpha.

  3. The experiments do not fully isolate whether the improvements come from the adaptive critic's transition-aware representation, its increased feature dimension, or differences in feature representations and hyperparameter tuning across methods. Additional comparisons using matched-dimensional critics and consistent feature representations would help establish the specific contribution of the proposed adaptive critic.

Questions for authors
  1. Appendix E.9 indicates that hyperparameters are selected using ground-truth pointwise MSE from pilot replications, which is generally unavailable in practical OPE. Would Mixed-AC still outperform the baselines if all hyperparameters were selected using only offline data? Given the limitations of the proposed offline validation criteria in Appendix E.16, what would the authors recommend for selecting α\alpha in practice?

  2. Mixed-AC uses a substantially larger hyperparameter search space (approximately 2,500 configurations versus 24 for FQE and 9 for Kernel BRM/MQL) and a tuned hybrid RFF+MLP feature representation, whereas Kernel BRM/MQL uses a standard RBF kernel on raw states. Could the authors provide comparisons with matched hyperparameter tuning budgets and consistent feature representations? In particular, would the performance gap narrow if Kernel BRM/MQL were given access to the same hybrid feature map?

  3. The adaptive critic uses 3d3d moment conditions, compared with dd for the current-state critic in Table 7. Could the authors include a matched-dimensional critic without predictive future features (e.g., ϕ(s)\phi(s) augmented with random or nonlinear features of the current state) to determine whether the improvements arise from transition-aware representations rather than simply increasing the number of moment conditions?

  4. Mixed-AC with α=0\alpha=0 already outperforms the baselines in Env II–VI, while additional improvements from intermediate α\alpha values are often modest. Apart from the paired comparison already reported for Env VI, could the authors provide paired confidence intervals for the MSE differences between α=0\alpha=0 and intermediate mixing weights across the remaining environments? Furthermore, would the conclusions change if hyperparameters were tuned separately for each α\alpha?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ovCB (1/2)

Thank you for the review and the follow-up questions. We reran the FQE comparison with equal data and a larger FQE grid, ran the two controls suggested and computed paired intervals for the intermediate values of α.

W1. Mixed-AC is a family of estimators indexed by α\alpha and α\alpha = 0 is a valid member of it. The point of residual mixing is not that it always helps but that in some more complex settings it does, and our framework provides a tunable parameter to achieve this. For example, on Faucet, α\alpha = 0.25 lowers MSE by about 17% (paired interval [−0.0472, −0.0026]), and with each α\alpha tuned separately, longer horizons lower MSE significantly on Button and Faucet. Where mixing does not help, as in Env I, α\alpha = 0 remains available within the family. We suspect α\alpha = 0 is a good choice in most simple settings. What the framework adds is that the best α changes from setting to setting and in complex settings neither end point, α\alpha = 0 or 1, is best.

W2. We reran the comparison with FQE under a restricted budget. Both methods receive the same 85 trajectories, behavior densities are fit only within each run, and Mixed-AC searches 24 configurations against an expanded FQE grid of 36 configurations that includes early stopping.

TaskMixed-ACFQE95% CI, Mixed-AC − FQE
Door0.1210.272[−0.186, −0.085]
Drawer0.05550.0658[−0.0216, 0.0168]
Button0.003750.0106[−0.00801, −0.00481]
Faucet0.2031.89[−1.763, −1.570]

Mixed-AC has lower MSE than FQE on all four tasks. It reduces MSE by about 56% on Door, 65% on Button and 89% on Faucet, with intervals that exclude zero. On Drawer it is about 16% lower, but that interval includes zero, so the Drawer difference is not significant. Selection without true values is discussed under Q1.

W3. We ran both controls you suggest; they are reported under Q2 (same feature map for the kernel baseline) and Q3 (dimension-matched critic).

Q1. In simpler situations, we think α=0\alpha = 0 should perform good (although an intermediate α\alpha can be better) as the two step information also enters through the critic. And we understand that practical usability is important. However, selecting hyperparameters for off-policy evaluation is known to be hard and this is not unique to our setting. It is hard because the true value is never observed (Tang & Wiens, 2021; Liu et al., 2025). In Appendix E.16 we study two observable rules for choosing α\alpha. When the critic-scaled rule chooses α\alpha, Mixed-AC still has the lowest scalar policy-value error in five of six environments (Tables 1 and 10). The pointwise MSE also remains lower than that of FQE on three of the four MetaWorld tasks: 0.106 against 0.272 on Door, 0.0041 against 0.0106 on Button and 0.222 against 1.89 on Faucet (Table 8 and W2). On Drawer FQE is lower (0.066 against 0.092). A configuration that uses no value information from the evaluation task gives the same conclusion, with MSE 0.261 on Door, 0.00510 on Button, 0.374 on Faucet and 0.334 on Drawer (details in our reply to Reviewer 5uX9). Recent work offers approaches that we would like to explore for Mixed-AC, such as pairwise Bellman validation (van der Laan et al., 2026), pairwise tournament selectors that compare candidate value functions two at a time (Xie & Jiang, 2021; Zhang & Jiang, 2021; Liu et al., 2025), and held-out validation based on more accurate Bellman-error estimates (Zitovsky et al., 2023). Finally, we would like to stress that our aim is not to show that a single α\alpha works in every setting. Our aim is to provide a framework in which α\alpha can be tuned, since the best α\alpha changes from one setting to another. This gives a tunable family from α=0\alpha = 0 to α=1\alpha = 1 rather than a fixed choice of the one-step or the two-step residual.

References

  • Liu, P., Zhao, L., Agarwal, S., Liu, J., Huang, A., Amortila, P., and Jiang, N. Model selection for off-policy evaluation: New algorithms and experimental protocol. NeurIPS 2025.
  • Tang, S., and Wiens, J. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. MLHC 2021, PMLR 149:2–35.
  • van der Laan, L., Syrgkanis, V., and Kallus, N. Model selection for off-policy evaluation via pairwise Bellman validation. arXiv:2610.35748, 2026.
  • Xie, T., and Jiang, N. Batch value-function approximation with only realizability. ICML 2021, PMLR 139:11404–11413.
  • Zhang, S., and Jiang, N. Towards hyperparameter-free policy selection for offline reinforcement learning. NeurIPS 2021.
  • Zitovsky, J. P., de Marchi, D., Agarwal, R., and Kosorok, M. R. Revisiting Bellman errors for offline model selection. ICML 2023, PMLR 202:43369–43406.
AuthorsOpenReview ↗

Response to Reviewer ovCB (2/2)

Q2. The same budget comparison with FQE is reported under W2. Note that Kernel BRM and MQL fix a kernel for the critic and this kernel in turn determines a feature map. Our Mixed-AC works in the opposite direction. We have a feature map ϕ\phi from which we construct a data-dependent critic z=[ϕ,γm^1,γ2m^2]z = [\phi, \gamma \widehat m_1, \gamma^2 \widehat m_2], which defines a data-dependent kernel z(s)⊤z(t)z(s)^\top z(t). When the kernel baselines use our feature map through the linear kernel ϕ(s)⊤ϕ(t)\phi(s)^\top \phi(t), they solve the same moment equation as importance-weighted LSTD on ϕ\phi, up to regularization. We therefore fit LSTD on the Mixed-AC value features with the same fitting data and a comparable budget (28 configurations against 24). Mixed-AC has lower MSE on all four tasks, 0.121 against 0.148 on Door, 0.0555 against 0.0937 on Drawer, 0.00375 against 0.00475 on Button and 0.203 against 0.223 on Faucet. Only the Drawer interval excludes zero (table in our reply to Reviewer 5uX9). The two estimators then differ only in the critic, and we study its effect in Q3. At α=0\alpha = 0 the critic still contains the predicted one-step and two-step features, so information about the transitions still exists.

Q3. We agree that a same dimension comparison can isolate the effect of transition information. However, augmenting the value features with independent random features of the current state would change the feature map, because it adds new full-rank directions of a different kind. To keep the feature map the same, we compared the predictive critic with a current-state critic from the same feature family as ϕ\phi extended to the same 3p dimension, i.e., the same random Fourier and MLP features with same bandwidth. We ran this control at α = 0.5 as it highlights the mixing better. Paired over 100 repetitions:

TaskPredictiveExtended current-stateExtended − predictive [95% CI]
Door0.07950.0793−0.0002 [−0.0020, 0.0016]
Drawer0.05130.0489−0.0024 [−0.0036, −0.0012]
Button0.003690.00411+0.00041 [0.00020, 0.00062]
Faucet0.13570.1504+0.0147 [0.0067, 0.0227]

On Button and Faucet the predictive critic is significantly better, lowering MSE by about 10% on each and winning in 76% and 69% of repetitions. On Door the two critics are indistinguishable. On Drawer the extended current-state critic is slightly better, by about 5%. This suggests that the transition information carried by the adaptive critic can help in more complex settings. We will add this control to Table 7.

Q4. Each entry is MSE(α) − MSE(0) at the submitted settings, with a paired 95% interval over 100 repetitions.

Env.α = 0.25α = 0.5α = 0.75
I+0.0433 [0.0167, 0.0700]+0.191 [0.132, 0.250]+0.416 [0.323, 0.509]
II−0.00021 [−0.00055, 0.00013]+0.00094 [0.00036, 0.00152]+0.00316 [0.00240, 0.00393]
III: Door−0.00177 [−0.00499, 0.00144]−0.00181 [−0.00735, 0.00372]−0.00111 [−0.00840, 0.00617]
IV: Drawer+0.00116 [−0.00162, 0.00393]+0.00571 [0.00062, 0.01080]+0.0119 [0.0049, 0.0189]
V: Button−0.00021 [−0.00108, 0.00067]+0.00030 [−0.00092, 0.00152]+0.00085 [−0.00052, 0.00221]
VI: Faucet−0.0249 [−0.0472, −0.0026]−0.0139 [−0.0468, 0.0190]+0.0016 [−0.0369, 0.0401]

At α=0.25\alpha = 0.25, MSE is lower than at α=0\alpha = 0 in four of the six environments, and significantly so on Faucet, where the reduction is about 17%. Tuning separately for each α\alpha makes the benefit clearer on two tasks. Setting α=0.25\alpha = 0.25 lowers the MSE on Button (difference −0.00147-0.00147, interval [−0.00286,−0.00007][-0.00286, -0.00007]), and α=1\alpha = 1 lowers it on Faucet (−0.0611-0.0611, [−0.0931,−0.0292][-0.0931, -0.0292]). On Drawer the differences include zero, and on Door α=1\alpha = 1 raises the MSE by 0.05750.0575 [0.0213,0.0938][0.0213, 0.0938]. In the equal-data comparison, best α\alpha is, α=0.25\alpha = 0.25 on Door and Button, α=0.5\alpha = 0.5 on Faucet and α=0\alpha = 0 on Drawer. The conclusion is therefore that the best α\alpha depends on the environment. In general, α=0\alpha = 0 can be a very good choice in most simple environments (although may not be the best) and the two step information already enters through the critic. Our main motivation is to provide a framework in which the experimenter can tune α\alpha according to the environment.

We hope this addresses your concerns, and we would be grateful if you would reconsider your score.

We would like to thank you again for your constructive review and feedback. We hope our response has addressed your questions. As the discussion period nears its end, we would be glad to provide any further clarification if needed.

Reviewer_toaZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I view this Mixed-AC paper as an off-policy evaluation (OPE) study. It combines one-step and two-step Bellman residuals in proportion to α\alpha, constructs a critic by predicting features for the next step and the next two steps, and then performs moment-based estimation. What I want to assess is: compared to existing compound-return, LSTD, FQE, and GMM-based OPE methods, what specific advantages do this mixed residual and the predictive critic each bring, rather than simply noting that a particular row of MSE is lower.

Critical strengths

S1/S2: I acknowledge the authors’ attempt to transform the learned critic into a solvable regression-based moment objective, thereby avoiding the iterative alternating training of the value function and the critic; the experiments also cover synthetic problems and the MetaWorld multi-task setting. However, I believe whether these designs are superior to existing methods depends on rigorous baselines and equivalent data budgets.

Critical weaknesses

W1: I initially had doubts about the novelty of the method. First, the convex combination of one- and two-step TD residuals is related to compound returns; if the paper claims that predicting future features is particularly important, I would expect this to be explicitly reflected in the value error bounds, rather than merely providing empirical moment control with a fixed critic. Furthermore, it remains to be verified whether the method outperforms learned-instrument and GMM estimators within the same class of value functions.

W2: What concerns me most is the initial experimental setup: the behavior-policy density was trained on the full dataset, while certain OPE baselines used only a subset, and the hyperparameter search budgets were not aligned. With such a comparison, even if the MSE is significantly reduced, I cannot definitively attribute the improvement to Mixed-AC.

Questions for authors

Q1: I would like to compare the predictive critic with neural GMM and traditional moment estimation within the same value function class, using the same data and hyperparameter tuning budgets.

Q2: I want to know how much of the original advantage remains after strictly limiting each run to 85 trajectories and estimating the behavior density only on the data from that run.

5 comments
AuthorsOpenReview ↗

Response to Reviewer toaZ (1/2)

Thank you for the detailed review. We derive a bound that connects the feature prediction error to the value error, added a comparison with Deep GMM, and reran the FQE comparison.

W1

Relation to compound returns. Daley et al. (2024) study compound returns, which are basically convex combinations of nn-step returns. Although it is true that we also use convex combinations, our main contribution is to use the mixed one and two-step return as the Bellman residual in a moment estimator and to test this residual with a data-dependent critic. At α=0\alpha = 0 the critic still uses the predicted one-step and two-step features, so transition information enters without two-step importance weights. With fixed features, the empirical moment is linear in the value coefficients, q^+M^θ\widehat q+\widehat M\theta, so for λ>0\lambda>0 the coefficients solve

(M^⊤M^+λI)θ^=−M^⊤q^.(\widehat M^\top\widehat M+\lambda I)\widehat\theta=-\widehat M^\top\widehat q.

Nonlinear state features and a learned critic therefore require no alternating optimization between critic and value function and no sample-by-sample Gram matrix. We will cite Daley et al. (2024) in the final version.

Prediction accuracy and value error. We will add the following derivation to the paper. Let zz be the population predictive critic and

gα=(1−α)γρ0ϕ(S′)+αγ2ρ01ϕ(S′′)−ϕ(S),Mz=E[z(S)gα⊤],g_\alpha=(1-\alpha)\gamma\rho_0\phi(S')+\alpha\gamma^2\rho_{01}\phi(S'')-\phi(S),\qquad M_z=E[z(S)g_\alpha^\top],

with σ∗=σmin⁡(Mz)\sigma_\ast=\sigma_{\min}(M_z), Gα=∥gα∥L2G_\alpha=\Vert g_\alpha\Vert_{L^2} and ez=∥z^−z∥L2(μb)e_z=\Vert\widehat z-z\Vert_{L^2(\mu_b)}. The critic error is equal to the weighted feature-prediction error, ez2=γ2E∥m^1−m1b∥22+γ4E∥m^2−m2b∥22e_z^2=\gamma^2E\Vert\widehat m_1-m_1^b\Vert_2^2+\gamma^4E\Vert\widehat m_2-m_2^b\Vert_2^2. Under realizability (Vπ=θ∗⊤ϕV^\pi=\theta^{\ast\top}\phi) and σ∗>ezGα\sigma_\ast>e_zG_\alpha,

∥V^−Vπ∥∞≤Bϕλ∥θ∗∥22+ΔNσ∗−ezGα,\Vert\widehat V-V^\pi\Vert_\infty\le\frac{B_\phi\sqrt{\lambda\Vert\theta^\ast\Vert_2^2+\Delta_N}}{\sigma_\ast-e_zG_\alpha},

where λ∥θ∗∥22\lambda\Vert\theta^\ast\Vert_2^2 is the regularization and ΔN\Delta_N is the sum of the error from estimated importance weights and the optimization. We bound ∥Mz^−Mz∥op≤ezGα\Vert M_{\widehat z}-M_z\Vert_{\mathrm{op}}\le e_zG_\alpha by the Cauchy--Schwarz inequality and apply a singular-value perturbation bound. Accurate prediction therefore preserves the identification provided by the population critic.

The bound also shows why the predicted features can help. The matrix MzM_z contains the rows of the current-state critic together with the rows of the predicted features so σ∗\sigma_\ast is at least as large as the smallest singular value under the current-state critic alone. Adding the predicted features therefore cannot weaken identification. There is also another reason to use predicted future features. Note that the residual is linear in θ\theta with slope gαg_\alpha, and

E[gα∣S]=(1−α)γ m1π(S)+αγ2m2π(S)−ϕ(S),E[g_\alpha\mid S]=(1-\alpha)\gamma\thinspace m_1^\pi(S)+\alpha\gamma^2 m_2^\pi(S)-\phi(S),

where m1πm_1^\pi and m2πm_2^\pi are the conditional feature means under the target policy. Under homoskedastic residuals this is the optimal instrument for the conditional moment restriction (Chamberlain, 1987). For every α\alpha it is a linear combination of the three components of our critic, ϕ\phi, γm1\gamma m_1 and γ2m2\gamma^2 m_2. Our critic uses conditional means under the behavior policy, so it has this form exactly when the two policies agree.

Learned-critic and instrumental-variable methods. We will discuss Chen et al. (2022) and related methods in the final version. Q1 reports a comparison with Deep GMM (Bennett et al., 2019), one of the methods studied there.

W2. We reran our experiments with density estimation restricted to each run, equal data and an expanded FQE grid. The results are under Q2.

AuthorsOpenReview ↗

Response to Reviewer toaZ (2/2)

Q1. We compared with Deep GMM (Bennett et al., 2019) in the formulation of Chen et al. (2022), with its own Q(s, a) network, on the same 100 repetitions and 400 evaluation states as Table 1:

TaskMixed-AC (α = 0)Deep GMM
Door0.0840.379
Drawer0.0590.031
Button0.00340.0067
Faucet0.1441.455

Mixed-AC has lower error on Door, Button and Faucet while Deep GMM is lower on Drawer.

We also tried the comparison you describe with the same value class and the same tuning grid. We kept Mixed-AC at α=0\alpha = 0 the same (i.e., the same features, the same linear value class, the same trajectories and the same configurations) and replaced only the predictive critic by a neural network of the current state trained adversarially. Unfortunately we could not train this critic reliably. When the value coefficients were solved in closed form for each critic, the fitted values were close to zero on all four environments, so the MSE was close to the squared true value. The standard min-max formulation with alternating gradient updates gave the same result on the Door repetition that we tried, with up to 15,000 rounds. The predictive critic does not require adversarial training, since it is fit by regression on separate trajectories and then fixed.

Q2. We reran with both methods receiving the same 85 trajectories and the behavior density is fit only on each run's own data. Mixed-AC searches 24 configurations and FQE an expanded grid of 36 configurations that includes early stopping. The 24 configurations are a full grid over the scale of the MLP features (4 values) and the ridge penalty (6 values), fixed before any run and shared across all tasks.

TaskMixed-ACFQE95% CI, Mixed-AC − FQE
Door0.1210.272[−0.186, −0.085]
Drawer0.05550.0658[−0.0216, 0.0168]
Button0.003750.0106[−0.00801, −0.00481]
Faucet0.2031.89[−1.763, −1.570]

Mixed-AC has lower MSE than FQE on all four tasks. It significantly reduces MSE by about 56% on Door, 65% on Button and 89% on Faucet, with intervals that exclude zero. On Drawer it is about 16% lower, but that interval includes zero, so the Drawer difference is not significant. Compared with Table 1 the difference narrows on Door and Drawer, where the MSE of FQE falls from 0.695 to 0.272 and from 0.089 to 0.066 in the new comparison. It remains large on Button and Faucet.

We hope this addresses your concerns, and we would be grateful if you would reconsider your score.

References

  • Bennett, A., Kallus, N., and Schnabel, T. Deep generalized method of moments for instrumental variable analysis. NeurIPS 2019.
  • Chamberlain, G. Asymptotic efficiency in estimation with conditional moment restrictions. Journal of Econometrics 34(3):305–334, 1987.
  • Chen, Y., Xu, L., Gulcehre, C., Paine, T. L., Gretton, A., de Freitas, N., and Doucet, A. On instrumental variable regression for deep offline policy evaluation. JMLR 23(302):1–40, 2022.
  • Daley, B., White, M., and Machado, M. C. Averaging n-step returns reduces variance in reinforcement learning. ICML 2024, PMLR 235:9904–9930.
Reviewer_toaZOpenReview ↗

Score update after the controlled comparisons

The rerun with 85 trajectories for both methods, density estimation within each run, and the 24-versus-36 configuration comparison resolves most of my concern about fairness. The gains on Door, Button and Faucet deserve credit. So do the roughly 10% improvements from the dimension-matched predictive critic on two tasks, and the transferred configurations that beat FQE on three of four tasks. I will raise my score from 1 to 2. What keeps me at 2 is the size of the contribution beyond existing moment-estimation and compound-return methods. Under the closest value-feature control, the LSTD comparison is statistically resolved only on Drawer, and the same-value-class neural-GMM comparison remains limited by optimization. I see a promising method, but the evidence for the specific new components is still mixed. The value-error bound helps under its realizability and identification assumptions; please include the sampling-concentration term alongside ratio-estimation and optimization errors, and state the independent sampling unit. That would make the remaining theoretical claim easier to assess.

Reviewer_X8i4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Mixed-AC combines one-step and two-step Bellman residuals in a minimax offline policy evaluation objective. Its critic augments current-state features with predicted future feature directions, learned on a separate data split. The paper gives population identification and finite-sample stability results, then evaluates pointwise and scalar value error in simulations and four MetaWorld tasks.

Critical strengths

The paper carefully distinguishes the full critic class used for identification from its restricted implemented class, and acknowledges the difficulty of selecting the mixing weight offline.

Critical weaknesses

The finite-sample result assumes exact, unnormalized, bounded importance ratios, whereas the MetaWorld implementation estimates behavior densities and self-normalizes weights. The practical estimator is therefore outside the stated guarantee. The contraction bound concerns a population residual; it does not imply lower estimation error once two-step importance-weight variance and critic misspecification are included. The best fixed mixing weight in the main table is selected post hoc using true MSE. The paper reports that its observable validation rules do not reliably recover this choice. A deployable offline selection rule is still missing. The adaptive critic uses behavior-policy future features, which may poorly reflect target-policy dynamics under strong mismatch. Baseline comparisons also differ in data usage and tuning budgets, while MetaWorld reference values are Monte Carlo approximations whose error should be quantified.

Questions for authors
  1. Which guarantees survive estimated and self-normalized importance ratios, and how large is their bias?
  2. How should the mixing weight be chosen without access to true values?
  3. What is the error of the Monte Carlo reference values, and are all methods given comparable data and tuning budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer X8i4 (1/2)

Thank you for the review and the follow-up questions. In response we extend Proposition 2 to the estimated, self-normalized weights used in practice, added a value error bound, and reran the FQE comparison.

Weaknesses

Contraction. It is true that contraction alone does not imply lower estimation error, as noted after Eq. (7) in our paper and we will add an explicit value error bound. Under realizability (Vπ=θ∗⊤ϕV^\pi=\theta^{\ast\top}\phi) and identification (σmin⁡(Mz^)>0\sigma_{\min}(M_{\widehat z})>0), we show

∥V^−Vπ∥∞≤Bϕλ∥θ∗∥22+ΔNσmin⁡(Mz^),\Vert\widehat V-V^\pi\Vert_\infty\le\frac{B_\phi\sqrt{\lambda\Vert\theta^\ast\Vert_2^2+\Delta_N}}{\sigma_{\min}(M_{\widehat z})},

where λ∥θ∗∥22\lambda\Vert\theta^\ast\Vert_2^2 is the regularization, ΔN\Delta_N is the sum of the error from estimated importance weights and the optimization error. Note that the term ΔN\Delta_N depends on α\alpha through the two-step weights and σmin⁡(Mz^)\sigma_{\min}(M_{\widehat z}) already depends on α\alpha because Mz^M_{\widehat z} is defined using the mixed residual. Unlike the contraction result, this bound clearly shows the effect of α\alpha in the value error. At α=0\alpha = 0 the critic still uses the predicted one-step and two-step features, so transition information is still there without two-step importance weights.

Behavior-policy mismatch. Note that the critic does not need to depend on the target policy. Any function of the current state still gives a valid moment condition and with exact ratios, the moment at VπV^\pi is zero for a critic learned under any policy. Therefore, conditional means under the target policy are also valid. We chose the behavior policy because it is simpler to compute as it needs no additional importance weights. We also tried (in the simulated MDP) replacing the conditional means with target-policy and the MSE changes by at most 0.0014 and no difference is significant after Holm correction.

Q1. We extended Proposition 2 to the weights used in practice. Let ρ01=ρ0ρ1\rho_{01}=\rho_0\rho_1, and let ρ~0\widetilde\rho_0 and ρ~01\widetilde\rho_{01} for the estimated, self-normalized weights. Define

εn=1n∑i=1n{c1 ∣ρ~0,i−ρ0,i∣+c2 ∣ρ~01,i−ρ01,i∣},\varepsilon_n=\frac1n\sum_{i=1}^n\Big\lbrace c_1\thinspace|\widetilde\rho_{0,i}-\rho_{0,i}|+c_2\thinspace|\widetilde\rho_{01,i}-\rho_{01,i}|\Big\rbrace,

where c1=Rmax⁡+(1−α)γRθBϕc_1=R_{\max}+(1-\alpha)\gamma R_\theta B_\phi and c2=αγ(Rmax⁡+γRθBϕ)c_2=\alpha\gamma(R_{\max}+\gamma R_\theta B_\phi). Under the assumptions of Proposition 2, the bound has an extra term 2BZ2(2Br+εn)εn+ηopt2B_Z^2(2B_r+\varepsilon_n)\varepsilon_n+\eta_{\mathrm{opt}}, where ηopt\eta_{\mathrm{opt}} is the optimization error. The value error bound above then holds along with ΔN\Delta_N. To see this, note that the residual is linear in the two weights, so ∥m~−m^∥2≤BZεn\Vert\widetilde m-\widehat m\Vert_2\le B_Z\varepsilon_n for every θ\theta and the estimated weights do not need to be bounded. Because the exact ratios have mean one under the behavior policy, self-normalization adds an Op(n−1/2)O_p(n^{-1/2}) term and the error from πb^\hat{\pi_b} enters through εn\varepsilon_n. The bounded-ratio assumption on the exact ratios also holds in our benchmark. The behavior policy is (1−ε)π+εu(1-\varepsilon)\pi+\varepsilon u (Appendix E.3), so ρ0≤(1−ε)−1\rho_0\le(1-\varepsilon)^{-1} and ρ01≤(1−ε)−2\rho_{01}\le(1-\varepsilon)^{-2}. These bounds equal 5 and 25 for Door and 2.5 and 6.25 for the other tasks.

Empirically, we compared exact and estimated behavior ratios, both self-normalized, over 100 new 85-trajectory subsamples per environment, with a fixed configuration. Estimating the ratios changes mean MSE by at most 0.0068 across the four tasks and α∈{0,0.25,0.5,1}\alpha\in\lbrace0,0.25,0.5,1\rbrace. Empirical squared bias (Appendix E.4) at α = 0:

TaskExact ratiosEstimated ratios
Door0.21550.2117
Drawer0.24210.2472
Button0.00027280.0002947
Faucet0.075490.07754

Self-normalization trades some bias for a much larger reduction in variance. On Drawer, normalizing the exact ratios raises squared bias from 0.003363 to 0.2421 but lowers variance from 6.132 to 0.08633, so MSE reduces from 6.074 to 0.3275.

AuthorsOpenReview ↗

Response to Reviewer X8i4 (2/2)

Q2. In simpler situations, we think α=0\alpha = 0 should perform good (although an intermediate α\alpha can be better) as the two step information also enters through the critic. And we understand that practical usability is important. However, selecting hyperparameters for off-policy evaluation is known to be hard and this is not unique to our setting. It is hard because the true value is never observed (Tang & Wiens, 2021; Liu et al., 2025). In Appendix E.16 we study two observable rules for choosing α\alpha. When the critic-scaled rule chooses α\alpha, Mixed-AC still has the lowest scalar policy-value error in five of six environments (Tables 1 and 10). The pointwise MSE also remains lower than that of FQE on three of the four MetaWorld tasks. With α\alpha chosen by this rule and a third of the data held out for validation, the MSE is 0.106 on Door, 0.0041 on Button and 0.222 on Faucet (Table 8), against 0.272, 0.0106 and 1.89 for FQE (see Q3). On Drawer FQE is lower (0.066 against 0.092). We also considered a configuration that uses no value information from the evaluation. Every hyperparameter is taken from the pilot choices of the other three tasks, with α=0\alpha = 0 (details in our reply to Reviewer 5uX9). Its MSE is 0.261 on Door, 0.00510 on Button and 0.374 on Faucet, against 0.272, 0.0106 and 1.89 for FQE tuned with true values. On Drawer FQE is lower (0.0658 against 0.334). Recent work provides approaches that we would like to explore for Mixed-AC, such as pairwise Bellman validation (van der Laan et al., 2026), pairwise tournament selectors that compare candidate value functions two at a time (Xie & Jiang, 2021; Zhang & Jiang, 2021; Liu et al., 2025), and held-out validation based on more accurate Bellman-error estimates (Zitovsky et al., 2023). Finally, we would like to stress that our aim is not to show that a single α\alpha works in every setting. Our aim is to provide a framework in which α\alpha can be tuned, since the best α\alpha changes from one setting to another. This gives a tunable family from α=0\alpha = 0 to α=1\alpha = 1 instead of a binary choice of the one-step or the two-step residual.

Q3. Each reference averages 50 target-policy rollouts of length 60 from each of 400 evaluation states, with γ=0.9\gamma = 0.9. The mean estimated sampling variance of these averages is 0.0159 (Door), 0.0026 (Drawer), 0.00018 (Button) and 0.0336 (Faucet). For predictions that do not depend on the reference noise, this adds the same constant to every method's expected MSE. It therefore does not change the expected differences between methods. Truncation at 60 steps contributes at most γ60Rmax⁡/(1−γ)≈0.01797Rmax⁡\gamma^{60}R_{\max}/(1-\gamma)\approx0.01797R_{\max}.

We reran the comparison with FQE. Both methods now receive the same 85 trajectories, with behavior densities fit only within each run. Mixed-AC searches 24 configurations and FQE an expanded grid of 36 configurations that includes early stopping.

TaskMixed-ACFQE95% CI, Mixed-AC − FQE
Door0.1210.272[−0.186, −0.085]
Drawer0.05550.0658[−0.0216, 0.0168]
Button0.003750.0106[−0.00801, −0.00481]
Faucet0.2031.89[−1.763, −1.570]

Mixed-AC has lower MSE than FQE on all four environments. It significantly reduces MSE by about 56% on Door, 65% on Button and 89% on Faucet, with intervals that exclude zero. On Drawer it is about 16% lower, but that interval includes zero, so the Drawer difference is not significant. Compared with Table 1 the difference narrows on Door and Drawer, where the MSE of FQE falls from 0.695 to 0.272 and from 0.089 to 0.066 in the new comparison. It remains large on Button and Faucet. The intervals come from 2,000 common bootstrap resamples that repeat configuration selection. The DICE baselines already use all 85 trajectories of each run, without sample splitting (Appendix E.18–E.19).

We hope this addresses your concerns, and we would be grateful if you would reconsider your score.

References

  • Liu, P., Zhao, L., Agarwal, S., Liu, J., Huang, A., Amortila, P., and Jiang, N. Model selection for off-policy evaluation: New algorithms and experimental protocol. NeurIPS 2025.
  • Tang, S., and Wiens, J. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. MLHC 2021, PMLR 149:2–35.
  • van der Laan, L., Syrgkanis, V., and Kallus, N. Model selection for off-policy evaluation via pairwise Bellman validation. arXiv:2610.35748, 2026.
  • Xie, T., and Jiang, N. Batch value-function approximation with only realizability. ICML 2021, PMLR 139:11404–11413.
  • Zhang, S., and Jiang, N. Towards hyperparameter-free policy selection for offline reinforcement learning. NeurIPS 2021.
  • Zitovsky, J. P., de Marchi, D., Agarwal, R., and Kosorok, M. R. Revisiting Bellman errors for offline model selection. ICML 2023, PMLR 202:43369–43406.

We would like to thank you again for your constructive review and feedback. We hope our response has addressed your questions. As the discussion period nears its end, we would be glad to provide any further clarification if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.