- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Mixed-AC combines one-step and two-step Bellman residuals in a minimax offline value estimator. Its critic uses current-state features together with predicted future features learned on separate trajectories. The paper gives contraction, identification, and conditional objective-concentration results, then evaluates the estimator in two synthetic environments and four MetaWorld tasks, with ablations of residual mixing and critic representation.
Critical strengths
-
The predictive critic is a useful construction: it incorporates transition information but remains a function of the current state. At zero mixing, it reduces MSE relative to the current-state critic by about 24%, 83%, and 85% on Drawer, Button, and Faucet (Table 7).
-
The empirical reporting is careful. The paper includes 100 repetitions, paired comparisons, unsuccessful offline selection rules, and explicit limits on the theory. This makes both the positive results and the method's limitations easier to assess.
Critical weaknesses
-
Residual mixing adds relatively little in several tasks. Door improves from 0.08361 to 0.08180, while Drawer favors no mixing; Faucet provides stronger evidence. The common-studentized selector in Table 9 is worse on average than zero mixing in all six environments. Thus, the practical case for combining the two residuals remains incomplete. The contraction result alone does not resolve this, as the paper correctly acknowledges.
-
The baseline comparison does not isolate the proposed components. On Door, neither the adaptive critic nor mixing explains the large gap over FQE. Representation and estimator differences could matter, and the methods use markedly different tuning grids. Moreover, representation and regularization settings are selected using true-value pilot MSE (E.9). The pilots are independent, but the complete tuning procedure still requires information unavailable in a purely offline deployment. I therefore lean toward rejection despite the promising critic results.
Questions for authors
-
Clarification: Without simulator value truth, how would you choose the complete configuration, including representation, regularization, and mixing? Is there a fixed configuration that retains the main gains?
-
Targeted check: Could you compare against linear FQE or LSTD using the same value features and fitting data, and check convergence of the existing FQE regressions? This would help identify what accounts for the large Door improvement beyond the small mixing effect.
4 comments
Response to Reviewer 5uX9 (2/2)
Q2. We fit importance-weighted state-value LSTD and fitted iteration on Mixed-AC's frozen value features, built on the same 43 trajectories, and the same 42 coefficient-fitting trajectories, with 28 configurations each (24 for Mixed-AC), selected in the same way. LSTD and fitted iteration agree to the displayed precision.
| Task | Mixed-AC | LSTD / fitted iteration | 95% CI, LSTD − Mixed-AC |
|---|---|---|---|
| Door | 0.121 | 0.148 | [−0.0025, 0.0485] |
| Drawer | 0.0555 | 0.0937 | [0.0131, 0.0499] |
| Button | 0.00375 | 0.00475 | [−0.00068, 0.00225] |
| Faucet | 0.203 | 0.223 | [−0.112, 0.077] |
Mixed-AC has lower MSE on all four environments, by about 18% on Door, 41% on Drawer, 21% on Button and 9% on Faucet. Only the Drawer interval excludes zero. LSTD here is the moment estimator with the current-state critic, so this difference is what the predictive critic and residual mixing add under the same representation. The value representation is thus strong even with standard linear estimation, and the Mixed-AC fit improves on it, clearly so on Drawer.
We checked FQE convergence on Door. Across 1,800 frozen-target checks, 2,000 additional regression updates changed the evaluation predictions by at most 1.45% in relative L2 norm, and raising the outer iterations from 100 to 400, or the inner updates fourfold, did not improve mean Door MSE. The stopping point mattered more, so we added iteration 10 to FQE's grid. This lowers its Door MSE from 0.678 to 0.272, against 0.121 for Mixed-AC. The original Door gap therefore reflects mainly the value representation, regularization and FQE's stopping point, not a large mixing or prediction effect.
We hope this addresses your concerns, and we would be grateful if you would reconsider your score.
References
- Liu, P., Zhao, L., Agarwal, S., Liu, J., Huang, A., Amortila, P., and Jiang, N. Model selection for off-policy evaluation: New algorithms and experimental protocol. NeurIPS 2025.
- Tang, S., and Wiens, J. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. MLHC 2021, PMLR 149:2–35.
- van der Laan, L., Syrgkanis, V., and Kallus, N. Model selection for off-policy evaluation via pairwise Bellman validation. arXiv:2610.35748, 2026.
- Xie, T., and Jiang, N. Batch value-function approximation with only realizability. ICML 2021, PMLR 139:11404–11413.
- Zhang, S., and Jiang, N. Towards hyperparameter-free policy selection for offline reinforcement learning. NeurIPS 2021.
- Zitovsky, J. P., de Marchi, D., Agarwal, R., and Kosorok, M. R. Revisiting Bellman errors for offline model selection. ICML 2023, PMLR 202:43369–43406.
Thank you for your comprehensive response. I have raised my score to 4.
Thank you for reading our response and for raising your score. We appreciate your time and constructive comments.
Response to Reviewer 5uX9 (1/2)
Thank you for the thoughtful review.
W1. Mixed-AC is a family of estimators indexed by α, and α = 0 is a valid member of it. At α = 0 the critic still uses the predicted one-step and two-step features, so transition information enters without two-step importance weights. The point of residual mixing is not that it always helps, but that in some more complex settings it does, and our framework provides a tunable parameter to achieve this. For example, on Faucet, α = 0.25 lowers MSE by about 17% (paired interval [−0.0472, −0.0026]), and with each α tuned separately, longer horizons lower MSE significantly on Button and Faucet. Where mixing does not help, as in Env I, α = 0 remains available within the family. We suspect α = 0 is a good choice in most simple settings. What the framework adds is that the best α changes from setting to setting, and in some settings neither end point, α = 0 or 1, is best. Choosing α offline is discussed under Q1.
W2. The Door gap is studied under Q2, and configuration choice without value truth under Q1. To check the effect of the critic in Table 7, we compared the predictive critic with a current-state critic of the same dimension, built from the same feature family. At α=0.5, the predictive critic lowers MSE by about 10% on Button and Faucet. On Door and Drawer the two critics perform about the same, with the predictive critic slightly worse. We will add this control next to Table 7.
Q1. In simpler situations, we think α=0 should perform good (although an intermediate α can be better) as the two step information also enters through the critic. And we understand that practical usability is important. However, selecting hyperparameters for off-policy evaluation is known to be hard and this is not unique to our setting. It is hard because the true value is never observed (Tang & Wiens, 2021; Liu et al., 2025). In Appendix E.16 we study two observable rules for choosing α. When the critic-scaled rule chooses α, Mixed-AC still has the lowest scalar policy-value error in five of six environments (Tables 1 and 10). The pointwise MSE also remains lower than that of FQE on three of the four MetaWorld tasks. With α chosen by this rule and a third of the data held out for validation, the MSE is 0.106 on Door, 0.0041 on Button and 0.222 on Faucet (Table 8), against 0.272, 0.0106 and 1.89 for FQE with an expanded grid (see Q2). On Drawer FQE is lower (0.066 against 0.092). These are means from separate runs and not a paired comparison. Recent work offers approaches that we would like to explore for Mixed-AC, such as pairwise Bellman validation (van der Laan et al., 2026), pairwise tournament selectors that compare candidate value functions two at a time (Xie & Jiang, 2021; Zhang & Jiang, 2021; Liu et al., 2025), and held-out validation based on more accurate Bellman-error estimates (Zitovsky et al., 2023). These selectors choose among a finite set of fitted value functions, which is what the α family provides. Finally, we would like to stress that our aim is not to show that a single α works in every setting. Our aim is to provide a framework in which α can be tuned, since the best α changes from one setting to another. This gives a tunable family from α=0 to α=1 rather than a fixed choice of the one-step or the two-step residual.
A nearly fixed configuration keeps the errors low on three of the four environments. For each task we took every hyperparameter from the pilot choices of the other three tasks (the majority choice, or the median when all three differ), so no value information from the evaluation task is used. This gives 64 random Fourier and 16 fixed neural features, neural-feature scaling 10, coefficient regularization 1, predictor width 32 and α = 0 on every task; only the predictor weight decay differs (10−4 for Door and Drawer, 10−5 for Button and Faucet). Over 100 fresh 85-trajectory subsamples, the MSE is 0.261 on Door, 0.334 on Drawer, 0.00510 on Button and 0.374 on Faucet. This remains below FQE tuned with true values on Door (0.261 against 0.272), Button (0.00510 against 0.0106) and Faucet (0.374 against 1.89). Drawer is the exception. There the transferred configuration differs from the pilot choice only in the coefficient regularization (1 instead of 0.1) and the predictor weight decay, and about three quarters of the MSE is squared bias (0.247 of 0.334). This suggests that Drawer is sensitive to the regularization.