- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies fitted Q-iteration with function approximation. By estimating and using the occupancy ratio for importance sampling, the authors overcome the norm mismatch challenge in fitted Q-iteration and establish convergence without strong assumptions of the Bellman completeness type.
Critical strengths
The instability of fitted Q-iteration and Q-learning has long been recognized as a major challenge, and this paper makes a meaningful contribution toward addressing it. The paper is generally well written.
Critical weaknesses
My main concern is the novelty of this work relative to [1]. Norm mismatch has long been recognized as a fundamental challenge and bottleneck in fitted Q-iteration. In particular, if the distribution mismatch can be corrected, the algorithm will work. Since [1] already provides a method for estimating the occupancy ratio, which is similar to importance sampling, the novelty of the present work is not completely clear.
[1] van der Laan, L., & Kallus, N. (2026). Fitted Occupancy-Ratio Evaluation without Bellman Completeness. arXiv preprint arXiv:2607.05375.
Questions for authors
Could the authors elaborate on the contributions of this work relative to [1]?
[1] van der Laan, L., & Kallus, N. (2026). Fitted Occupancy-Ratio Evaluation without Bellman Completeness. arXiv preprint arXiv:2607.05375.
2 comments
The authors have addressed my concerns, and I have increased my score.
Response to Reviewer iLfs
Revised manuscript (anonymous PDF; key changes in blue): PDF
Response to Reviewer iLfs
Thank you for recognizing the contribution to stabilizing fitted value learning and the clarity of the paper.
Novelty and relationship to FORE.
Our contribution is a finite-sample analysis of occupancy-weighted FQE with estimated weights and general function approximation. We do not claim occupancy weighting or the estimation of occupancy ratios as new ideas. Rather, we establish guarantees for FQE when both the occupancy weights and Bellman regressions must be estimated from finite data, accounting explicitly for statistical error, function-class misspecification, and ratio-estimation error.
Importantly, FORE (van der Laan and Kallus, 2026) is a subsequent (unpublished) paper focused on occupancy-ratio estimation that explicitly builds on and cites the present work. Its analysis of occupancy-weighted FQE considers population Bellman regressions with estimated weights, yielding a perturbation bound that is first order in weight-estimation error and proportional to the uniform Bellman projection residual. It refers to our work for the additional statistical error arising from empirical Bellman regressions.
Our analysis goes further by characterizing how approximate realizability and approximate Bellman completeness mitigate weight-estimation error. Approximate completeness controls the Bellman projection residual uniformly, while approximate realizability provides sharper control along the FQE iterates. Under exact value-function realizability, this yields higher-order dependence on weight-estimation error even without Bellman completeness, rather than only the first-order interaction with the uniform projection residual in van der Laan and Kallus (2026).
Finally, combining our finite-sample FQE analysis with FORE gives end-to-end guarantees that account for approximation and estimation errors in both the value and occupancy-ratio classes. Under joint realizability, we obtain consistency without Bellman completeness of either class.
References.