acceptodds
Under review as a conference paper at ICLR 2027

Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

Abstract

Fitted -evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift its projected Bellman operator need not be contractive under the offline distribution. Consequently, value-function realizability alone does not ensure convergence, and existing analyses often impose Bellman completeness or related structural conditions. We study occupancy-weighted FQE and show that distribution weighting provides a different route to stability. Weighting each Bellman regression by the density ratio of a suitably discounted target-policy occupancy distribution to the offline distribution aligns the regression norm with the target-policy dynamics, making the population projected Bellman operator contractive. With exact occupancy weights, this removes the need for Bellman completeness. Our main contribution is a finite-sample analysis showing how this population stability extends to estimated weights, nonlinear function classes, and misspecification. The bounds separate finite-iteration, statistical, approximation, and ratio-estimation errors and show that ratio-estimation error enters through its interaction with the Bellman projection residual rather than as a separate additive term. This interaction can be controlled by approximate Bellman completeness or value-function realizability; under exact realizability, the effect of ratio-estimation error is higher order, while under exact Bellman completeness it vanishes. We also analyze an end-to-end procedure combining occupancy-weighted FQE with fitted occupancy-ratio evaluation, obtaining consistency under coverage and joint realizability of the value function and occupancy ratio without completeness assumptions on either class. Controlled experiments illustrate the contraction mechanism and the tradeoff between target alignment and coverage.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, off-policy evaluation, fitted Q-evaluation, Bellman completeness, occupancy ratios, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_iLfsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies fitted Q-iteration with function approximation. By estimating and using the occupancy ratio for importance sampling, the authors overcome the norm mismatch challenge in fitted Q-iteration and establish convergence without strong assumptions of the Bellman completeness type.

Critical strengths

The instability of fitted Q-iteration and Q-learning has long been recognized as a major challenge, and this paper makes a meaningful contribution toward addressing it. The paper is generally well written.

Critical weaknesses

My main concern is the novelty of this work relative to [1]. Norm mismatch has long been recognized as a fundamental challenge and bottleneck in fitted Q-iteration. In particular, if the distribution mismatch can be corrected, the algorithm will work. Since [1] already provides a method for estimating the occupancy ratio, which is similar to importance sampling, the novelty of the present work is not completely clear.

[1] van der Laan, L., & Kallus, N. (2026). Fitted Occupancy-Ratio Evaluation without Bellman Completeness. arXiv preprint arXiv:2607.05375.

Questions for authors

Could the authors elaborate on the contributions of this work relative to [1]?

[1] van der Laan, L., & Kallus, N. (2026). Fitted Occupancy-Ratio Evaluation without Bellman Completeness. arXiv preprint arXiv:2607.05375.

2 comments
AuthorsOpenReview ↗

Response to Reviewer iLfs

Revised manuscript (anonymous PDF; key changes in blue): PDF

Response to Reviewer iLfs

Thank you for recognizing the contribution to stabilizing fitted value learning and the clarity of the paper.

Novelty and relationship to FORE.

Our contribution is a finite-sample analysis of occupancy-weighted FQE with estimated weights and general function approximation. We do not claim occupancy weighting or the estimation of occupancy ratios as new ideas. Rather, we establish guarantees for FQE when both the occupancy weights and Bellman regressions must be estimated from finite data, accounting explicitly for statistical error, function-class misspecification, and ratio-estimation error.

Importantly, FORE (van der Laan and Kallus, 2026) is a subsequent (unpublished) paper focused on occupancy-ratio estimation that explicitly builds on and cites the present work. Its analysis of occupancy-weighted FQE considers population Bellman regressions with estimated weights, yielding a perturbation bound that is first order in weight-estimation error and proportional to the uniform Bellman projection residual. It refers to our work for the additional statistical error arising from empirical Bellman regressions.

Our analysis goes further by characterizing how approximate realizability and approximate Bellman completeness mitigate weight-estimation error. Approximate completeness controls the Bellman projection residual uniformly, while approximate realizability provides sharper control along the FQE iterates. Under exact value-function realizability, this yields higher-order dependence on weight-estimation error even without Bellman completeness, rather than only the first-order interaction with the uniform projection residual in van der Laan and Kallus (2026).

Finally, combining our finite-sample FQE analysis with FORE gives end-to-end guarantees that account for approximation and estimation errors in both the value and occupancy-ratio classes. Under joint realizability, we obtain consistency without Bellman completeness of either class.

References.

The authors have addressed my concerns, and I have increased my score.

Reviewer_82NUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies occupancy-weighted fitted Q-evaluation as an approach to obtain the statistical convergence without Bellman completeness. Its main contribution is a finite-sample analysis with estimated weights, characterizing how ratio-estimation error interacts with the Bellman projection residual. It also provides an end-to-end guarantee using fitted occupancy-ratio evaluation and illustrates the method on several controlled environments.

Critical strengths

The analysis identifies a useful connection between weight-estimation error, value-function approximation, and Bellman completeness.

The method preserves the regression structure of FQE. The controlled experiments illustrate the stabilization mechanism.

Critical weaknesses

The significance of the theoretical advance needs clearer justification. Changing the projection distribution to stabilize off-policy value learning is well established. In particular, generalized TD already provides a closely related discounted-weight contraction bound (e.g., Hallak et al., 2016). The estimated-weight finite-sample analysis is the main distinction, but I am not yet convinced that this constitutes a sufficiently substantial advance.

The guarantees have a restricted scope. The analysis assumes exact regression over closed convex function classes, excluding ordinary neural FQE. Higher-order robustness requires additional regularity of the Bellman projection residuals, while the end-to-end result assumes occupancy ratios bounded away from zero. These conditions limit the applicability of the conclusions/

The experiments provide limited evidence about the main contribution. Baird and Garnet use exact weights, leaving only one CartPole configuration to evaluate learned weights. Comparisons are limited to unweighted FQE, and the experiments do not directly examine the predicted interaction between ratio-estimation error and value-function misspecification

Questions for authors

see comments

4 comments
AuthorsOpenReview ↗

Response to Reviewer 82NU (Part 1 of 2)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for recognizing the connection between weight-estimation error and value-function approximation.

1. Significance of the theoretical contribution (Weakness 1).

We agree that occupancy weighting and discounted-weight contraction are established ideas, including in generalized TD. Our contribution is a finite-sample analysis of FQE with estimated occupancy weights and nonlinear function approximation. Existing finite-sample analyses of FQE typically require Bellman completeness or related assumptions. To our knowledge, prior work has not shown how learned occupancy weighting can replace Bellman completeness in FQE while accounting for estimation error in both the weights and Bellman regressions.

Our analysis goes beyond the exact-weight contraction by characterizing how weight-estimation error interacts with the Bellman projection residual. We show that approximate Bellman completeness controls this residual uniformly over the value class, while approximate realizability controls it along the FQE iterates through the current Q-function error and the approximation error of QπQ^\pi. Under our interpolation condition, for Hellinger-type weight error ϵw\epsilon_w, approximation error ϵapp\epsilon_{\mathrm{app}}, and 0<α<10 < \alpha < 1, the resulting contribution is of order

+\epsilon_w\epsilon_{\mathrm{app}}^\alpha,$$ up to constants and discount-factor dependence (Theorem 1). Thus, accurate approximation of $Q^\pi$ reduces sensitivity to weight-estimation error even without approximate Bellman completeness. Under exact value-function realizability, consistent occupancy-ratio estimation yields consistent FQE without Bellman completeness. Combining this analysis with FORE gives an end-to-end guarantee under joint realizability of the value function and occupancy ratio, without Bellman completeness of the value class or a completeness requirement for the auxiliary critic class used in ratio estimation. To our knowledge, these guarantees have not been established previously. **2. Scope of the assumptions (Weakness 2).** Exact ERM over closed convex classes is a broad and standard setting for theoretical analysis in RL. It includes linear function approximation, which is the focus of much of the existing TD and FQE theory, as well as rich nonlinear classes such as RKHS balls and smoothness classes. We focus on this setting to isolate the geometric properties of FQE and characterize how occupancy-weight estimation error interacts with value-function approximation. Closed convexity ensures a unique, nonexpansive $L^2$ projection, which is central to our analysis. Extending these guarantees to ordinary neural-network FQE introduces the separate challenge of nonconvex optimization, whose theoretical analysis is a substantial research problem in its own right. Our interpolation condition controls the regularity of the Bellman projection residuals, measuring how concentrated these residuals can be relative to their $L^2$ size. Specifically, it bounds their supremum norm by a power $\alpha\in[0,1]$ of their $L^2$ norm. **Under boundedness alone, the condition always holds with $\alpha=0$**, yielding our basic first-order bound on weight-estimation error without additional regularity assumptions. More regular residual classes admit larger exponents and therefore stronger robustness to weight-estimation error. For example, finite-dimensional residual classes with equivalent supremum and $L^2$ norms satisfy $\alpha=1$, while smoothness classes can satisfy intermediate exponents $0 < \alpha < 1$. Thus, the interpolation condition determines the order of dependence on weight-estimation error, rather than whether our guarantees apply. We have also removed the positive lower bound on the true occupancy ratio from our end-to-end result. Our revised Hellinger-error analysis directly controls $$\epsilon_w = \\|\sqrt{\widehat w_\beta}-\sqrt{w_\beta}\\|_{2,\nu_b},$$ allowing the true ratio to vanish. The revised result requires only an upper bound on the true occupancy ratio, together with explicit coverage and ratio-approximation conditions.
AuthorsOpenReview ↗

Response to Reviewer 82NU (Part 2 of 2)

3. Experiments with inexact weights (Weakness 3).

Prior work has already demonstrated empirical benefits of distribution correction (Gelada and Bellemare, 2019; Sinha et al., 2022). Our experiments instead investigate the interaction between weight-estimation error, value approximation, and Bellman completeness predicted by our theory.

To study this interaction directly, we added a controlled four-state population example (Appendix C.4 and Figure S1). We vary value approximation error and inherent Bellman error separately while holding the sampling distribution and weight perturbation fixed. Starting both iterations from zero, we compare FQE with exact occupancy weights against FQE with prescribed perturbed weights. The table reports the L2(dβ)L^2(d_\beta) distance between the resulting Q-functions.

Value approximation errorInherent Bellman errorAfter 5 updatesAfter 20 updatesAt convergence
0.100.400.098280.210820.24000
0.100.160.039310.084330.09600
0.100.080.019660.042160.04800
0.050.160.021840.046850.05333
00.16000

Reducing either value approximation error or inherent Bellman error reduces sensitivity to weight error, both during iteration and at convergence. Under exact realizability, the effect vanishes even though the value class remains Bellman incomplete.

We also expanded our CartPole experiments with learned occupancy weights, varying sample size, policy mismatch, and value- and ratio-model complexity. With 25,000 transitions, 64 Q features, and stronger policy mismatch, FORE128 reduces mean absolute value error from 10.351 for unweighted FQE and 9.000 for action-only weighting to 6.961. Its paired improvement over action-only weighting is 2.038 (95% interval: [1.188, 2.878]). Appendix C.3 and Table S1 report the full experimental grid.

References.

I appreciate the authors’ response and their careful and thoughtful answers to my questions. However, I remain concerned about the relatively narrow scope of the paper and the limited experimental evaluation. While the rebuttal has addressed some of my concerns, I believe my initial overall rating still accurately reflects my assessment of the paper. Therefore, I would like to keep my overall rating unchanged.

Thank you for considering our response. We believe the paper’s value is in clarifying when occupancy weighting can make FQE reliable without Bellman completeness and what is lost when the weights are estimated imperfectly. Bellman completeness can be difficult to justify for a chosen function class, while accurate occupancy weights can be difficult to estimate. Our finite-sample analysis connects these two challenges and shows how the effect of weight error depends on the Bellman projection residual. In particular, even imperfect weights can reduce the error arising from a lack of Bellman completeness. A small Bellman projection residual (which can result from either approximate realizability of the hypothesis class or approximate Bellman completeness) also mitigates the effect of weight error. The additional experiments study this tradeoff across sample sizes, policy mismatches, and function-class sizes.

Reviewer_HJtmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the stability of fitted Q-evaluation in off-policy evaluation. The authors reweight the regression objective using occupancy distributions so that, under certain conditions, the projected Bellman operator becomes a contraction. This allows the analysis to avoid requiring Bellman completeness. The paper further examines errors arising from finite samples, function approximation, and weight estimation, and combines the analysis with occupancy ratio estimation methods to provide end-to-end guarantees. Experiments in finite-state environments and CartPole suggest that weighting can improve stability and reduce evaluation error, although the practical benefits and scope of applicability need more thorough validation.

Critical strengths

The paper studies the instability of FQE in off-policy settings. Using distribution weighting to improve stability is a sensible approach.

The theoretical analysis is thorough. I appreciate the connection between weight estimation error and the Bellman projection residual.

Critical weaknesses

The theory still requires the model and environment to satisfy specific conditions, and its applicability to used neural networks is limited.

The new conditions that replace Bellman completeness are themselves difficult to satisfy. The paper does not sufficiently explain which practical settings make this trade-off worthwhile.

The experiments are limited, particularly in their analysis of weight estimation error, dataset size, and the mismatch between the behavior and target policies. As a result, they do not adequately validate the new theoretical results or establish whether the benefits outweigh the cost of estimating the weights.

Questions for authors

Please address the concerns listed under Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer HJtm (Part 1 of 2)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for recognizing the thoroughness of our analysis and the connection between weight-estimation error and the Bellman projection residual. We have expanded the experiments and reorganized the theoretical results to clarify when learning occupancy weights is beneficial.

1. Scope of the theory (Weakness 1).

Exact ERM over closed convex classes is a broad and standard setting for theoretical analysis in RL. It includes linear function approximation, which is the focus of much of the existing TD and FQE theory, as well as rich nonlinear classes such as RKHS balls and smoothness classes. We focus on this setting to isolate the geometric properties of FQE and characterize how occupancy-weight estimation error interacts with value-function approximation. Closed convexity ensures a unique, nonexpansive L2L^2 projection, which is central to our analysis. Extending these guarantees to ordinary neural-network FQE introduces the separate challenge of nonconvex optimization, whose theoretical analysis is a substantial research problem in its own right.

2. Trade-off between occupancy weighting and Bellman completeness (Weakness 2).

Bellman completeness requires the value class to approximate the Bellman update of every function in the class, rather than only the target QπQ^\pi. For example, a linear class may contain QπQ^\pi even when its span is not preserved by the Bellman operator. Our result shows that accurate occupancy-ratio estimation can replace Bellman completeness in FQE. The one-step analysis explains why: weight-estimation error interacts with the Bellman projection residual. Approximate Bellman completeness controls this residual uniformly, while approximate realizability controls it along the FQE iterates. Thus, occupancy weighting is especially useful when the value class approximates QπQ^\pi well but is not Bellman complete, and the occupancy ratio can be estimated accurately. We now state this case separately in a realizable-case corollary.

We have also weakened and clarified the conditions needed for our guarantees. Our revised finite-sample analysis provides a first-order bound on weight-estimation error without the norm-comparability condition, which is needed only for the sharper higher-order bound. Our interpolation condition controls how concentrated Bellman projection residuals can be relative to their L2L^2 size, bounding their supremum norm by a power α∈[0,1]\alpha\in[0,1] of their L2L^2 norm. Under boundedness alone, this condition always holds with α=0\alpha=0, giving our basic first-order bound. More regular residual classes admit larger exponents and higher-order robustness, including finite-dimensional classes with equivalent norms (α=1\alpha=1) and smoothness classes with 0<α<10 < \alpha < 1.

We have also removed the positive lower bound on the true occupancy ratio from our end-to-end analysis. By controlling ratio-estimation error directly in Hellinger distance, we allow the true ratio to vanish. Under joint realizability of the value function and occupancy ratio, consistent FORE estimation yields consistent FQE without Bellman completeness of either function class. The general bound also accounts for approximation errors when either class is misspecified.

AuthorsOpenReview ↗

Response to Reviewer HJtm (Part 2 of 2)

3. Weight-estimation error, sample size, policy mismatch, and cost (Weakness 3).

Our main contribution is the finite-sample analysis, and we have expanded the experiments to examine its predictions directly.

First, we added a controlled population experiment that varies value-function approximation error and inherent Bellman error separately while holding weight-estimation error fixed (Appendix C.4 and Figure S1). Reducing either error reduces sensitivity to weight misspecification. Under exact realizability, this sensitivity vanishes despite Bellman incompleteness, illustrating the interaction predicted by our theory.

Second, we expanded CartPole to vary sample size, behavior--target policy mismatch, and value- and ratio-model complexity, and added action-only weighting as a baseline. The table reports mean absolute policy-value error (lower is better).

MismatchTransitionsQ featuresUnweightedAction-onlyFORE32FORE128
Stronger25,0006410.3519.0006.1296.961
Original25,000645.8085.3064.9715.069
Original5,000325.1334.8255.0026.218

These results illustrate when learning occupancy weights is worthwhile. With 25,000 transitions and 64 Q features, occupancy weighting provides substantial gains under stronger policy mismatch and smaller gains under the original mismatch. In the 5,000-transition, 32-feature setting with original mismatch, learned occupancy weighting no longer outperforms action-only weighting, reflecting the cost of estimating ratios with less data.

In practice, we recommend tempering estimated occupancy weights, following Sinha et al. (2022). We have added a discussion of weight tempering to Section 5 of the revised paper. Tempering interpolates between fully importance-weighted FQE (T=1T=1) and unweighted FQE (T→∞T\to\infty), controlling weight variability and allowing practitioners to tune the finite-sample bias–variance tradeoff.

Computationally, occupancy weights are estimated once and reused across all FQE regressions, so their estimation cost is amortized across iterations. Appendix C.3 reports the ratio-fitting times and full experimental grid (Table S1).

References.

The authors have given a reasonable explanation for my first concern. I understand that the difficulties of analyzing standard neural-network models limit the theory to the current setting.

For my second concern, the clarification and relaxation of the assumptions address some of my doubts, but it is still unclear in which practical settings this trade-off would be worthwhile. Please also restore the missing heading for this response so that other readers can easily find it and do not mistakenly think the concern was left unanswered.

For my third concern, the new four-state experiment provides a useful illustration of the predicted interaction between errors. However, it uses a specially constructed population model with manually perturbed weights, rather than weights actually learned from data. I therefore see it mainly as an illustration of the mechanism, with limited evidence of how much it matters in more general settings. The CartPole experiments provide some additional evidence, but their scope remains limited.

Reviewer_hw16OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the canonical fitted-Q evaluation (FQE) without Bellman completeness assumption. Without any additional restrictions, there exists a geometry mismatch between the projection operator and the Bellman operator, possibly leading to the whole iterative process exploding up. The paper proposes occupancy-weighted FQE: by using additionally learned weight to correct the geometry of the projected Bellman operator, making it a contraction operator (under the occupancy distribution of the target policy). And hence the authors provide convergence analysis and finite-sample guarantee for the proposed algorithm, under appropriate regularity and coverage assumptions. The paper also gives some empirical evidence on the geometry mismatch phenomenon and the effectiveness of the proposed algorithm.

Critical strengths
  1. The overall method and intuition are very natural and effective. To deal with the geometry mismatch between the projection operator and the Bellman operator, the proposed algorithm uses a learned occupancy-ratio weighting to change the regression geometry, making the overall projected Bellman operator a contraction. The framework just follows FQE, while the guarantee can get rid of the (sometimes annoying) Bellman completeness assumption (or incorporate with either approximate realizability or approximate completeness).

  2. The finite-sample analysis is well-structured as 3-fold. The authors firstly provide the population contraction for the projected Bellman operator, and then the estimation bound with several decomposed terms, and finally combined with the FORE-type weight estimator with sample complexity rate. Most of the previous analyses of FQE-based methods requires Bellman completeness to proceed (or with potentially-blowing-up bounds with only value function realizability). This work gives the first analysis of FQE based on the "corrected-geometry" intuition, and can handle lots of cases (e.g., general function approximation).

Critical weaknesses
  1. The theoretical guarantees rely on several assumptions that seem restrictive (the necessarily and implications could be better justified):
  • C6 (coverage): in additional to population coverage, the empirical coverage (i.e., bounded w^β\hat{w}_\beta) and the support-domination condition might need some clarifications (since why not just get the empirical coverage from a concentration argument + population coverage?).

  • C7 (norm comparability): I'm not sure whether this condition can be derived directly from occupancy-ratio estimation accuracy under suitable regularity assumptions, rather than imposed separately. The guarantee depends on 1/λF1/\lambda_{\mathcal{F}}, which might scales with the minimum eigenvalue of the empirical covariance matrix (in the linear case), which might exacerbate.

  • C9 (lower bounded occupancy ratio): this is much stronger than standard (one-sided) density-ratio coverage (aka concentrability) and may exclude important settings, such as deterministic target policies.

  1. Though the core idea is simple and intuitive, the analysis part introduces numerous assumptions, new notations, and error terms, making the readers hard to follow up and identify the main theoretical insights (e.g., Thm1 combines several sources of error and different structural conditions, which obscures the key results). I would suggest a cleaner representation as first establishing a simple finite-sample guarantee under exact realizability and exact occupancy weights, and then extend the analysis to handle misspecification.
Questions for authors

1. Choice of the occupancy discount β\beta.

Why introduce a separate occupancy discount factor β\beta, rather than simply choosing β=γ\beta=\gamma? Could the authors clarify the motivation for this additional flexibility and whether there are settings where β≠γ\beta\ne\gamma provably yields better statistical guarantees?

2. Generality of the occupancy-ratio estimator and the associated assumptions.

The end-to-end guarantee relies on FORE, which is a relatively recent and less widely studied occupancy-ratio estimator. How would the analysis extend to more conventional occupancy-ratio estimators, such as minimax or DICE-style methods? In particular, would C7/9 still be necessary, and could they be verified under the guarantees provided by such estimators? The uniformly positive lower bound in C9 appears particularly restrictive (at least to me). It would be helpful to clarify which assumptions are intrinsic to weighted FQE and which arise specifically for FORE.

3. Dimension-free scalar policy-value estimation.

The main result provides an L2 error bound for the learned Q-function, from which a scalar policy-value estimation bound can be derived. However, recent OPE methods, such as Q-MMR [1], can achieve dim-free guarantees when only the scalar policy value J(π)J(\pi) is of interest. Can the proposed analysis be refined to obtain a dim-free bound for ∣J^(π)−J(π)∣|\widehat J(\pi)-J(\pi)|, even when the Q-function estimation error remains dim-dependent?

4. The role of occupancy-ratio realizability in circumventing existing hardness results.

[2] establish that value-function realizability and standard density-ratio coverage alone are insufficient for offline RL. Thus, due to my understanding, at least one of the followings should be applied in order to achieve learnability under value-function realizability alone: 1) stronger structural assumptions (e.g., Bellman completeness); 2) stronger data coverage assumptions (e.g., push-forward coverage); 3) additional structural assumptions (e.g., weight/occupancy ratio realizability). I guess here uses 3), right? Could the authors make this assumption and its role more explicit in the main theoretical statements? In particular, it would be helpful to clarify how this ratio-realizability enables the proposed approach to circumvent the existing hardness results.

Ref: [1] Li, Xiang, and Nan Jiang. "Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching." arXiv preprint arXiv:2605.06474 (2026). [2] Foster, Dylan J., et al. "Offline reinforcement learning: Fundamental barriers for value function approximation." arXiv preprint arXiv:2111.10919 (2021).

4 comments
AuthorsOpenReview ↗

Response to Reviewer hw16 (Part 1 of 2)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Response to Reviewer hw16

Thank you for the careful review and for recognizing the benefit of correcting the regression geometry. We have simplified the theoretical presentation and strengthened the guarantees, including removing the positive lower bound on the true occupancy ratio.

1. Simplified presentation of the theory.

We have reorganized the assumptions by role: A1--A2 concern population geometry, B1--B3 the sampling setup, C1--C4 weighted regression, and D1--D3 only the FORE end-to-end result.

We have also revised Section 3.2 to explain where each term in the finite-sample bound comes from. The one-step regression lemma separates the usual statistical regression error from the error due to estimated occupancy weights. The latter interacts with the Bellman projection residual, explaining why exact weights or Bellman completeness eliminate this term, and how realizability controls it along the FQE iterates.

To make the main result easier to interpret, we now state a separate corollary for the QπQ^\pi-realizable case. The bound then contains only finite-iteration error, statistical regression error, and weight-estimation error. With exact occupancy weights, the last term also disappears. The general theorem extends this result to approximate realizability and approximate Bellman completeness.

2. Conditions and lower-bound removal (C6, C7, C9).

Coverage (former C6, now C1). C1 requires target-occupancy coverage and bounded estimated weights. These are distinct conditions: population coverage does not itself imply a uniform bound on an estimated ratio, but the latter can be enforced by clipping. We have also removed the former support-domination requirement.

Norm comparability (former C7, now C2). C2 simply requires comparability of two population norms of the same function, one under the estimated weights and one under the true weights. With exact weights, it holds automatically with λF=1\lambda_{\mathcal F}=1. It does not impose a minimum-eigenvalue condition on an empirical covariance matrix.

A simple sufficient condition is w^β≥w‾>0\widehat w_\beta\ge\underline w > 0 and wβ≤κ‾β<∞w_\beta\le\overline\kappa_\beta < \infty, giving λF=min⁡1,w‾/κ‾β\lambda_{\mathcal F}=\min\\{1,\underline w/\overline\kappa_\beta\\}. This is only a worst-case sufficient condition. Appendix B.1 establishes weaker conditions based on Gram-matrix and covariance-operator comparability for linear and RKHS classes, respectively.

Importantly, C2 is needed only for our sharper higher-order bound, in which weight-estimation error interacts with the Bellman projection residual. We now also establish a finite-sample guarantee without C2, with first-order dependence on weight error (Lemma S7 and Corollary S2).

Positive lower bound (former C9). This condition arose specifically in the FORE analysis through the log-ratio/KL comparison. We have removed it by controlling weight-estimation error directly in Hellinger distance,

\\|\sqrt{\widehat w_\beta}-\sqrt{w_\beta}\\|_{2,\nu_b},$$ allowing the true occupancy ratio to vanish. The revised FORE end-to-end result requires only $w_\beta\le\overline\kappa_\beta$. Since FORE's fitted weights are bounded below by construction, this upper bound also implies C2 with $\lambda_{\mathcal F}=e^{-2R_w}/\overline\kappa_\beta$. Appendix G.3 establishes the Hellinger rate, and Appendix G.4 combines it with the FQE analysis. **3. Choice of $\beta$ (Q1).** We introduce a separate occupancy discount $\beta$ to cover two weighting regimes commonly studied in the literature: discounted occupancy weighting and stationary weighting. We recommend $\beta=\gamma$ as the default. This matches the policy-value discount, and the discounted occupancy measure is well defined without ergodicity assumptions. As $\beta\to1$, the weighting approaches the stationary regime; taking $\beta=1$ requires a stationary target-policy distribution. Increasing $\beta$ improves the contraction factor $\gamma/\sqrt{\beta}$, but can make the occupancy ratio harder to estimate. Thus, a smaller contraction factor does not necessarily give a better overall finite-sample guarantee, which also depends on ratio-estimation accuracy.
AuthorsOpenReview ↗

Response to Reviewer hw16 (Part 2 of 2)

4. Generality of occupancy-ratio estimation (Q2).

Our FQE analysis is not tied to FORE. The basic finite-sample bound requires bounded estimated weights and control of their Hellinger error. The sharper higher-order bound additionally requires C2. Neither result requires the true occupancy ratio to be bounded away from zero.

We use FORE for the end-to-end guarantee because existing analyses of DualDICE and minimax ratio estimation typically introduce an auxiliary critic class that must be sufficiently rich relative to the ratio class, leading to a completeness-type condition on the weight-estimation side. FORE instead provides direct Hellinger control through ratio-class approximation error. Under joint realizability of the value and occupancy-ratio classes, we obtain consistency without completeness of either class.

DualDICE and other minimax estimators can also be used within our FQE analysis when their weight-estimation error is controlled, with C2 additionally needed for the higher-order guarantee.

5. Dimension-free scalar policy-value estimation (Q3).

Q-MMR obtains a dimension-free data-dependent guarantee for scalar policy-value estimation, expressed in terms of empirical matching losses and second moments of the learned weights. In its data-independent random-design analysis, function-class complexity reappears in higher-order terms, while the leading scalar-value term remains dimension-free.

Our result instead controls the entire Q-function in L2L^2, from which scalar-value control follows by averaging. Obtaining a comparable dimension-free leading term for the scalar value would require an analysis targeting that functional directly, rather than refining the L2L^2 Q-function bound.

6. Hardness and occupancy-ratio realizability (Q4).

Yes, we use exactly the third route you identify: occupancy-ratio realizability. Our end-to-end bound explicitly includes the ratio-class approximation error, which vanishes under ratio realizability. We therefore obtain consistency under joint realizability of the value function and occupancy ratio, without Bellman completeness.

We do not claim learnability from value-function realizability and standard coverage alone. Occupancy-ratio realizability is the additional structural assumption that allows our result to avoid the hardness result in Foster et al. (2022).

References.

The authors have addressed all my questions and confusions (especially they removed some harsh conditions). Therefore I maintain my score.

Thank you for engaging with our response. We are glad to hear we addressed all your concerns. Since all of these are addressed we do hope this increases your confidence in the suitability of the paper for acceptance and that you can raise your score accordingly. Thank you again for your efforts reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Our main contribution is a finite-sample analysis of FQE when both occupancy weights and Bellman regressions are learned from data. We show how weight-estimation error interacts with value approximation and Bellman completeness, and obtain end-to-end consistency under coverage and joint value-function and occupancy-ratio realizability, without completeness assumptions.

We have expanded the theory and experiments to address the reviewers’ main concerns:

  1. Contribution beyond distribution correction and FORE. Our contribution is the analysis of FQE with estimated weights, including how approximate realizability reduces sensitivity to weight error, rather than exact-weight contraction or a new ratio estimator. The FQE guarantees apply beyond FORE; FORE provides one end-to-end implementation.

  2. Weaker assumptions. We removed the positive lower bound on the true occupancy ratio and added a finite-sample guarantee without norm comparability or residual interpolation. Norm comparability and residual interpolation are needed only for the sharper rates, not the basic guarantee. We also reorganized the theory and added a separate realizable-case corollary to make these distinctions clear.

  3. Experiments that test the theory. We added a controlled experiment that varies value approximation and Bellman incompleteness separately, directly testing their interaction with weight error. We also expanded CartPole across sample sizes, policy mismatch, and model complexity, with an action-only baseline. Under stronger mismatch with 25,000 transitions, mean value error falls from 10.35 unweighted and 9.00 action-only to 6.96 with learned occupancy weights.

These additions make the statistical contribution explicit, weaken the assumptions for the basic result, and directly examine when estimating occupancy weights improves FQE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.