Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

Fitted $Q$-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift its projected Bellman operator need not be contractive under the offline distribution. Consequently, value-function realizability alone does not ensure convergence, and existing analyses often impose Bellman completeness or related structural conditions. We study occupancy-weighted FQE and show that distribution weighting provides a different route to stability. Weighting each Bellman regression by the density ratio of a suitably discounted target-policy occupancy distribution to the offline distribution aligns the regression norm with the target-policy dynamics, making the population projected Bellman operator contractive. With exact occupancy weights, this removes the need for Bellman completeness. Our main contribution is a finite-sample analysis showing how this population stability extends to estimated weights, nonlinear function classes, and misspecification. The bounds separate finite-iteration, statistical, approximation, and ratio-estimation errors and show that ratio-estimation error enters through its interaction with the Bellman projection residual rather than as a separate additive term. This interaction can be controlled by approximate Bellman completeness or value-function realizability; under exact realizability, the effect of ratio-estimation error is higher order, while under exact Bellman completeness it vanishes. We also analyze an end-to-end procedure combining occupancy-weighted FQE with fitted occupancy-ratio evaluation, obtaining consistency under coverage and joint realizability of the value function and occupancy ratio without completeness assumptions on either class. Controlled experiments illustrate the contraction mechanism and the tradeoff between target alignment and coverage.