acceptodds
Under review as a conference paper at ICLR 2027

Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling

Abstract

Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly robust (DR) estimation remains unbiased under common support but retains these high-variance action-level weights. A prior estimator, Off-policy evaluation with Conjunct Effect Model (OffCEM), replaces them with more stable cluster-level weights, at the cost of relying on local correctness of the reward model. In this paper, we show that, under the assumptions required by DR and OffCEM, there exists an unbiased family of estimators that interpolates between OffCEM and DR. Building on this result, we propose the *Variance Optimal-CEM* (VOCEM) estimator, which selects the interpolation coefficient to minimize variance. We derive the population-optimal coefficient in closed form and show that the resulting estimator has variance no larger than either endpoint, OffCEM and DR. Experiments in controlled synthetic settings and on two large-action benchmarks show that VOCEM improves upon both endpoints in all 23 evaluated conditions, exhibiting greater stability and empirical robustness.

Primary area:
reinforcement learning
Keywords:
Off-Policy Evaluation, Doubly robust, clustering, bias, variance, contextual bandits

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_njDnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes VOCEM, an OPE estimator that interpolates between OffCEM and DR. Both estimators share the same direct-method term but use different weights for the reward-model residual: cluster-level weights for OffCEM and action-level weights for DR. The proposed family can be written as:

V^λ=(1−λ)V^OffCEM+λV^DR,λ∈[0,1].\hat V_\lambda = (1-\lambda)\hat V_{\mathrm{OffCEM}} + \lambda\hat V_{\mathrm{DR}}, \qquad \lambda\in[0,1].

Under common support and local correctness, the authors show that every fixed coefficient gives an unbiased estimator and derive the variance-minimizing coefficient in closed form. In practice, VOCEM uses data to estimate this coefficient. Empirically, VOCEM outperforms both DR and OffCEM in both synthetic and semi-synthetic scenarios.

Critical strengths

S1. Practicality: VOCEM is simple to implement. It computes the mixing coefficient in closed form from two empirical moments using logged data, without additional hyperparameter tuning

S2. Theoretical clarity: The paper provides a clean connection between OffCEM and DR. The exact bias path and variance calculation appear correct under the stated assumptions.

S3. Empirical performance: The improvements over OffCEM are consistent across the main experiments. The sample-size and action-space sweeps help characterize where the method provides larger gains.

Critical weaknesses

W1. Limited novelty and insufficient analysis.

The estimator is a scalar interpolation between OffCEM and DR, and its population-optimal coefficient follows from standard quadratic variance minimization. Given this relatively incremental construction, a deeper analysis of the practical estimator (where the coefficient is computed from logged data) would be important to establish the contribution. One possible trick would be to estimate the coefficient on one subset of logged data and perform OPE on the remaining subset of logged data, which could simplify the theoretical analysis. Then, it would be beneficial that the theoretical version that only uses a subset of logged data is tested empirically.

W2. Missing baselines.

The experiments mainly compare VOCEM with unclipped DR and OffCEM using ridge regression. Given the weak overlap between the target and logging policies (e.g., Table 3), comparisons with clipped DR, SNIPS/SNDR, and other methods that normalize or shrink importance weights would strengthen the evaluation. These baselines are particularly relevant because they directly address the instability caused by high importance weights. It would also be useful to evaluate OffCEM with its original two-stage regression procedure or a more expressive reward model, using the same fitted model for VOCEM to ensure a controlled comparison. These experiments would clarify whether the reported gains persist when existing importance-weight stabilization methods and stronger reward-model training procedures are used.

Questions for authors

Q1. Could the authors clarify which theoretical guarantees extend to the practical estimator where λ^\hat\lambda is computed from the evaluation log itself? Moreover, at κ=1\kappa=1, we can prove (unless my derivations are wrong) that λ⋆=0\lambda^\star=0, so VOCEM in this case should match OffCEM. Yet empirical VOCEM still outperforms OffCEM. Is this expected?

Q2. How does VOCEM compare with clipped DR, SNIPS/SNDR, and other methods that normalize or shrink importance weights under the weak policy overlap considered here?

Q3. Do the improvements persist when OffCEM uses its original two-stage regression procedure or a more expressive reward model?

Reviewer_mRwzOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper observes that the DR and OffCEM off-policy estimators differ only in the importance weight applied to the residual term, and defines a one-parameter family that interpolates between them. Under OffCEM’s local-correctness assumption, every member of the family is unbiased. The variance-minimizing interpolation coefficient has a closed form, and the resulting oracle estimator has variance no larger than either endpoint. The proposed estimator, VOCEM, plugs in sample estimates of this coefficient, and it achieves lower MSE than OffCEM and DR on synthetic data and on two extreme-classification datasets.

Critical strengths
  1. A clean unifying view. Framing DR and OffCEM as endpoints of a single residual-weighting family is simple and clarifying, and the unbiasedness of the whole family follows transparently from a cluster-projection lemma.
  2. A practical, tuning-free estimator with careful empirical reporting. The coefficient has a closed form computed from logged data. The experiments report paired-bootstrap intervals and the full distribution of the estimated coefficient.
Critical weaknesses
  1. Under the paper’s assumptions, the difference between the DR and OffCEM terms is a mean-zero control variate. The optimal coefficient is therefore the standard minimum-variance control-variate (or optimal-combination) result, and prior work on variance-optimal baselines for off-policy evaluation is not discussed. Moreover, the variance guarantee holds only for the oracle coefficient. VOCEM estimates the coefficient on the same log, which makes it biased and leaves it without any guarantee, yet the abstract and conclusion claim that “the resulting estimator” has variance no larger than either endpoint.

  2. In the homoskedastic synthetic setting (κ = 1) with an oracle reward model, the paper’s own formulas give an optimal coefficient of exactly zero, so the oracle estimator reduces to OffCEM. VOCEM nevertheless reports about 25% lower MSE than OffCEM there. In addition, the gain shrinks as n grows (MSE ratio 0.19 at n = 500 vs. 0.87 at n = 8,000). This suggests that the improvements come from finite-sample, data-dependent switching away from DR when extreme weights appear, rather than from the claimed variance-optimal mechanism. Such switching is the same idea behind existing methods like SWITCH-DR and DR with shrinkage, which are not compared.

Questions for authors

Please see weakness.

Reviewer_udwXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies off-policy evaluation in contextual bandits with large action spaces. The authors propose VOCEM, which interpolates between DR and OffCEM by combining action-level and cluster-level importance weights. Under common support and local correctness, they show that the resulting estimator is unbiased and derive a closed-form expression for the variance-optimal interpolation coefficient. They then estimate this coefficient from logged data and show that VOCEM achieves lower MSE than DR and OffCEM in synthetic and semi-synthetic experiments.

Critical strengths
  • I think the problem is well motivated. Off-policy evaluation can suffer from high variance when the action space is large, and combining action-level and cluster-level importance weighting seems like a reasonable approach to this problem.

  • The proposed method is simple and easy to implement. The authors show that the interpolation between DR and OffCEM remains unbiased under local correctness, and derive a closed-form expression for the variance optimal coefficient. The theoretical analysis is also easy to follow.

  • The experimental results are quite promising. The authors evaluate the method on both synthetic and semi-synthetic datasets, including experiments with learned reward models. VOCEM consistently achieves lower MSE than DR and OffCEM across the reported settings.

Critical weaknesses
  • I think the novelty of this paper is rather limited. The main idea is to take a convex combination of two existing estimators, DR and OffCEM, and the optimal coefficient follows from a simple quadratic minimization. I am not sure how much is new here compared to existing works on combining OPE estimators (e.g., Jeunen, 2025). Also, the variance optimality is only within this particular family of estimators. A stronger result, such as a semiparametric efficiency characterization under local correctness, would make the theoretical contribution more convincing.

  • I am also concerned about the local correctness assumption. It seems quite strong, as it requires the reward model to correctly predict the reward differences between actions within each cluster. In particular, I wonder how reasonable this assumption is in the learned-regression experiments, where the clusters are randomly assigned. Since the true reward function is available in these semi-synthetic experiments, I think the authors could at least check how well this assumption holds.

  • The theoretical result is only for the oracle estimator with the population-optimal coefficient, while the actual estimator uses a coefficient estimated from the same data. There is no asymptotic variance guarantee for this plug-in estimator, which seems like a missing piece. It would also be nice to have some finite-sample analysis, especially given that the coefficient is estimated and evaluated on the same log.

Questions for authors

See the weaknesses. Could you address the last two? Especially, it would be nicer to have some intuition / examples on the local correctness assumption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.