- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes VOCEM, an OPE estimator that interpolates between OffCEM and DR. Both estimators share the same direct-method term but use different weights for the reward-model residual: cluster-level weights for OffCEM and action-level weights for DR. The proposed family can be written as:
Under common support and local correctness, the authors show that every fixed coefficient gives an unbiased estimator and derive the variance-minimizing coefficient in closed form. In practice, VOCEM uses data to estimate this coefficient. Empirically, VOCEM outperforms both DR and OffCEM in both synthetic and semi-synthetic scenarios.
Critical strengths
S1. Practicality: VOCEM is simple to implement. It computes the mixing coefficient in closed form from two empirical moments using logged data, without additional hyperparameter tuning
S2. Theoretical clarity: The paper provides a clean connection between OffCEM and DR. The exact bias path and variance calculation appear correct under the stated assumptions.
S3. Empirical performance: The improvements over OffCEM are consistent across the main experiments. The sample-size and action-space sweeps help characterize where the method provides larger gains.
Critical weaknesses
W1. Limited novelty and insufficient analysis.
The estimator is a scalar interpolation between OffCEM and DR, and its population-optimal coefficient follows from standard quadratic variance minimization. Given this relatively incremental construction, a deeper analysis of the practical estimator (where the coefficient is computed from logged data) would be important to establish the contribution. One possible trick would be to estimate the coefficient on one subset of logged data and perform OPE on the remaining subset of logged data, which could simplify the theoretical analysis. Then, it would be beneficial that the theoretical version that only uses a subset of logged data is tested empirically.
W2. Missing baselines.
The experiments mainly compare VOCEM with unclipped DR and OffCEM using ridge regression. Given the weak overlap between the target and logging policies (e.g., Table 3), comparisons with clipped DR, SNIPS/SNDR, and other methods that normalize or shrink importance weights would strengthen the evaluation. These baselines are particularly relevant because they directly address the instability caused by high importance weights. It would also be useful to evaluate OffCEM with its original two-stage regression procedure or a more expressive reward model, using the same fitted model for VOCEM to ensure a controlled comparison. These experiments would clarify whether the reported gains persist when existing importance-weight stabilization methods and stronger reward-model training procedures are used.
Questions for authors
Q1. Could the authors clarify which theoretical guarantees extend to the practical estimator where is computed from the evaluation log itself? Moreover, at , we can prove (unless my derivations are wrong) that , so VOCEM in this case should match OffCEM. Yet empirical VOCEM still outperforms OffCEM. Is this expected?
Q2. How does VOCEM compare with clipped DR, SNIPS/SNDR, and other methods that normalize or shrink importance weights under the weak policy overlap considered here?
Q3. Do the improvements persist when OffCEM uses its original two-stage regression procedure or a more expressive reward model?