- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes POLICYLIFT for improving policies over structured actions using observational contextual-bandit data. It learns discrete action codes through outcome prediction and balancing objectives, optimizes a support-constrained policy over these codes, and deploys the policy through behavior-consistent lifting: sampling the historical generator until an action matches the selected code. The theoretical analysis establishes value equivalence under exact lifting and characterizes prediction, overlap, and sampling considerations. Experiments include synthetic diagnostics, a semi-synthetic headline task, and the OfflinePrompts benchmark.
Critical strengths
- A coherent solution to the deployment mismatch. Preserving the behavior distribution within each context–code pair makes code-level value agree with deployed action-level value. This avoids requiring codes to fully mediate the action’s effect on outcomes or training a separate decoder. The distinction between improving true code-level value and faithfully deploying that policy is useful.
- Useful mechanism diagnostics and encouraging empirical gains. The experiments examine deployment, predictiveness, and overlap separately, alongside code and estimator alternatives. On OfflinePrompts, POLICYLIFT improves value from to without observing prompt identities. These results support the practical potential of learning actionable representations from logged outcomes.
Critical weaknesses
- The text experiments only partially validate the claimed generative setting. The headline inputs include simulator context labels and noisy action-code/strategy features (Appendix E.4). OfflinePrompts avoids these features, but uses one cached, greedily decoded sentence per movie–prompt pair and shares the movie/prompt catalog across splits (Appendix F). These are useful controlled evaluations, but leave limited evidence for learning from text alone and deploying fresh generations—the distinction emphasized in the introduction.
- Practical support and sampling reliability remain insufficiently characterized. Appendix C.4 provides a finite-cap error bound, but deployment selects codes using estimated propensities. Passing does not establish adequate true support. For a code with true probability , the OfflinePrompts cap gives failure probability . This is a threshold illustration, not an observed failure rate. Actual text-task fallback rates and propensity calibration are needed to interpret deployment reliability.
- The theoretical justification of the implemented overlap penalty is incomplete. Theorem 5.6 assumes the error-shift inequality in Eq. (9), involving distributions over . The implementation instead measures sliced Wasserstein distance over learned , without establishing conditions connecting this distance to that inequality. Moreover, the main synthetic configuration uses (Appendix H.1). Although Figure 4 provides sensitivity analyses, the independent contribution of distributional overlap relative to marginal code balancing remains unclear.
- A relevant comparison under the same information constraints is missing. A regularized outcome predictor trained directly on the same logs could rerank candidates from the same behavior generator. This requires neither preference labels nor prompt identities or model-weight access. The existing modular ablations and prompt-aware references do not establish the advantage of code selection and lifting over this direct alternative at comparable sampling cost.
Questions for authors
- Generative validation: Could you report a headline ablation using text embeddings without the injected simulator features, and a bounded evaluation using fresh stochastic generations on held-out contexts? The existing reward evaluator can be retained. This would help distinguish representation learning and generative deployment from selection over recoverable simulator structure or cached outputs.
- Deployment reliability: Could you provide a small sweep over and , reporting deployed policy improvement, actual fallback rates, selected-code propensity calibration, and generator calls or tokens? Please relate the measured failure rates to the finite-cap bound already provided.
- Overlap justification and ablation: Under what assumptions does Eq. (9) apply to the implemented distance over ? Could you compare prediction-only, prediction plus overlap, prediction plus marginal balancing, and the full objective under matched settings, reporting held-out regression-adjustment error and policy value?
- Direct reranking baseline: Could you compare against a regularized reward-regression model trained on the same logs and used for Best-of- selection from the same generator? Matching the average generator-call budget would make this comparison informative about the benefit of the proposed representation and lifting mechanism.
3 comments
Response (2/3)
(b) Empirical reliability. On headlines and OfflinePrompts, the behavior systems are finite pools of texts, so the true propensities are known. We vary the threshold and . For (5 seeds), we report the improvement, the improvement predicted by Proposition C.5, the fraction of contexts with a fallback draw, the behavior draws per context (including the fallback draw), and the generated tokens per context:
| Dataset | Improvement | Prop. C.5 | Fallback | Draws | Tokens | |
|---|---|---|---|---|---|---|
| OfflinePrompts | 5 | 0.068 ± 0.002 | 0.069 | 0.66 | 4.7 | 142 |
| 10 | 0.098 ± 0.003 | 0.098 | 0.55 | 7.7 | 230 | |
| 20 | 0.130 ± 0.004 | 0.130 | 0.45 | 12.6 | 376 | |
| 50 | 0.172 ± 0.006 | 0.171 | 0.33 | 23.8 | 713 | |
| Headlines | 5 | 0.004 ± 0.001 | 0.003 | 0.61 | 4.7 | |
| 10 | 0.007 ± 0.002 | 0.007 | 0.48 | 7.3 | ||
| 20 | 0.015 ± 0.003 | 0.015 | 0.37 | 11.3 | ||
| 50 | 0.028 ± 0.005 | 0.028 | 0.29 | 21.0 |
We find: (1) The improvement is positive in all 32 settings and never decreases with . (2) It matches Proposition C.5 (largest difference 0.003 across all settings and seeds). (3) For , the selected code can be generated by the behavior system in 89% (OfflinePrompts) and 77% (headlines) of contexts, so the improvement can be realized there. In the remaining contexts, PolicyLift returns a behavior draw and thus never falls below the behavior policy. (4) Relating to your illustration: at on OfflinePrompts, the observed fallback rate is 0.45, below the 0.544 of your threshold illustration. (5) The estimated propensities of the selected codes are well calibrated on OfflinePrompts (0.109 estimated vs. 0.108 true for most contexts) and moderately too high on headlines (0.27 vs. 0.16 above 0.2). A larger reduces the fallback rate and the number of draws (e.g., fallback 0.35 at and on OfflinePrompts).
The draw budget trades deployment cost against improvement, but never leads to a deterioration.
Action: We added the finite draw cap to Section 5.1 and as new Proposition C.5 (Appendix C.4), a sentence on estimated propensities to Section 6, and the full sweep to Appendix G.4 (Figure 5, Tables 16 and 17).
W3/Q3. Overlap justification and ablation. Thank you for this point. We compute the overlap penalty on the learned representation , as in balanced representation learning (Johansson et al., 2016; Shalit et al., 2017), so that the sliced Wasserstein distance is computed in a low-dimensional space. We agree that the link to Eq. (9) should be explicit, and we now establish it:
(a) Assumptions for Eq. (9) on . Suppose the code-level prediction error is a Lipschitz function of , i.e., with -Lipschitz . Then Eq. (9) holds exactly for the distributions of and , with and (new Lemma C.7). This condition holds whenever both and depend on only through , as does by construction. Further, the sliced Wasserstein distance used in training controls on bounded supports, (Bonnotte, 2013). Hence, driving the implemented penalty to zero drives the overlap term of Theorem 5.6 to zero.
Response (3/3)
(b) Matched ablation. We vary only the weights of the Stage-1 objective. We report the improvement and the held-out RA error on contexts whose selected code can be generated by the behavior system (5 seeds):
| Objective | Headlines: improvement | Headlines: RA error | OfflinePrompts: improvement | OfflinePrompts: RA error |
|---|---|---|---|---|
| Prediction only | 0.003 ± 0.001 | 0.137 ± 0.018 | 0.148 ± 0.014 | 0.083 ± 0.021 |
| + overlap | 0.003 ± 0.001 | 0.139 ± 0.018 | 0.122 ± 0.024 | 0.122 ± 0.032 |
| + balancing | 0.012 ± 0.002 | 0.342 ± 0.012 | 0.131 ± 0.005 | 0.208 ± 0.014 |
| Full | 0.011 ± 0.002 | 0.345 ± 0.012 | 0.130 ± 0.004 | 0.205 ± 0.011 |
The two penalties play different roles. Balancing prevents the codes from collapsing: without it, 6 of 8 codes are empty on headlines, so the policy stays close to the behavior policy, with a small RA error but almost no improvement. With balancing, PolicyLift improves the policy on headlines. At our weight, the overlap penalty changes the improvement only slightly, and its role is to control the overlap term of Theorem 5.6 (Lemma C.7). On synthetic data, and give similar improvements (0.146 vs. 0.150), so our main synthetic choice is not critical.
Action: We added the link to Section 5.3, Lemma C.7 (Appendix C.8), and the ablation on all three datasets to Appendix G.5 (Table 18).
W4/Q4. Direct reranking baseline. Thank you for this very helpful suggestion. Best-of-N with an outcome model fit on the same logs uses the same information as PolicyLift and can perform strongly when this model ranks individual actions well, but without a guarantee, since its value depends on how well the model extrapolates across raw actions. Your suggestion motivated us to go one step further: our learned codes and code-level policy are modular and can be combined with reranking as an alternative Stage-3 instantiation rule. In this within-code reranking, we draw actions from the behavior system and return the one with the highest predicted outcome among those with the selected code (or among all draws if none has it).
We compare behavior-consistent lifting (BCL), Best-of-N, and within-code reranking at the same average number of behavior draws per context (in brackets). Best-of-N and within-code reranking use the same regularized outcome model, tuned on logged validation data (5 seeds):
| Instantiation rule | Synthetic | Headlines | OfflinePrompts |
|---|---|---|---|
| Best-of-N | 0.129 ± 0.034 [5.4] | 0.014 ± 0.000 [9.4] | 0.231 ± 0.005 [12.4] |
| BCL (PolicyLift, Ours) | 0.145 ± 0.008 [5.5] | 0.011 ± 0.002 [9.4] | 0.130 ± 0.004 [12.5] |
| PolicyLift w/ within-code reranking (Ours) | 0.131 ± 0.034 [5.4] | 0.013 ± 0.001 [9.4] | 0.238 ± 0.003 [12.4] |
(1) With within-code reranking, our codes achieve the highest value on OfflinePrompts (0.238 vs. 0.231 for Best-of-N). (2) BCL achieves the highest improvement on synthetic data, and (3) on headlines, all rules are within 0.003.
Our learned codes perform well with both instantiation rules. BCL is the rule of choice for high-stakes decisions where improvement must be guaranteed, and within-code reranking for low-stakes decisions where maximizing value matters more.
Action: We added within-code reranking to Section 6, a remark to Section 7, and all instantiation rules, including a PoTEC-style rule, to Appendix G.6 (Table 19).
We thank you again for your detailed review, which considerably strengthened our paper. We hope that our responses and revisions resolve all of your points. If anything remains unclear or would further support a positive assessment, we would be very grateful for a short note and are happy to provide additional clarifications during the discussion period.
Response (1/3)
Thank you for your careful and constructive review. We are glad that you found PolicyLift to be a coherent solution to the deployment mismatch, and our mechanism diagnostics useful with encouraging empirical gains. Your points helped us to strengthen the paper, and we incorporated them through focused additions (i.e., one new proposition, one new lemma, additional experiments in Appendix G, and one new remark on alternative instantiation rules). The method and the main results remain unchanged. We mark all resulting paper changes as Action, and all changes are highlighted in blue in the revised PDF.
W1/Q1. Generative validation. Thank you for this suggestion. Our text benchmarks were designed to be realistic while keeping a known ground truth: the headline benchmark adds simulator features so that a learnable signal can be recovered, and OfflinePrompts uses pre-generated sentences so that all methods are evaluated on identical outputs. We agree that both can be pushed further toward learning from text alone and deploying fresh generations. Hence, we ran both experiments you suggested:
(a) Headlines without injected simulator features. We replace the constructed action features by MiniLM text embeddings of the headline, (i) keeping the original context features, or (ii) also replacing these, including the context labels, by text embeddings of the article. Even in this small-sample setting, where the outcome signal is not ensured to be recoverable from text embeddings alone, PolicyLift still improves over the behavior policy: 0.0083 ± 0.0027 in (i) and 0.0043 ± 0.0010 in (ii).
(b) Fresh stochastic generations on held-out contexts. On OfflinePrompts, the behavior system now generates a new sentence with Mistral-7B for every draw (temperature 0.7, top-p 0.95), both for the logged data and at deployment. We evaluate with the existing reward evaluator on held-out users combined with held-out movies (3 seeds). Below, Tmax is the maximum number of behavior draws per context, and the behavior policy uses 30 generated tokens per context.
⇒ PolicyLift improves the policy for every draw budget, comparable to the setting with pre-generated sentences (0.130). The deployed sentences are indeed fresh: only 0.09% of them appear verbatim in the logged data.
(c) Held-out movies. With pre-generated sentences but disjoint training and test movies, PolicyLift keeps most of its improvement (0.112 vs. 0.130, 5 seeds).
⇒ PolicyLift learns from text and deploys fresh generations for unseen contexts.
Action: We describe both setups in Appendix F and report all results in our new Appendix G.3 (new Tables 13 to 15).
W2/Q2. Deployment reliability. Thank you for this valuable point. Checking the support constraint with estimated propensities is standard practice in propensity-based policy learning, since the true propensities are typically unknown. We agree, however, that estimated and true support can differ, and we therefore now characterize exactly what happens in this case, both in theory and empirically.
(a) Theory: exact deployed value under a finite draw cap. Let pfail(x,c)=(1−μC(c∣x))Tmax be the probability, under the true code propensity μC, that none of the Tmax draws has code c, and let mˉ(x)=E(Y∣X=x) be the behavior value and mC(x,c)=E(Y∣X=x,C=c) the code regression at context x. For any code-level policy πC, the capped deployed policy πTmax↑ satisfies (see new Proposition C.5)
V(πTmax↑)−V(μ)=E(∑cπC(c∣X)(1−pfail(X,c))(mC(X,c)−mˉ(X))).
Hence: (i) an overestimated propensity only lowers the acceptance probability 1−pfail, which scales the improvement of a context down but never reverses it; (ii) a selected code that the behavior system cannot generate at a context returns a behavior draw there; (iii) whenever the selected codes improve over the behavior value, the improvement never decreases with Tmax. The finite-cap bound of the submission now follows from our new Proposition C.5.