acceptodds
Under review as a conference paper at ICLR 2027

Generative Policy Improvement from Observational Data

Abstract

We study how to improve decision policies from observational data to *generate* better actions when actions are complex, structured objects, such as text messages or images. This task is subject to two major challenges: (i) action-level overlap is often too weak for reliable off-policy evaluation and improvement, and (ii) generating new structured actions often requires expensive training or fine-tuning (e.g., of an LLM) and can be ill-defined due to information loss in action representations. To address both, we propose PolicyLift, a three-stage framework for *generative policy improvement* with structured actions from observational data. In Stage 1, PolicyLift learns a small set of discrete action codes that capture high-level strategies in the observed action space and are tailored to support policy improvement under limited overlap. In Stage 2, it learns a context-dependent policy over these action codes. At deployment time, Stage 3 "lifts" the learned code-level policy back to the original action space by generating actions from the historical behavior system in a code-conditioned way. We show theoretically that improving the policy over action codes guarantees improvement of the induced policy in the original action space. We demonstrate PolicyLift across experiments in synthetic and real-world text generation tasks with consistent policy improvements over the behavior policy.

Primary area:
causal reasoning
Keywords:
causal inference, off-policy learning, limited overlap, structured action spaces, generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vCegOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes POLICYLIFT for improving policies over structured actions using observational contextual-bandit data. It learns discrete action codes through outcome prediction and balancing objectives, optimizes a support-constrained policy over these codes, and deploys the policy through behavior-consistent lifting: sampling the historical generator until an action matches the selected code. The theoretical analysis establishes value equivalence under exact lifting and characterizes prediction, overlap, and sampling considerations. Experiments include synthetic diagnostics, a semi-synthetic headline task, and the OfflinePrompts benchmark.

Critical strengths
  1. A coherent solution to the deployment mismatch. Preserving the behavior distribution within each context–code pair makes code-level value agree with deployed action-level value. This avoids requiring codes to fully mediate the action’s effect on outcomes or training a separate decoder. The distinction between improving true code-level value and faithfully deploying that policy is useful.
  2. Useful mechanism diagnostics and encouraging empirical gains. The experiments examine deployment, predictiveness, and overlap separately, alongside code and estimator alternatives. On OfflinePrompts, POLICYLIFT improves value from 0.10310.1031 to 0.2363±0.00340.2363 \pm 0.0034 without observing prompt identities. These results support the practical potential of learning actionable representations from logged outcomes.
Critical weaknesses
  1. The text experiments only partially validate the claimed generative setting. The headline inputs include simulator context labels and noisy action-code/strategy features (Appendix E.4). OfflinePrompts avoids these features, but uses one cached, greedily decoded sentence per movie–prompt pair and shares the movie/prompt catalog across splits (Appendix F). These are useful controlled evaluations, but leave limited evidence for learning from text alone and deploying fresh generations—the distinction emphasized in the introduction.
  2. Practical support and sampling reliability remain insufficiently characterized. Appendix C.4 provides a finite-cap error bound, but deployment selects codes using estimated propensities. Passing μ^C(c∣x)≥τ\hat{\mu}_C(c \mid x) \geq \tau does not establish adequate true support. For a code with true probability 0.030.03, the OfflinePrompts cap Tmax⁡=20T_{\max}=20 gives failure probability (1−0.03)20≈0.544(1-0.03)^{20} \approx 0.544. This is a threshold illustration, not an observed failure rate. Actual text-task fallback rates and propensity calibration are needed to interpret deployment reliability.
  3. The theoretical justification of the implemented overlap penalty is incomplete. Theorem 5.6 assumes the error-shift inequality in Eq. (9), involving distributions over XX. The implementation instead measures sliced Wasserstein distance over learned ψ(X)\psi(X), without establishing conditions connecting this distance to that inequality. Moreover, the main synthetic configuration uses β=0\beta=0 (Appendix H.1). Although Figure 4 provides sensitivity analyses, the independent contribution of distributional overlap relative to marginal code balancing remains unclear.
  4. A relevant comparison under the same information constraints is missing. A regularized outcome predictor trained directly on the same (X,A,Y)(X,A,Y) logs could rerank candidates from the same behavior generator. This requires neither preference labels nor prompt identities or model-weight access. The existing modular ablations and prompt-aware references do not establish the advantage of code selection and lifting over this direct alternative at comparable sampling cost.
Questions for authors
  1. Generative validation: Could you report a headline ablation using text embeddings without the injected simulator features, and a bounded evaluation using fresh stochastic generations on held-out contexts? The existing reward evaluator can be retained. This would help distinguish representation learning and generative deployment from selection over recoverable simulator structure or cached outputs.
  2. Deployment reliability: Could you provide a small sweep over τ\tau and Tmax⁡T_{\max}, reporting deployed policy improvement, actual fallback rates, selected-code propensity calibration, and generator calls or tokens? Please relate the measured failure rates to the finite-cap bound already provided.
  3. Overlap justification and ablation: Under what assumptions does Eq. (9) apply to the implemented distance over ψ(X)\psi(X)? Could you compare prediction-only, prediction plus overlap, prediction plus marginal balancing, and the full objective under matched settings, reporting held-out regression-adjustment error and policy value?
  4. Direct reranking baseline: Could you compare against a regularized reward-regression model trained on the same logs and used for Best-of-NN selection from the same generator? Matching the average generator-call budget would make this comparison informative about the benefit of the proposed representation and lifting mechanism.
3 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for your careful and constructive review. We are glad that you found PolicyLift to be a coherent solution to the deployment mismatch, and our mechanism diagnostics useful with encouraging empirical gains. Your points helped us to strengthen the paper, and we incorporated them through focused additions (i.e., one new proposition, one new lemma, additional experiments in Appendix G, and one new remark on alternative instantiation rules). The method and the main results remain unchanged. We mark all resulting paper changes as Action, and all changes are highlighted in blue in the revised PDF.

W1/Q1. Generative validation. Thank you for this suggestion. Our text benchmarks were designed to be realistic while keeping a known ground truth: the headline benchmark adds simulator features so that a learnable signal can be recovered, and OfflinePrompts uses pre-generated sentences so that all methods are evaluated on identical outputs. We agree that both can be pushed further toward learning from text alone and deploying fresh generations. Hence, we ran both experiments you suggested:

(a) Headlines without injected simulator features. We replace the constructed action features by MiniLM text embeddings of the headline, (i) keeping the original context features, or (ii) also replacing these, including the context labels, by text embeddings of the article. Even in this small-sample setting, where the outcome signal is not ensured to be recoverable from text embeddings alone, PolicyLift still improves over the behavior policy: 0.0083 ± 0.0027 in (i) and 0.0043 ± 0.0010 in (ii).

(b) Fresh stochastic generations on held-out contexts. On OfflinePrompts, the behavior system now generates a new sentence with Mistral-7B for every draw (temperature 0.7, top-p 0.95), both for the logged data and at deployment. We evaluate with the existing reward evaluator on held-out users combined with held-out movies (3 seeds). Below, Tmax⁡T_{\max} is the maximum number of behavior draws per context, and the behavior policy uses 30 generated tokens per context.

Tmax⁡T_{\max}Improvement V(π)−V(μ)V(\pi)-V(\mu)Generated tokens per context
50.057 ± 0.005150
100.081 ± 0.006248
200.114 ± 0.006413

⇒\Rightarrow PolicyLift improves the policy for every draw budget, comparable to the setting with pre-generated sentences (0.130). The deployed sentences are indeed fresh: only 0.09% of them appear verbatim in the logged data.

(c) Held-out movies. With pre-generated sentences but disjoint training and test movies, PolicyLift keeps most of its improvement (0.112 vs. 0.130, 5 seeds).

⇒\Rightarrow PolicyLift learns from text and deploys fresh generations for unseen contexts.

Action: We describe both setups in Appendix F and report all results in our new Appendix G.3 (new Tables 13 to 15).

W2/Q2. Deployment reliability. Thank you for this valuable point. Checking the support constraint with estimated propensities is standard practice in propensity-based policy learning, since the true propensities are typically unknown. We agree, however, that estimated and true support can differ, and we therefore now characterize exactly what happens in this case, both in theory and empirically.

(a) Theory: exact deployed value under a finite draw cap. Let pfail(x,c)=(1−μC(c∣x))Tmax⁡p_{\mathrm{fail}}(x,c)=(1-\mu_C(c\mid x))^{T_{\max}} be the probability, under the true code propensity μC\mu_C, that none of the Tmax⁡T_{\max} draws has code cc, and let mˉ(x)=E(Y∣X=x)\bar m(x)=E(Y\mid X=x) be the behavior value and mC(x,c)=E(Y∣X=x,C=c)m_C(x,c)=E(Y\mid X=x,C=c) the code regression at context xx. For any code-level policy πC\pi_C, the capped deployed policy πTmax⁡↑\pi_{T_{\max}}^{\uparrow} satisfies (see new Proposition C.5)

V(πTmax⁡↑)−V(μ)=E(∑cπC(c∣X)(1−pfail(X,c))(mC(X,c)−mˉ(X))).V(\pi_{T_{\max}}^{\uparrow})-V(\mu)=E\Big(\sum_{c}\pi_C(c\mid X)\big(1-p_{\mathrm{fail}}(X,c)\big)\big(m_C(X,c)-\bar m(X)\big)\Big).

Hence: (i) an overestimated propensity only lowers the acceptance probability 1−pfail1-p_{\mathrm{fail}}, which scales the improvement of a context down but never reverses it; (ii) a selected code that the behavior system cannot generate at a context returns a behavior draw there; (iii) whenever the selected codes improve over the behavior value, the improvement never decreases with Tmax⁡T_{\max}. The finite-cap bound of the submission now follows from our new Proposition C.5.

AuthorsOpenReview ↗

Response (2/3)

(b) Empirical reliability. On headlines and OfflinePrompts, the behavior systems are finite pools of texts, so the true propensities are known. We vary the threshold τ∈{0.03,0.05,0.1,0.2}\tau\in\{0.03, 0.05, 0.1, 0.2\} and Tmax⁡∈{5,10,20,50}T_{\max}\in\{5, 10, 20, 50\}. For τ=0.03\tau=0.03 (5 seeds), we report the improvement, the improvement predicted by Proposition C.5, the fraction of contexts with a fallback draw, the behavior draws per context (including the fallback draw), and the generated tokens per context:

DatasetTmax⁡T_{\max}ImprovementProp. C.5FallbackDrawsTokens
OfflinePrompts50.068 ± 0.0020.0690.664.7142
100.098 ± 0.0030.0980.557.7230
200.130 ± 0.0040.1300.4512.6376
500.172 ± 0.0060.1710.3323.8713
Headlines50.004 ± 0.0010.0030.614.7
100.007 ± 0.0020.0070.487.3
200.015 ± 0.0030.0150.3711.3
500.028 ± 0.0050.0280.2921.0

We find: (1) The improvement is positive in all 32 settings and never decreases with Tmax⁡T_{\max}. (2) It matches Proposition C.5 (largest difference 0.003 across all settings and seeds). (3) For τ=0.03\tau=0.03, the selected code can be generated by the behavior system in 89% (OfflinePrompts) and 77% (headlines) of contexts, so the improvement can be realized there. In the remaining contexts, PolicyLift returns a behavior draw and thus never falls below the behavior policy. (4) Relating to your illustration: at Tmax⁡=20T_{\max}=20 on OfflinePrompts, the observed fallback rate is 0.45, below the 0.544 of your threshold illustration. (5) The estimated propensities of the selected codes are well calibrated on OfflinePrompts (0.109 estimated vs. 0.108 true for most contexts) and moderately too high on headlines (0.27 vs. 0.16 above 0.2). A larger τ\tau reduces the fallback rate and the number of draws (e.g., fallback 0.35 at τ=0.1\tau=0.1 and Tmax⁡=20T_{\max}=20 on OfflinePrompts).

⇒\Rightarrow The draw budget trades deployment cost against improvement, but never leads to a deterioration.

Action: We added the finite draw cap to Section 5.1 and as new Proposition C.5 (Appendix C.4), a sentence on estimated propensities to Section 6, and the full sweep to Appendix G.4 (Figure 5, Tables 16 and 17).

W3/Q3. Overlap justification and ablation. Thank you for this point. We compute the overlap penalty on the learned representation ψ(X)\psi(X), as in balanced representation learning (Johansson et al., 2016; Shalit et al., 2017), so that the sliced Wasserstein distance is computed in a low-dimensional space. We agree that the link to Eq. (9) should be explicit, and we now establish it:

(a) Assumptions for Eq. (9) on ψ(X)\psi(X). Suppose the code-level prediction error is a Lipschitz function of ψ(x)\psi(x), i.e., ec=hc∘ψe_c=h_c\circ\psi with LL-Lipschitz hch_c. Then Eq. (9) holds exactly for the distributions of ψ(X)\psi(X) and ψ(X)∣C=c\psi(X)\mid C=c, with D=W1D=W_1 and λov=4BL\lambda_{\mathrm{ov}}=4BL (new Lemma C.7). This condition holds whenever both m^(⋅,c)\hat m(\cdot,c) and mC(⋅,c)m_C(\cdot,c) depend on xx only through ψ(x)\psi(x), as m^\hat m does by construction. Further, the sliced Wasserstein distance used in training controls W1W_1 on bounded supports, SW1≤W1≤CdRd/(d+1)SW11/(d+1)SW_1\le W_1\le C_d R^{d/(d+1)} SW_1^{1/(d+1)} (Bonnotte, 2013). Hence, driving the implemented penalty to zero drives the overlap term of Theorem 5.6 to zero.

AuthorsOpenReview ↗

Response (3/3)

(b) Matched ablation. We vary only the weights (β,γ)(\beta,\gamma) of the Stage-1 objective. We report the improvement and the held-out RA error ∥V^RA(πC)−VC(πC)∥\|\hat V_{\mathrm{RA}}(\pi_C)-V_C(\pi_C)\| on contexts whose selected code can be generated by the behavior system (5 seeds):

Objective (β,γ)(\beta,\gamma)Headlines: improvementHeadlines: RA errorOfflinePrompts: improvementOfflinePrompts: RA error
Prediction only (0,0)(0,0)0.003 ± 0.0010.137 ± 0.0180.148 ± 0.0140.083 ± 0.021
+ overlap (10−4,0)(10^{-4},0)0.003 ± 0.0010.139 ± 0.0180.122 ± 0.0240.122 ± 0.032
+ balancing (0,0.2)(0,0.2)0.012 ± 0.0020.342 ± 0.0120.131 ± 0.0050.208 ± 0.014
Full (10−4,0.2)(10^{-4},0.2)0.011 ± 0.0020.345 ± 0.0120.130 ± 0.0040.205 ± 0.011

The two penalties play different roles. Balancing prevents the codes from collapsing: without it, 6 of 8 codes are empty on headlines, so the policy stays close to the behavior policy, with a small RA error but almost no improvement. With balancing, PolicyLift improves the policy on headlines. At our weight, the overlap penalty changes the improvement only slightly, and its role is to control the overlap term of Theorem 5.6 (Lemma C.7). On synthetic data, β=0\beta=0 and β=10−4\beta=10^{-4} give similar improvements (0.146 vs. 0.150), so our main synthetic choice β=0\beta=0 is not critical.

Action: We added the link to Section 5.3, Lemma C.7 (Appendix C.8), and the ablation on all three datasets to Appendix G.5 (Table 18).

W4/Q4. Direct reranking baseline. Thank you for this very helpful suggestion. Best-of-N with an outcome model fit on the same logs uses the same information as PolicyLift and can perform strongly when this model ranks individual actions well, but without a guarantee, since its value depends on how well the model extrapolates across raw actions. Your suggestion motivated us to go one step further: our learned codes and code-level policy are modular and can be combined with reranking as an alternative Stage-3 instantiation rule. In this within-code reranking, we draw NN actions from the behavior system and return the one with the highest predicted outcome among those with the selected code (or among all NN draws if none has it).

We compare behavior-consistent lifting (BCL), Best-of-N, and within-code reranking at the same average number of behavior draws per context (in brackets). Best-of-N and within-code reranking use the same regularized outcome model, tuned on logged validation data (5 seeds):

Instantiation ruleSyntheticHeadlinesOfflinePrompts
Best-of-N0.129 ± 0.034 [5.4]0.014 ± 0.000 [9.4]0.231 ± 0.005 [12.4]
BCL (PolicyLift, Ours)0.145 ± 0.008 [5.5]0.011 ± 0.002 [9.4]0.130 ± 0.004 [12.5]
PolicyLift w/ within-code reranking (Ours)0.131 ± 0.034 [5.4]0.013 ± 0.001 [9.4]0.238 ± 0.003 [12.4]

(1) With within-code reranking, our codes achieve the highest value on OfflinePrompts (0.238 vs. 0.231 for Best-of-N). (2) BCL achieves the highest improvement on synthetic data, and (3) on headlines, all rules are within 0.003.

⇒\Rightarrow Our learned codes perform well with both instantiation rules. BCL is the rule of choice for high-stakes decisions where improvement must be guaranteed, and within-code reranking for low-stakes decisions where maximizing value matters more.

Action: We added within-code reranking to Section 6, a remark to Section 7, and all instantiation rules, including a PoTEC-style rule, to Appendix G.6 (Table 19).

We thank you again for your detailed review, which considerably strengthened our paper. We hope that our responses and revisions resolve all of your points. If anything remains unclear or would further support a positive assessment, we would be very grateful for a short note and are happy to provide additional clarifications during the discussion period.

Reviewer_7BaeOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This work studies how off policy learning should deal with high-dimensional action spaces that require generative models. The fundamental idea is to deconstruct high dimensional actions into discrete codes (e.g. skills), learn a policy over these codes, and map those codes back to the raw actions via known unconditional generator via rejection sampling. For the code-learning phase, a set of codes is learned such that the codes are predictive of the raw action, the codes are similar in distribution to the uniform distribution, and the distribution of global codes is close to the marginal distribution of codes for each action. The policy is then trained via a regression objective on the offline data. In terms of evaluation procedure, mostly experiments are using synthetic data along with a previous OfflinePrompts benchmark.

Critical strengths

The setup and proposed methodology in the paper appears reasonable, and the authors have done a theoretical analysis on the proposed method showing a lower bound on the estimation error.

Critical weaknesses

The primary concern in this paper is that the method is not presented clearly. Section 4 uses terminology and abbreviations that are not defined (e.g. IPW, DR). In general this section introduces many technical terms but does not provide a simple explanation for what the method is doing in practice or in an implementation. The proposed method has multiple stages but experiments do not ablate the effect of each stage.

In the experimental setup, the closest work is mentioned to be Kiyohara 2025, but this baseline does not appear in Figure 2.

There are considerable papers that study skill discovery in offline RL which should be cited and discussed. In general the paper is missing references in general to field of deep reinforcement learning.

Questions for authors

Why must a gumbel-softmax network be used to learn the discrete codes? Why not a categorical loss?

4 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your feedback and the opportunity to clarify our method. We are glad that you found the setup, methodology, and theoretical analysis reasonable. We mark all resulting paper changes as Action, and all changes are highlighted in blue in the revised PDF.

W1. Clarity. Thank you for this valuable feedback! A clear presentation for readers from various backgrounds is indeed very important to us.

(a) Abbreviations. Upon reading your comment, we realized that Section 4 used the standard terminology of causal inference and off-policy evaluation but without introducing it for readers from adjacent fields. The abbreviations IPW and DR refer to the well-established estimators of inverse propensity weighting (Horvitz & Thompson, 1952) and doubly robust estimation (Dudík et al., 2014), which we introduce in Section 2. Similarly, overlap (Rosenbaum & Rubin, 1983) and balanced representations (Johansson et al., 2016; Shalit et al., 2017) follow their standard use in this literature. We now write out all abbreviations again at their first use there.

(b) Clear explanation. We also revised Section 4 to make it easier to follow. In particular, we added a practical explanation of each step here already, especially before the neural implementation later described in Section 6. In doing so, we explain each step with the running example of our paper (i.e., headline generation).

Action: We wrote out all abbreviations in Section 4 and expanded the explanation of each step using the headline case as a running example.

W2. Ablation of each stage. Thank you for raising this point. We would like to kindly clarify that our experiments already ablate each of the three stages: (a) Stage 1: Figure 2(b) varies the quality of the codes with an oracle Stage 2, and Table 2 replaces our codes with random, kk-means, or predictive-only codes on all three datasets. (b) Stage 2: Figure 2(c) shows how overlap affects the Stage-2 estimation error, and Table 2 replaces our objective with MIPS-, OffCEM-, and DSO-adapted objectives on all three datasets. (c) Stage 3: Figure 2(a) replaces behavior-consistent lifting with uniform, context-free, and learned decoding, which lose value as the reward varies more within codes.

Following your comment, we now additionally ablate Stage 3 on all three datasets by comparing behavior-consistent lifting with alternative rules that turn the selected code into an action (instantiations), i.e. Best-of-N, a PoTEC-style rule, and within-code reranking. This also shows that our learned codes perform well with these alternative rules.

Action: We added this new Stage-3 ablation to Section 7 and Appendix G.6.

AuthorsOpenReview ↗

Response (2/2)

W3. Kiyohara et al. (2025) in Figure 2. Thank you for raising this point. We would kindly like to clarify that we already include Kiyohara et al. (2025) in our comparisons, but not in Figure 2: Figure 2 is not a benchmark, but validates the theoretical properties of PolicyLift by isolating its three stages: each panel varies one property of the data (within-code reward, code predictiveness, or overlap) and compares variants of the corresponding stage (e.g., behavior-consistent lifting vs. a learned decoder), while the other stages are fixed to their oracle versions (e.g., the true codes). Since these oracles must be known, we use only our synthetic dataset here (one of our three), which we designed for this purpose, so its actions are numeric vectors without any LLM or prompts. The full method of Kiyohara et al. (2025) (DSO) requires observed prompts and is therefore not directly applicable here. However, since it is the closest related work, we include it in two other ways: (a) We adapted its estimator as a Stage-2 objective (DSO-adapted) and report it on all three datasets in Table 2, including the synthetic data. (b) On OfflinePrompts, where prompts are observed, we additionally compare with the full DSO method (Table 3), even though it uses this extra information.

Action: We renamed the Stage-2 rows of Table 2 to MIPS-, OffCEM-, and DSO-adapted to make this important comparison even clearer and more visible.

W4. Skill discovery and offline RL. Thank you for these suggestions. We agree that skill discovery in offline RL is an important area, but it has quite a different motivation than our current setting. Skill-discovery methods learn latent skills or action codes from sequences of actions from logged trajectories and deploy them through a learned decoder or low-level policy (e.g., Pertsch et al., 2021; Ajay et al., 2021; Jiang et al., 2023; Luo et al., 2023). In contrast, we are essentially learning a discrete code (i.e., a learned cluster/embedding) of a single action. In particular, PolicyLift focuses on single action encoding that is relevant to policy improvement, which point to two large differences:

(a) Codes optimized for the decision. Our codes are learned to be predictive of the outcome and to satisfy overlap, i.e., the contexts of the logged actions with a given code follow the overall distribution of contexts, so that the policy can reliably select every code for every context. This directly targets how much the policy can improve and how reliably it can be learned (Theorems 5.5 and 5.6). Skill latents, in contrast, are typically learned to reconstruct the actions, so they are not directly optimized for the subsequent policy learning.

(b) No decoder. In skill discovery, a learned decoder must generate new actions from the latent codes, which is expensive to train for text and hard to learn well from limited logged data. Its actions can then leave the support of the logged data and even perform worse than the behavior policy. PolicyLift instead samples from the existing behavior system for the given context, so it needs no generator training, and the improvement of the policy over the codes provably carries over to the deployed actions.

Both differences also implicitly appear in our ablations: kk-means codes, which resemble skills learned from the actions alone, yield smaller improvements on all three datasets (Table 2), and learned decoding loses value in Figure 2(a). If you have further specific works in mind, we would be very happy to discuss and compare them as well.

Action: We added a paragraph on skill discovery in offline RL to Section 2 and extended Appendix B.

Q1. Gumbel-Softmax. Thank you for the opportunity to elaborate on this. A categorical (cross-entropy) loss requires a target label for each action, i.e., its correct/optimal code. Such labels do not exist, since the codes are exactly what we want to learn. Instead, we learn the codes through our Stage-1 objective, so that they are predictive of the outcome and satisfy overlap and balance. The challenge is that assigning an action to one discrete code (taking the arg-max) has no gradient, so the encoder could not be trained with this objective. The Gumbel-Softmax (Jang et al., 2017) solves exactly this: it is a differentiable approximation of choosing one discrete category. Hence, the network still assigns each action to one code, while the gradients of our objective flow back into the encoder. This allows us to learn the discrete codes end-to-end and without any code labels, such that we can directly optimize for the properties that matter for policy improvement (outcome predictiveness and distributional balancing).

Action: We added this explanation to Section 6.

We hope that these clarifications resolve your concerns, and we thank you again for your feedback. We would be very happy to address any remaining points or questions during the discussion period.

Thank you to the authors for the detailed response, which is acknowledged. Several of my concerns have been addressed via the responses here, and I have updated my score accordingly.

Thank you very much for your positive response and for raising your score! We truly appreciate your engagement during the discussion. Your feedback has considerably improved the presentation and clarity of our paper, and we are very grateful for it.

Reviewer_YcFJOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PolicyLIFT, an Off-Policy Learning (OPL) framework that starts from logged data and focuses on generative structured actions (such as text generated by LLMs). The core issue is that the action space generated by the historical behavior policy μ\mu is high-dimensional and structured (text, images, planning), and the target policy needs to generate new actions rather than selecting from a fixed action set; meanwhile, the overlap relied on by traditional OPL is almost impossible to achieve in structured action spaces (the behavior system only covers a very small subset of the action space), and "learning a policy in the representation space and then decoding it back to the original action" introduces a deployment gap. PolicyLIFT separates strategy learning from action generation. This work evaluates the proposed PolicyLIFT in three datasets.

Critical strengths
  • It cleanly separates the two core difficulties of structured-action OPL, decomposes deployed value into code-level value plus deployment gap, and uses BCL to drive the gap exactly to zero.

  • Stage 3 trains no decoder and uses the behavior system itself as the generator, avoiding the common pitfall of decoder-introduced error.

Critical weaknesses
  • The experimental scale and baseline coverage are limited, with a significant emphasis on synthetic/semi-synthetic components.

  • The core premise is that the behavior system can be repeatedly queried at deployment. If it is not queryable, or deployment must use a different generator (new model, new API, a retired legacy system), BCL cannot run, and the framework does not apply

Questions for authors

See weaknesses above.

1 comment

Thank you for your positive assessment and your valuable suggestions, which we incorporated into the revised paper. We are glad that you found that PolicyLift cleanly separates the two core difficulties of structured-action OPL and drives the deployment gap exactly to zero without training a decoder. We mark all resulting paper changes as Action, and all changes are highlighted in blue in the revised PDF.

W1. Experimental scale and baseline coverage. We conducted several new experiments to further strengthen our empirical evaluation, as detailed below. So far, our evaluation was mainly based on synthetic and semi-synthetic data, consistent with related work (Saito & Joachims, 2022; Saito et al., 2023, 2025) . Additionally, we also evaluated our framework on the realistic OfflinePrompts benchmark with real MovieLens users and Mistral-7B-generated text (Kiyohara et al., 2025). To address your comment, we additionally ran the following new experiments:

New experiments. (i) In addition to the comparisons in Tables 2 and 3, we now also compare additional rules that turn the selected code into a generated text (instantiation rules): Best-of-N, a PoTEC-style rule, and within-code reranking, which returns, among several behavior draws with the code selected by our code-level policy, the one with the highest outcome predicted by an outcome model fit on the logged data. With within-code reranking, our codes achieve the highest value on OfflinePrompts (0.238 vs. 0.231 for Best-of-N). (ii) With new stochastic generations from Mistral-7B for held-out users and held-out movies, PolicyLift improves the policy by 0.114, comparable to the original setting (0.130), and 99.9% of the deployed sentences are new. (iii) On movies that never appear in the logged data, PolicyLift keeps also most of its improvement (0.112). (iv) Across thresholds on the code propensity and numbers of draws per context, the improvement is positive in all settings and never decreases with more draws.

Beyond this, we agree that additional real-world validation such as a live A/B test, e.g., in personalized online marketing, would be exciting. Such a test would evaluate one specific deployment and lies beyond the scope of methods papers such as the works above and ours. A rollout of PolicyLift would also be low-risk by design, since the deployed policy only returns texts generated by the existing behavior system.

Action: We added the new experiments to Appendices G.3, G.4, and G.6, and the alternative instantiation rule to Section 6 and the Remark in Section 7.

W2. Queryable behavior system. Thank you for this precise observation. PolicyLift indeed assumes sampling access to the behavior system at deployment (Section 3). We believe this requirement is mild in practice, for two reasons:

(a) Only sampling access is needed, i.e., no model weights, likelihoods, or training data, and only a few draws per context (12.5 on average on OfflinePrompts). This is the same access as for Best-of-N reranking, and less than fine-tuning (model weights) or controlled decoding (decoding-time access) require.

(b) The behavior system is available by design in the setting PolicyLift targets. PolicyLift improves the decisions of a deployed generator, and this generator is both the system that logged the data and the system used for deployment. For example, in headline generation, a platform logs reader engagement for the headlines of its own generator and then improves the headlines of exactly this generator. In OfflinePrompts, the behavior system is the open-weight Mistral-7B, which thus also remains directly available at deployment.

Beyond this setting, generators are sometimes replaced in practice. While behavior-consistent lifting then loses its improvement guarantee, our learned codes from Stages 1 and 2 can still be useful when combined with a different instantiation rule, e.g., within-code reranking from W1(i), provided that its outcome model ranks the texts with the selected code well.

Action: We sharpened this limitation in Appendix A, including (a) and the case of a new generator.

We hope that these additions further strengthen your positive assessment, and we are happy to answer any further questions.

Comments on the paper

1 comment

Thank you very much to all reviewers for the constructive evaluation of our paper and the helpful comments! We addressed all of them in our responses below and uploaded a revised version of the paper, in which we highlight all changes in blue.

Our main improvements are the following:

  • Fresh generations and held-out contexts. We show that PolicyLift also improves the policy on OfflinePrompts

    • (i) with fresh stochastic generations from Mistral-7B for held-out users and held-out movies (improvement of 0.114, with 99.9% new sentences),

    • (ii) on movies that never appear in the logged data, and

    • (iii) on headlines with text embeddings only (new Appendix G.3).

  • Deployment reliability.

    • (i) We derive the exact deployed value under a finite draw cap and show that errors in the estimated propensities and a limited number of draws can scale the improvement down but never reverse it (new Section 5.1, Proposition C.5).

    • (ii) We confirm this empirically with a sweep over the threshold τ\tau and the draw budget TmaxT_{max}: the improvement is positive in all settings and matches Proposition C.5 (new Appendix G.4).

  • Overlap justification.

    • (i) We prove that the overlap term of Theorem 5.6 holds for the implemented sliced Wasserstein penalty on learned representations whenever the code-level prediction error is a Lipschitz function of the representation. Hence, driving the implemented penalty to zero also drives the overlap term to zero (new Lemma C.7 in Section 5.3).

    • (ii) We ablate the loss terms and show the distinct roles of overlap and balance (Appendix G.5).

  • Alternative instantiation rules. We compare behavior-consistent lifting with Best-of-N, a PoTEC-style rule, and a new within-code reranking on all three datasets. We find that our learned codes perform well with all rules, and that our within-code reranking achieves the highest value on OfflinePrompts. This shows that our PolicyLift is highly modular: behavior-consistent lifting guarantees improvement for high-stakes decisions, and within-code reranking can maximize value in low-stakes decisions (Sections 6 and 7, Appendix G.6).

  • Clarity and related work. We appreciate the suggestion to make sure our paper is clear to researchers working across the space of RL and causal inference. Accordingly, we expanded our explanation of the different steps and provided more practical elaborations for each step in Section 4 using our running example. We wrote out all abbreviations, explained our choice of Gumbel-Softmax (Section 6), and added related work on skill discovery in offline reinforcement learning (Section 2, Appendix B).

We incorporated all changes as highlighted additions, and the method and main results remain unchanged.

We also welcome the opportunity to address any remaining concerns or new suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.