acceptodds
Under review as a conference paper at ICLR 2027

Optimizing the Ineffable: Generative Policy Learning for Human-Centered Decision-Making

Abstract

Algorithmic decision-making is widely adopted in high-stakes applications affecting our daily lives but often requires human decision-makers to exercise their discretion within the process to ensure alignment. Explicitly modeling human values and preferences is challenging when tacit knowledge is difficult to formalize, as Michael Polanyi observed, "We can know more than we can tell." To address this challenge, we propose generative near-optimal policy learning (GenNOP). Our framework leverages a conditional generative model to reliably produce diverse, near-optimal, and potentially high-dimensional stochastic policies. Our approach involves a re-weighting scheme for training generative models according to the estimated probability that each training sample is near-optimal. Under our framework, decision-making algorithms focus on a primary, measurable objective, while human decision-makers apply their tacit knowledge to evaluate the generated decisions, rather than developing explicit specifications for the ineffable, human-centered objective. Through extensive synthetic and real-world experiments, we demonstrate the effectiveness of our method.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
generative models, human-AI collaboration, human-centered decision-making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dETUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Generative Near-Optimal Policy Learning (GenNOP), a framework for human-centered decision-making in settings where algorithmic objectives are measurable but important aspects of human judgment are difficult to formalize. The central premise is that optimizing a measurable outcome YY alone may lead to decisions that are suboptimal with respect to an overall utility V=v(Y,U)V=v(Y,U), where UU represents an implicit human-centered objective. The synthetic results demonstrate substantial improvements in near-optimal action coverage over several baselines.

Critical strengths

  1. A meaningful formulation of human–algorithm complementarity that avoids explicitly modeling implicit human preferences.

The most compelling aspect of the paper is its formulation of human-centered decision-making as a problem in which measurable performance and implicit human preferences are handled by different components.

  1. A concrete generative-learning formulation supported by synthetic results.

The paper goes beyond a conceptual proposal by developing a concrete learning procedure combining a conditional extreme-value filter, propensity-based reweighting, and generative modeling.

The derivation of the reweighted training distribution is particularly useful because it clarifies how observational data can be transformed into a distribution that emphasizes potentially near-optimal actions while reducing dependence on historical action assignments.

The synthetic results provide encouraging evidence. In Table 2, GenNOP substantially improves both precision and recall relative to the evaluated baselines. The ablations further demonstrate that the near-optimality filter is important, while the end-to-end simulations suggest that preserving diverse alternatives can improve decision quality when implicit preferences matter.

These results support the feasibility of the proposed approach and suggest that learning a distribution of acceptable actions, rather than a single optimal action, can be beneficial in appropriately structured decision problems.

Critical weaknesses
  1. A substantial gap remains between the theoretical results, the learned distribution, and the claimed human-centered decision objective.

The paper's central motivation is to improve overall utility VV through human selection among actions that are near-optimal in measurable performance YY. However, the theoretical analysis does not sufficiently establish this connection.

In particular, Proposition 3.2 establishes a local ordering relationship between UU and VV for candidates whose YY-values lie sufficiently close to a particular reference point. In Appendix E, this reference point is the YY-coordinate of the interior maximizer of overall utility VV. In contrast, Definition 3.3 and the proposed policy in Definition 4.1 define near-optimality relative to the maximum attainable measurable outcome YY.

These reference points are generally distinct under the paper's own assumptions. Indeed, Proposition 3.1 explicitly considers the case in which maximizing YY differs from maximizing VV.

Consequently, the local ordering result does not directly justify selecting the action with the largest UU from the ϵ\epsilon-optimal set used by GenNOP. Moreover, the tolerance in Proposition 3.2 is sample-dependent, whereas the algorithm uses a practitioner-specified tolerance. The practical implication of the proposition is therefore limited.

As a consequence, the theoretical and algorithmic connections underlying the paper's central claim remain insufficiently established.

  1. The experimental evidence does not adequately establish the practical effectiveness of the proposed human-centered decision-making framework.

The synthetic experiments are promising, but several aspects limit their ability to substantiate the central claim.

First, the end-to-end evaluations simulate human judgment using explicitly defined functions for UU and VV. This is reasonable for controlled experimentation, but the core motivation of the paper concerns preferences that cannot readily be formalized. There is no evaluation involving actual human decision-makers, making it difficult to assess whether the proposed alternatives are useful, interpretable, or meaningfully improve human decisions.

Second, the preference-axis experiment in Figure 6 uses an oracle tolerance

ϵ=max⁡aYa−YaV⋆,aV⋆=arg⁡max⁡aVa,\epsilon=\max_a Y_a-Y_{a_V^\star}, \qquad a_V^\star=\arg\max_a V_a,

which requires knowledge of the true overall-utility maximizer. The authors acknowledge that this is not a practical calibration procedure. Nevertheless, the dependence of performance on ϵ\epsilon is substantial, as illustrated by Figure 5 and Appendix K.4. Since selecting this parameter is an essential part of the proposed framework, stronger evidence under realistically chosen tolerances is necessary.

Third, the high-dimensional evaluation relies on Fashion-MNIST with class-dependent outcomes. Although the generated actions are images, the optimization structure is determined primarily by a small number of discrete classes. Furthermore, recall is assessed through class-level coverage. This limits the evidence that GenNOP can effectively handle complex high-dimensional action spaces with genuinely high-dimensional outcome landscapes.

Finally, the real-data experiments provide relatively weak validation. The MIMIC-IV results are evaluated primarily through two-dimensional distributional visualizations rather than quantitative measures of near-optimality, decision quality, or human benefit.

Importantly, Appendix K.5 states that the clinical experiments use a conditional Gaussian-process generator with GEV-derived observation weights, rather than the conditional diffusion architecture emphasized in the proposed implementation. This distinction does not invalidate the general framework, but it limits the relevance of these experiments as evidence for the specific approach being proposed.

Additionally, the observational clinical data involve substantial causal-identification challenges. The selected covariates are limited, the analysis includes only patients who survived to discharge, and hospital length of stay is influenced by many factors beyond the recorded medication decisions. The authors appropriately avoid claiming clinical optimality, but this also means the real-data experiments offer little direct evidence for the central decision-making objective.

The current evaluation therefore establishes that GenNOP can generate plausible and diverse alternatives under selected conditions, but not convincingly that these alternatives improve human-centered decisions in realistic settings.

This weakness is consequential because the principal advantage claimed for the method is improved human–algorithm decision-making, whereas the strongest evidence concerns a surrogate distribution-learning task.

Questions for authors
  1. Proposition 3.2 establishes local consistency between the ordering induced by UU and the ordering induced by VVaround the measurable-outcome coordinate of the overall-utility optimum. However, the ϵ\epsilon-optimal action set used by GenNOP is defined around the maximum of YY.

How does Proposition 3.2 justify selecting the UU-maximizing action from the set defined in Definition 4.1? Can the authors establish a corresponding result for that set, with a nontrivial tolerance that does not depend on the realized candidate sample?

A rigorous clarification or a corrected guarantee would meaningfully strengthen the theoretical contribution.

  1. Evidence concerning practical tolerance selection and comparison with simpler alternatives.

Figure 6 relies on an oracle choice of ϵ\epsilon, even though this information would be unavailable in the intended applications.

How does GenNOP perform when ϵ\epsilon is selected solely from observable data and a predefined acceptable loss in YY, without reference to the ground-truth overall utility VV?

Additionally, how does the method compare with a simpler baseline that first estimates expected outcomes and then applies threshold-based filtering and diversity-aware candidate selection?

1 comment

Thank you for your review! Please see our responses below:

  1. Theoretical Link Between Fixed-ϵ\epsilon Sampling and Overall Utility: We disambiguated the measurable outcome maximum y⋆(x)y^\star(x) from the latent utility-optimal outcome pair (yV⋆,uV⋆)(y_V^\star, u_V^\star) across Sec. 3 and App. E, refined our Pareto-frontier and coverage assumptions (Assumps. E.1, E.4), and proved Lemma E.6 and Prop. 3.4 directly on the fixed tolerance window [y⋆(x)−ϵ,y⋆(x)][y^\star(x)-\epsilon, y^\star(x)]. In Prop. 3.4 (Sec. 3, p. 4), we decomposed finite-sample expected utility regret into three interpretable components: an under-exploration bias when ϵ\epsilon is too small to reach yV⋆(x)y_V^\star(x), a linear over-exploration outcome loss when ϵ\epsilon exceeds the optimal trade-off, and an O(ϵ/m)O(\sqrt{\epsilon/m}) finite-sample selection gap across the mm generated candidates. Using this decomposition, we proved that for an interval of fixed ϵ>0\epsilon > 0 and sufficiently large mm, human selection over ϵ\epsilon-optimal candidates strictly dominates deterministic YY-maximization in expected overall utility VV, theoretically explaining the U-shaped ϵ\epsilon trade-off and O(m−1/2)O(m^{-1/2}) convergence observed in Fig. 5. We also added Prop. 4.2 (Sec. 4.1, p. 6; App. H.2) bounding the total-variation distance between our importance-weighted surrogate target qϵq_\epsilon and the uniform ϵ\epsilon-optimal policy πϵ⋆\pi_\epsilon^\star, and showing that stabilized propensity weighting preserves Prop. 3.4's finite-sample guarantees with the near-frontier density constant scaled by κp(x)−1\kappa_p(x)^{-1}.
  2. Non-Oracle ϵ\epsilon Calibration, Matched-Budget Baseline Grids, and Threshold+FPS Comparison: We added observable-only ϵ\epsilon calibration rules based on empirical outcome dispersion (standard-deviation, interquartile-range, and inter-quantile multiples of observed YY) in Sec. 3 (p. 5) and evaluated them over five seeds in Sec. 5 (Table 3) and App. K.4 (Tables 8–10, Fig. 13) alongside Threshold+FPS, Hard Threshold+Uniform, and full hyperparameter grids for DDOM, AWR, GP-UCB, and Perturb at matched candidate budgets mm. On the 1D multi-peak benchmark (Sec. 5, Table 3, Table 10), where unweighted outcome regression smooths over narrow secondary modes, GenNOP Pure at observable ϵ=q95−q75\epsilon = q_{95}-q_{75} achieves the lowest mean regret across all evaluated methods and significantly outperforms Threshold+FPS, DDOM, AWR, and IPTW Only while matching oracle Hard Threshold+Uniform. On the 2D Ackley benchmark (Sec. 5, Table 3, Tables 8–9, Fig. 13), observable rules for GenNOP Pure match Pure Oracle ϵ⋆\epsilon^\star without statistically significant regret differences, and GenNOP+RF at observable 1.00 SD(Y)1.00\,\mathrm{SD}(Y) significantly outperforms GP-UCB, Perturb, and Point Policy while achieving lower mean regret than grid-best DDOM and AWR; full comparisons across all threshold baselines and hyperparameter grids appear in Table 3 and Tables 8–10. Moreover, as demonstrated on 784784-D Fashion-MNIST (Sec. 5, Table 2), amortized conditional diffusion scales natively to high-dimensional continuous action spaces where grid-based thresholding and farthest-point sampling become impractical.
  3. Semi-Synthetic Fashion-MNIST Design and Quantitative MIMIC-IV Validation: We clarified in Sec. 5 and App. K.3 that Fashion-MNIST uses class-archetype outcome mappings solely as an evaluation oracle to quantify multi-modal mode coverage under confounding, while our conditional diffusion model conditions only on demographic covariates (without access to class labels) and generates directly in 784784-D pixel space, achieving substantially lower FID and higher target recall than all baselines (Sec. 5, Table 2). Furthermore, we added a five-seed patient-grouped quantitative evaluation of the Section 4 conditional diffusion and VAE pipelines on the cardio and sepsis MIMIC-IV cohorts in Sec. 5 (Table 4) and App. K.7 (Tables 13–14, Fig. 16) using train-only Stage-1 estimators and dual gradient-boosted and random-forest outcome evaluators. Across both clinical cohorts, GenNOP (Diff + GPS + GEV) achieves high held-out ϵ\epsilon-optimal precision and significantly improves precision and predicted LOS over Behavior Cloning and GenNOP w/o GEV (as well as short-stay W1W_1 on cardio and relative to AWR on sepsis), while preserving substantial intra-patient candidate diversity (Sec. 5, Table 4, Table 14).
Reviewer_2oZROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a general framework called GenNOP to tackle the alignment problem with implicit human values and preferences involved. Specifically, 1. human experts define a measurable objective YY and an optimality gap tolerance ϵ\epsilon; 2. a trained generative model generates near-optimal actions; 3. human experts select actions based on their implicit human-centered objective UU.

Experimental results show that: 1. in synthetic settings, GenNOP outperforms DRPolicyForest, GP-UCB, and DDOM on precision, recall and FID; 2. in real-data settings of mimic-icu-cardio and mimic-icu-sepsis, GenNOP shows better properties than Gaussian perturbations, where GenNOP covers more of the action range in the cardiovascular cohort and better follows the main axis of the observed actions in the sepsis cohort.

Critical strengths

The paper is well motivated and tries to tackle the important problem of alignment. The writing is also clear. The proposed paradigm is simple and easy to understand. Different experiments are provided to help understand the proposed paradigm's properties in synthetic settings.

Critical weaknesses

This paper has a critical weakness in empirical quality. Although a new human-in-the-loop framework is proposed, the experiments are never conducted with real human evaluators. Neither the synthetic nor the real-data experiment has real human evaluators involved, so the empirical evidence is insufficient to justify the effectiveness of the paradigm.

The significance of the paradigm is also a problem. Human experts here do the heavy lifting of measurable objective definition, hyperparameter ϵ\epsilon choice, and policy candidate selection, and there is no useful guidance on how the empirical value of those terms should be decided, which hurts the scalability of the approach. It would be recommended to further reduce human experts' effort in the process.

Questions for authors

It would be nice if the authors could clarify more on the best empirical practice for choosing ϵ\epsilon.

1 comment

Thank you for your review! Please see our responses below:

  1. Simulated & Observational Evaluation vs. Live Trials: We evaluated human-algorithm complementarity across our synthetic benchmarks under varying decision-maker preferences and perception noise (Sec. 5, Table 3, Figs. 5–6, 15) and added a five-seed quantitative evaluation of the Section 4 diffusion and VAE pipelines on MIMIC-IV (Sec. 5, Table 4, App. K.7, Table 14, Fig. 16). Across both settings, GenNOP substantially reduces utility regret relative to single-point outcome maximization and significantly improves held-out ϵ\epsilon-optimal precision and predicted LOS over unweighted baselines, while we explicitly discuss prospective clinical validation as essential future work in Sec. 6.
  2. Human Expert Burden vs. RLHF: We clarified in Sec. 2 and Sec. 3 that GenNOP requires zero human preference annotations during training and asks the practitioner to inspect only a small candidate set at inference time (such as m∈[5,10]m \in [5, 10] in Sec. 3), with expected regret already close to its minimum at m=10m = 10 in Fig. 5 and decaying as O(m−1/2)O(m^{-1/2}) in Prop. 3.4.
  3. Selecting ϵ\epsilon in Practice: We added practical guidelines in Sec. 3 (p. 5) for calibrating ϵ\epsilon and mm from observable data and domain knowledge alone: (a) specifying a domain-anchored tolerance in the natural units of YY (Sec. 5, Table 13), (b) calibrating ϵ\epsilon from the empirical dispersion of observed outcomes (Sec. 5, Table 3, App. K.4, Tables 8, 10, Fig. 13), and (c) monitoring effective sample size and candidate diversity across a grid of ϵ\epsilon values (Sec. 3).
Reviewer_XAdpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies decisions in which an algorithm can learn a measurable outcome YY from observational data (X,A,Y)(X,A,Y), but the final choice should also reflect a human-centered criterion UU that only a human can judge, with an unknown overall utility V=v(Y,U)V=v(Y,U). Instead of recommending arg⁡max⁡aE[Ya∣X=x]\arg\max_a \mathbb{E}[Y_a\mid X=x], the proposed GenNOP learns a conditional generative policy that approximates the uniform distribution over ϵ\epsilon-optimal actions (Def. 4.1); a human then samples mm candidates and selects one by UU. Training re-weights the denoising loss of a conditional diffusion model by w=gϵ(y,x)/p(a∣x)w=g_\epsilon(y,x)/p(a\mid x) (Eq. 4), where gϵg_\epsilon is the CDF, evaluated at y+ϵy+\epsilon, of a covariate-dependent generalized extreme value (GEV) distribution fitted to block maxima of outcomes among nearest neighbours (Sec. 4.2), and p(a∣x)p(a\mid x) is a generalized propensity score estimated with a VAE (App. I.1). Two propositions (Sec. 3, App. E) state that, under concave-frontier assumptions, the YY-maximizer and the VV-maximizer of a large sample differ with probability tending to one, and that within a sufficiently narrow YY-window, ranking by UU also ranks by VV. Experiments comprise a 1-D synthetic and a Fashion-MNIST semi-synthetic dataset scored by precision, recall and FID against DRPolicyForest, GP-UCB and DDOM (Table 2); a 1-D end-to-end simulation with a scripted selector reporting regret as a function of ϵ\epsilon, mm and preference weights (Figs. 5–6); and qualitative 2-D embeddings of generated ICU medication doses for two MIMIC-IV cohorts (Fig. 7).

Critical strengths
  1. A clear, auditable formulation with a concrete learning target. The paper turns the vague goal of “leaving room for human judgment” into a precise object, the uniform distribution over ϵ\epsilon-optimal actions (Def. 4.1, p. 5, l. 235–243). It then derives which distribution the weighted denoising loss targets (qϵ∝hϵq_\epsilon\propto h_\epsilon, p. 6, l. 278–284; App. H.2), states when this coincides with πϵ⋆\pi^\star_\epsilon, and notes how stabilized propensity weights change it (App. I.1, p. 22, l. 1134–1138). The division of labour (the algorithm handles a measurable YY up to a tolerance stated in units of YY; the human chooses by UU without anyone modelling UU) is distinct from preference-learning approaches and from generative curation, which models residual utility (Sec. 2, p. 3, l. 133–135). The paper is also candid about its limits (e.g., p. 9, l. 436–437 and l. 454–455). This formulation is the main reusable contribution and would stay valuable even if the particular estimator were replaced.

  2. Synthetic evidence that the outcome filter is the essential ingredient and that small candidate sets suffice. In Table 2 (p. 8), removing gϵg_\epsilon lowers precision from 0.85 to 0.01 (fully synthetic) and from 0.83 to 0.38 (semi-synthetic), and raises FID from 5.0 to 14.3; GenNOP has higher precision and recall than all three baselines on both datasets, and its FID (5.0) is below theirs (19.1–34.7). In the end-to-end simulation with a UU-maximizing selector, regret is U-shaped in ϵ\epsilon (about 0.15 at ϵ=0\epsilon=0, a minimum of about 0.04 near ϵ≈0.15\epsilon\approx0.15–0.20.2; Fig. 5 left, values read from the plot) and already about 0.06 with m=10m=10 candidates (Fig. 5 right). In this controlled setting this supports the paper’s qualitative thesis that a moderate tolerance can lower VV-regret.

Critical weaknesses
  1. The evidence does not support the headline claims of effectiveness, especially for human-in-the-loop use and real data.

    • Human selection is only simulated, with an oracle tolerance, and the result is mixed. The full workflow (sample mm candidates, select by UU) is evaluated only on a 1-D synthetic dataset with deterministic Y,U,VY,U,V (App. K.3, p. 27, l. 1435–1447) and a scripted selector Q=λYY+λUUQ=\lambda_Y Y+\lambda_U U (p. 8, l. 414–417); no human participants are involved (p. 9, l. 436–437). In the headline comparison (Fig. 6), GenNOP uses the oracle tolerance ϵ=max⁡aYa−YaV⋆\epsilon=\max_a Y_a-Y_{a^\star_V} computed from the ground-truth VV (p. 9, l. 438–440), while no comparable oracle tuning is described for the baselines. Even so, GenNOP is ahead only as λU→1\lambda_U\to1 (0.085 vs. 0.168 for DDOM at λU=1\lambda_U=1; p. 9, l. 432–434). Near λU=0.5\lambda_U=0.5, DDOM, GP-UCB and local perturbation have lower regret (p. 9, l. 434–435), and in Fig. 6 the lowest regret any baseline reaches on the axis (about 0.03, for GP-UCB and Perturb) is below GenNOP’s minimum (about 0.06) (values read from the plot). Because Fig. 5 (left) shows GenNOP’s regret under UU-maximizing selection ranging from about 0.04 to about 0.18 over ϵ∈[0,1]\epsilon\in[0,1], the ranking at λU=1\lambda_U=1 may depend on the oracle choice.

    • The real-data experiment evaluates neither the proposed implementation nor its objective. The MIMIC-IV results (Fig. 7) are density plots in a 2-D VAE embedding with no quantitative metric. App. K.5 states that they come from “a conditional Gaussian-process generator with GEV-derived observation weights and ϵ=15\epsilon=15” days of hospital stay (p. 30, l. 1602–1603), not from the conditional diffusion model with propensity weighting of Sec. 4, which Sec. 5 presents as “GenNOP” (p. 9, l. 450–455). The only comparator is Gaussian perturbation of recorded short-stay actions, configured differently in the two cohorts (noise s.d. 0.1×0.1\times vs. 1×1\times the training s.d.; p. 30, l. 1608–1612), so whether GenNOP “covers more of the observed action range” depends heavily on which recorded actions are perturbed and with what noise scale, choices that differ between the cohorts. The cohorts are also restricted to patients discharged alive (p. 30, l. 1576 and l. 1589–1590), a post-treatment selection, and generation is conditioned on only age and sex (cardio, l. 1591) or age, sex and SOFA (sepsis, l. 1577); the unconfoundedness assumption of App. F, on which the weights rely, is hard to defend here. The authors themselves note that the plots “do not establish that generated doses are clinically optimal or safe” (p. 9, l. 454–455).

    • Statistical support is limited. The ±\pm in Table 2 is variability over ten generated samples (p. 7, l. 374–375), and the five runs of Fig. 6 resample candidates “from each policy” (App. K.4, p. 27, l. 1452–1453); variability over training seeds is not reported, nor are the ϵ\epsilon and mm used for Table 2. Table 2 does not establish what the propensity component contributes (semi-synthetic precision 0.83±0.030.83\pm0.03 vs. 0.87±0.030.87\pm0.03 without pp; FID 5.0±1.85.0\pm1.8 vs. 6.5±3.46.5\pm3.4). I also could not reconcile the scale of the regret curves with the stated data-generating process (Question 3).

    The abstract therefore overstates the evidence when it says GenNOP can “reliably produce diverse, near-optimal” policies and that “extensive synthetic and real-world experiments” demonstrate its effectiveness (p. 1, l. 017–018 and l. 023–024). For example, GenNOP’s fifth-percentile precision is 0.47 (fully synthetic) and 0.38 (semi-synthetic) (Table 2). This is partly fixable: run the actual pipeline on MIMIC-IV with a quantitative proxy (e.g., coverage of held-out short-stay actions or doubly-robust value estimates), use non-oracle ϵ\epsilon with matched baseline tuning, and report training-seed variability. Otherwise the claims should be scoped down. In my view a clinical or human-subject study is not required for ICLR, but without one the paper should not claim demonstrated real-world effectiveness.

  2. The theory and the estimator are only loosely connected to the stated target, and the key mechanism is not isolated.

    • The propositions of Sec. 3 do not concern the procedure GenNOP runs. In App. E, y⋆y^\star is the YY-coordinate of the unique VV-maximizer, assumed interior (Assumption E.3, p. 18, l. 960–963), and candidates are assumed to lie in [y⋆−ϵ,y⋆][y^\star-\epsilon,y^\star] within η\eta of the frontier (E.5, l. 969–971). GenNOP’s window instead ends at y⋆(x)=max⁡aE[Ya∣X=x]y^\star(x)=\max_a\mathbb{E}[Y_a\mid X=x] (Def. 3.3, p. 4, l. 213–215), i.e., at y+y_+ in the notation of App. E, and the same symbol is used for both. The window of Prop. 3.2, [y⋆−ϵˉ,y⋆][y^\star-\bar\epsilon,y^\star], therefore lies below the unknown VV-optimal YY-value. Moreover, ϵˉ=min⁡{ϵ,(mU/LY)ΔUmin⁡}\bar\epsilon=\min\lbrace \epsilon,(m_U/L_Y)\Delta_U^{\min}\rbrace (p. 19, l. 1003–1006) shrinks with the smallest gap between sampled UU-values, and hence with the number of candidates. Prop. 3.1 follows directly from the assumption (E.3) that the VV-maximizer is interior. Neither result says anything about candidates drawn from πϵ⋆\pi^\star_\epsilon or πθ\pi_\theta with a fixed, user-chosen ϵ\epsilon. The paper concedes that the ordering result “requires a sample-dependent tolerance” (p. 5, l. 221–222) but does not discuss the mismatch in anchor point.

    • The learned distribution generally differs from πϵ⋆\pi^\star_\epsilon, and ϵ\epsilon loses its stated meaning. By the paper’s own analysis, the weighted loss targets qϵ∝hϵ(a,x)=E[gϵ(Y,x)∣A=a,X=x]q_\epsilon\propto h_\epsilon(a,x)=\mathbb{E}[g_\epsilon(Y,x)\mid A=a,X=x]. This equals πϵ⋆\pi^\star_\epsilon only for deterministic outcomes and an exact hard filter (p. 6, l. 281–284; App. H.2, p. 21, l. 1115–1117); with the VAE’s stabilized weights it becomes ∝p(a)hϵ(a,x)\propto p(a)h_\epsilon(a,x) (App. I.1, p. 22, l. 1134–1136). With noisy outcomes and a hard filter, hϵ(a,x)=P(Ya≥y⋆(x)−ϵ∣X=x)h_\epsilon(a,x)=P(Y_a\ge y^\star(x)-\epsilon\mid X=x) (l. 1116). For Gaussian outcomes, an action whose mean lies below the threshold then gets more weight the larger its outcome variance, which is a concern given the clinical motivation. Further, gϵ(y,x)=Fb,x(y+ϵ)g_\epsilon(y,x)=F_{b,x}(y+\epsilon) is the CDF of block maxima of observed outcomes under the logging policy (Sec. 4.2, p. 6, l. 295–298). Its effective cut-off thus depends on the block size bb, on how often the logging policy takes near-optimal actions, and on noise, not only on ϵ\epsilon (acknowledged qualitatively at p. 6, l. 313–317). This undercuts reading ϵ\epsilon as “an acceptable loss expressed in the units of YY” (p. 5, l. 222–224). No bound on the gap between qϵq_\epsilon and πϵ⋆\pi^\star_\epsilon is given, and the evaluated settings have deterministic outcomes (fully synthetic, App. K.3, p. 27, l. 1406–1411; end-to-end, l. 1438–1447) or “small” noise (semi-synthetic, l. 1425–1426), so robustness to noise and coverage is untested.

    • The max-stable/GEV component is not shown to be needed. The only filter ablation sets gϵ≡1g_\epsilon\equiv1 (App. K.1, p. 26, l. 1374–1376). This shows that some outcome filter is essential, but not that the GEV fit beats simpler outcome-based weights, e.g., a hard threshold on a local kkNN maximum or quantile, exponentiated-outcome weights as in advantage-weighted regression (Peng et al., 2019), or re-weighting toward high outcomes as already done by DDOM (Krishnamoorthy et al., 2023), one of the baselines. Table 1 compares GEV negative log-likelihoods across block sizes, i.e., likelihoods of different random variables (maxima over 10 vs. 30 observations), and says nothing about downstream precision or regret.

    This is partly fixable: use consistent notation, prove a statement about the window actually used, add a sweep over noise and coverage, and add ablations against simpler filters.

Minor points. (i) “Table A” (p. 4, l. 184) should be Table 3; App. H.1 refers to a nonexistent “Equation (4.1)” (p. 20, l. 1066); the =d\overset{d}{=} in Eq. (3) means equality up to an additive constant.

(ii) The legend of Fig. 4 labels the band “95%-CB of Estimated y⋆(x)y^\star(x)”, which contradicts p. 7, l. 336–337.

(iii) The fully synthetic outcome Y=exp⁡(−50(A−X)2)sin⁡(9πA)Y=\exp(-50(A-X)^2)\sin(9\pi A) (p. 27, l. 1410–1411) is negative whenever A∈(1/9,2/9)A\in(1/9,2/9), contrary to “bounded on (0,1)(0,1)” (p. 7, l. 364–365).

(iv) In the proof of Prop. 3.2 (p. 19, l. 998–1001), the second bracket applies E.4 at (Y(j),U(i))(Y^{(j)},U^{(i)}), but E.4 is assumed only for u,u′∈[g(y)−η0,g(y)]u,u'\in[g(y)-\eta_0,g(y)] (p. 18, l. 964–968), and U(i)U^{(i)} need not lie in that band at y=Y(j)y=Y^{(j)}; stating E.4 on a rectangle around (y⋆,g(y⋆))(y^\star,g(y^\star)) would fix this.

(v) The FID protocol uses five sampled actions per profile (p. 27, l. 1429–1433); please state the total sample size and the feature extractor, since FID is biased at small sample sizes.

(vi) The GEV “regularization” (Sec. 4.3; Tables 1 and 5) is never defined, and the logging policy of the end-to-end dataset is not specified (App. K.3), although it is needed to interpret the large effect of removing pp in Fig. 5.

(vii) The reproducibility statement mentions repetitions “needed to establish statistical significance” (p. 10, l. 498–499), but no significance tests are reported.

(viii) Related work does not discuss outcome-weighted policy learning from logged data in offline RL, e.g., advantage-weighted regression (Peng et al., 2019) and its use with diffusion policies (Kang et al., 2023, who combine diffusion policies with likelihood-based offline RL algorithms such as CRR and IQL), which is close to Eq. (4).

Questions for authors
  1. [Clarification] Which model produced Fig. 7? App. K.5 states that the clinical notebooks use a conditional Gaussian-process generator with GEV-derived weights and ϵ=15\epsilon=15 days (p. 30, l. 1602–1603). Were the diffusion model and the propensity weights of Sec. 4 ever run on MIMIC-IV? Please also report the distribution of hospital length of stay (to judge how restrictive ϵ=15\epsilon=15 days is) and a quantitative comparison, e.g., coverage of held-out short-stay actions or doubly-robust estimates of YY for generated doses, against at least one baseline tuned on the same data. Results of the actual pipeline with such a metric would raise my assessment of the real-data claim; confirmation that only the GP variant was run would lower it.

  2. [Clarification] Tolerance and baseline tuning in Fig. 6. Is the oracle ϵ=max⁡aYa−YaV⋆\epsilon=\max_a Y_a-Y_{a^\star_V} computed per covariate? How were the baselines’ free parameters chosen (perturbation radius, DDOM’s conditioning value and guidance strength, GP-UCB’s exploration parameter and how a distribution of actions is obtained from it, and the action discretization for DRPolicyForest)? How does GenNOP compare when ϵ\epsilon is fixed without access to VV (e.g., the values 0.05, 0.2 and 0.8 used in Fig. 13) and the baselines receive comparable tuning? If GenNOP’s advantage at λU≈1\lambda_U\approx1 persists, my concern in Weakness 1 would be substantially reduced; if it disappears, the comparative claim would not hold.

  3. [Clarification] Scale of the regret curves. I evaluated the end-to-end data-generating process of App. K.3 (p. 27, l. 1438–1447) numerically on an action grid over [0,1][0,1] (the action support is not stated). For X∼U(0.1,0.9)X\sim U(0.1,0.9) I obtain Ex[v⋆(x)]≈0.37\mathbb{E}_x[v^\star(x)]\approx0.37, and the expected regret of the YY-maximizing action is ≈0.09\approx0.09 (at most 0.230.23 for any xx). Since Y,U≥0Y,U\ge0 implies V≥0V\ge0, the regret at any xx cannot exceed v⋆(x)v^\star(x), so the expected regret cannot exceed ≈0.37\approx0.37. Yet Fig. 6 shows ≈0.6\approx0.6 for the ground-truth policy at λU=0\lambda_U=0 (where the selector picks the candidate with the highest YY), and Fig. 5 (right) shows ≈0.44\approx0.44 for GenNOP w/o gϵg_\epsilon at m=1000m=1000. Separately, Fig. 5 (right) shows ≈0.04\approx0.04 for GenNOP at m=100m=100 under UU-maximizing selection, whereas Sec. 5 reports 0.085±0.0040.085\pm0.004 at λU=1\lambda_U=1 with 100 candidates (p. 9, l. 432–434). Is regret normalized, evaluated on a particular set of covariates, or computed under a different data-generating process or trained model? Are models retrained across the runs of Fig. 6 and the samples of Table 2? A clear reconciliation would remove this concern; if the figures came from a setup other than the one described, the quantitative claims of Sec. 5 would need to be re-derived.

  4. [Clarification] Which target was learned in each experiment? Algorithm 1 (step 8, p. 21) and App. I.1 allow either an exact propensity density or the VAE ratio, and App. I.1 notes that the stabilized ratio yields a target ∝p(a)hϵ(a,x)\propto p(a)h_\epsilon(a,x) (p. 22, l. 1134–1136). For each experiment, which propensity estimate was used? If 1/p(a∣x)1/p(a\mid x) was recovered, how was p(a)p(a) obtained (e.g., for image actions)? Were weights clipped or normalized? Please also report ϵ\epsilon, mm, (k,b)(k,b), the GEV regularizer, the guidance scale and the network architectures used for Table 2. If the stabilized weights were used, the precision/recall comparison against the uniform πϵ⋆\pi^\star_\epsilon should be re-interpreted.

  5. [Clarification] Anchor of the theory. In Prop. 3.2 and App. E, is y⋆y^\star the YY-coordinate of the VV-maximizer (Assumption E.3) rather than max⁡aE[Ya∣X=x]\max_a\mathbb{E}[Y_a\mid X=x] (Def. 3.3)? If so, can you state a result for the window GenNOP actually uses, [y⋆(x)−ϵ,y⋆(x)][y^\star(x)-\epsilon,y^\star(x)] with fixed ϵ\epsilon? For example, under what conditions does the UU-maximizer among mm draws from πϵ⋆\pi^\star_\epsilon have higher expected VV than the YY-maximizer, and how does this depend on mm and ϵ\epsilon? A result of this kind would materially strengthen the soundness of Sec. 3.

  6. [Suggestion / future work] Isolate the GEV filter and test robustness. Using the same diffusion backbone, compare the GEV filter with simpler outcome weights: a hard threshold on a local kkNN maximum or quantile, exponentiated-outcome weights as in advantage-weighted regression, and DDOM-style re-weighting. Also sweep the outcome noise and the logging policy’s mass on near-optimal actions, reporting the fraction of generated actions with E[Ya∣x]<y⋆(x)−ϵ\mathbb{E}[Y_a\mid x]<y^\star(x)-\epsilon. An end-to-end comparison with generative curation (Li & Zhu, 2024), which targets the same selection setting, would also clarify when it is preferable not to model UU.

1 comment

Thank you for your review! Please see our responses below:

  1. W1 & Q1: We added empirical hospital LOS distributions for the cardio and sepsis cohorts (Sec. 5, App. K.7, Table 13, showing that ϵ=15\epsilon = 15 days aligns closely with one standard deviation of survivor LOS) and conducted a five-seed patient-grouped evaluation of the Section 4 diffusion and VAE pipelines (Sec. 5, Table 4, App. K.7, Table 14, Fig. 16), while clarifying in Sec. 5 that Fig. 7 presents a qualitative Gaussian-process visualization. Across both cohorts, GenNOP (Diff + GPS + GEV) achieves high held-out ϵ\epsilon-optimal precision and significantly improves precision and predicted LOS over Behavior Cloning and GenNOP w/o GEV (as well as short-stay W1W_1 on cardio and relative to AWR on sepsis) while maintaining substantial intra-patient candidate diversity (Sec. 5, Table 4, Table 14).
  2. W1 & Q2: We added observable-only ϵ\epsilon calibration rules based on empirical outcome dispersion in Sec. 3 and evaluated them over five seeds alongside Threshold+FPS, Hard Threshold+Uniform, and full hyperparameter grids for DDOM, AWR, GP-UCB, and Perturb (Sec. 5, Table 3, App. K.4, Tables 8–10, Fig. 13). On the 1D benchmark, our recommended sharp-landscape rules (q95−q75q_{95}-q_{75} and 0.50 SD(Y)0.50\,\mathrm{SD}(Y)) match the oracle tolerance without statistically significant differences, and q95−q75q_{95}-q_{75} significantly outperforms tuned DDOM, AWR, Threshold+FPS, and IPTW Only while matching oracle Hard Threshold+Uniform; on the 2D benchmark, observable rules for GenNOP Pure match Pure Oracle ϵ⋆\epsilon^\star, and GenNOP+RF at 1.00 SD(Y)1.00\,\mathrm{SD}(Y) significantly outperforms GP-UCB, Perturb, and Point Policy while attaining lower mean regret than grid-best DDOM and AWR (Sec. 5, Table 3, Tables 8–10).
  3. W1 & Q3: We documented the complete data-generating equations and evaluation protocols for both end-to-end synthetic benchmarks in Sec. 5 and App. K.3. We clarified that Fig. 5 presents the oracle-filter hyperparameter sensitivity study on the 1D multi-peak Beta benchmark (where unweighted IPTW Only selects the propensity-weighted maximum over raw logging-policy draws while normalized importance resampling decreases monotonically with mm), paired with the trained five-seed benchmark in Table 3, whereas Fig. 6 evaluates the separate 2D Ackley Circular-Pareto benchmark.
  4. W1/W2 & Q4: We specified the propensity-weighting targets used in each experiment (locally normalized inverse propensity weights in 1D and stabilized marginal-to-conditional ratios in Fashion-MNIST and MIMIC-IV, targeting the prior-regularized policy characterized in Prop. 4.2), defined the GEV L2L_2 and support-barrier regularizers in Sec. 4.3 and App. J, and tabulated all model architectures and training schedules in App. I.1, K.3, K.7.
  5. W2 & Q5: We disambiguated the measurable outcome optimum y⋆(x)y^\star(x) from the overall utility optimum (yV⋆,uV⋆)(y_V^\star, u_V^\star) across Sec. 3 and App. E, refined the support and regularity conditions (Assumps. E.1, E.4), and proved Prop. 3.4 (Sec. 3, p. 4) directly on the fixed interval [y⋆(x)−ϵ,y⋆(x)][y^\star(x)-\epsilon, y^\star(x)], showing that finite-sample regret decomposes into under-exploration bias, over-exploration loss, and an O(ϵ/m)O(\sqrt{\epsilon/m}) selection gap that guarantees strict expected utility gain over the YY-maximizer. We also added Prop. 4.2 (Sec. 4.1, p. 6; App. H.2) bounding the total-variation gap between qϵq_\epsilon and πϵ⋆\pi_\epsilon^\star and showing that stabilized propensity weighting preserves Prop. 3.4's guarantees with near-frontier density constant scaled by κp(x)−1\kappa_p(x)^{-1}.
  6. W2 & Q6: We added a five-seed controlled ablation in Sec. 5 and App. K.5 (Tables 11–12, Fig. 14) comparing Eq. (4)'s soft GEV filter against hard kk-NN maxima, empirical quantile cutoffs, AWR, DDOM, and propensity-only weights under fixed propensity estimation across outcome-noise and logging-shift sweeps. Soft GEV filtering preserves near-perfect ϵ\epsilon-optimal mode recall as outcome noise and logging shifts increase, whereas hard kk-NN maxima and exponential tilting lose recall under noise outliers, and soft GEV significantly outperforms fixed quantile cutoffs and propensity-only weighting (Sec. 5, App. K.5, Tables 11–12, Fig. 14).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Revision

We thank the reviewers for their questions and constructive feedback. We made the following revisions to our paper:

  • Fixed-ϵ\epsilon Theory & Notation (Sec. 3, Prop. 3.4, App. E): We disambiguated the measurable outcome optimum y⋆(x)y^\star(x) from the overall utility optimum (yV⋆,uV⋆)(y_V^\star, u_V^\star), refined Assumps. E.1 and E.4, and proved Lemma E.6 and Prop. 3.4 establishing finite-sample regret bounds and strict utility dominance over the YY-maximizer on [y⋆(x)−ϵ,y⋆(x)][y^\star(x)-\epsilon, y^\star(x)].
  • Target Approximation & Stabilized GPS (Sec. 4.1, Prop. 4.2, App. H.2): We added Prop. 4.2 (Sec. 4.1, p. 6; App. H.2) bounding the total-variation distance between the importance-weighted surrogate qϵq_\epsilon and the uniform ϵ\epsilon-optimal policy πϵ⋆\pi_\epsilon^\star, and characterizing the prior-regularized target induced by stabilized propensity ratios.
  • Observable-Only ϵ\epsilon Calibration & Baseline Grids (Secs. 3, 5, Table 3, App. K.4, Tables 8–10, Fig. 13): We added practical rules for selecting ϵ\epsilon and mm from observable outcomes (Sec. 3) and reported five-seed comparisons against matched-tolerance and full baseline grids in Table 3 (Sec. 5) and App. K.4.
  • Unconfounded GEV Soft-Filter Ablation (Sec. 5, App. K.5, Tables 11–12, Fig. 14): We isolated Eq. (4)'s soft GEV filter against hard kk-NN maxima, empirical quantiles, AWR, DDOM, and propensity-only weights across outcome-noise and logging-shift sweeps.
  • MIMIC-IV Quantitative Pipeline & LOS Distributions (Sec. 5, Table 4, App. K.7, Tables 13–14, Fig. 16): We reported empirical survivor and cohort length-of-stay (LOS) distributions and added a 5-seed patient-grouped evaluation of the Section 4 conditional diffusion + VAE-GPS + GEV pipeline in Table 4 (Sec. 5) and App. K.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.