acceptodds
Under review as a conference paper at ICLR 2027

From Bids to Modes: Policy Refinement with Latent Decision Modes for Auto-Bidding

Abstract

Auto-bidding plays a central role in modern computational advertising, yet learning improved policies solely from offline trajectories remains challenging. Existing methods typically perform policy improvement directly in the original continuous bid space, which is only sparsely covered by training data. Moreover, bid-level optimization obscures distinctions among different decision tendencies, making it difficult to switch among them dynamically as market and budget conditions evolve. To address this, we introduce **PRISM** (**P**olicy **R**efinement with latent dec**IS**ion **M**odes), a framework for auto-bidding that optimizes policies over a compact mode-level decision space. To construct this space, PRISM uses return-tilted mode discovery to learn a partition that captures outcome-relevant distinctions between modes. Building on the discovered modes, it selects the best-evaluated mode from contextually supported candidates and realizes it through a continuous-bid policy trained with mode-referenced weighting. Experiments on AuctionNet show that PRISM consistently outperforms competitive auto-bidding baselines, while D4RL evaluations highlight its potential as a general policy-refinement framework for offline data containing diverse decision behaviors.

Primary area:
reinforcement learning
Keywords:
Auto-Bidding, Offline Reinforcement Learning, Latent Decision Modes

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MsVmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the optimization of auto-bidding policies within the context of offline reinforcement learning. The authors argue that optimizing strategies directly within a continuous bidding space is problematic: it is highly susceptible to issues stemming from insufficient coverage in historical data, and it tends to blend distinct decision tendencies, hindering the ability to adapt flexibly to varying market conditions.

To address these challenges, the authors propose PRISM. This framework first learns a set of latent decision modes based on the long-term returns of historical decisions. It then selects an appropriate mode based on the current context and employs a conditional policy to generate specific bidding actions. The methodology centers on mode discovery via "Return-Tilted Information Decomposition," combined with a policy optimization process that incorporates return-weighting and data-support-based filtering.

Experimental evaluations were conducted using the AuctionNet and D4RL datasets. PRISM demonstrated strong performance across various budget settings in AuctionNet, and ablation studies confirmed that both the mode discovery mechanism and the dynamic mode selection process are the core contributions of the approach.

Critical strengths
  1. Instead of optimizing the bidding strategy directly within a continuous bid space, the authors first learn latent decision modes and then dynamically select a mode based on the current market conditions and budget status. The design of both mode discovery and mode selection is comprehensive.
  2. The experimental results are robust, particularly the validation of dynamic mode selection, with strong performance demonstrated on AuctionNet, further supported by ablation studies and comparisons against fixed-mode baselines.
Critical weaknesses
  1. There may be a mismatch in Mode History between the training and inference phases. For instance, during training, the historical modes are derived from the actual actions present in offline data, whereas during inference, the model relies on the modes it has selected itself. Such discrepancies could potentially affect subsequent mode evaluation and selection; however, the paper has not yet explicitly analyzed whether the errors arising from this process accumulate throughout the decision-making sequence.
  2. The computational overhead of the method lacks thorough analysis. PRISM comprises several modules (Mode Encoder, Mode Evaluator, Support Gate…), with both the Evaluator and the Support Gate utilizing Transformers. Given the requirements for decision-making efficiency in automated bidding, an analysis of inference latency would be helpful.
Questions for authors
  1. Could the authors empirically evaluate the potential impact of the inconsistency in Mode History between the training and inference phases? Furthermore, as the decision-making process unfolds, does this discrepancy affect the accuracy of subsequent Mode Evaluation and Mode Selection steps?
  2. Could the authors also provide a comparison of the proposed method against existing baselines in terms of inference latency?
4 comments
AuthorsOpenReview ↗

Response Overview

We thank the reviewer for the careful assessment and constructive suggestions. We organize our response around the following two questions:

  • Q1: Mode History Across Training and Inference
  • Q2: Inference Latency and Computational Overhead

To address these questions, we provide additional mode-history diagnostics, a deployment ablation, and inference-latency measurements against existing baselines.

AuthorsOpenReview ↗

Q1: Mode History Across Training and Inference

We agree that the mode history is constructed differently in the two phases: during training, historical modes are derived from the logged actions in the offline data, whereas at inference the history stores the modes selected by the policy itself. We have conducted a dedicated analysis to evaluate whether this discrepancy affects subsequent mode evaluation and mode selection, and whether its effect accumulates over the decision sequence.

Experimental Setup. We replay the same offline trajectories under two mode-history conventions, keeping the logged states and actions fixed. Under hrefh_{\mathrm{ref}}, historical modes are assigned to the logged actions by the encoder, following the training convention. Under hinfh_{\mathrm{inf}}, the history is updated with the modes selected by the gate and evaluator, following the inference convention. These selections update only the mode history and do not alter the replayed trajectory. This setup isolates the effect of the history convention on subsequent evaluation and selection, without introducing differences in visited states. All model parameters remain frozen throughout the analysis.

Experimental results.

  1. Relatively small changes in candidate sets and evaluation rankings. The two history conventions produce different selections at some steps, but their candidate sets and evaluation rankings remain largely aligned: the candidate-set Jaccard is ≈0.87\approx 0.87, and the Kendall ranking correlation is ≈0.90\approx 0.90, compared with expected values of ≈0.36\approx 0.36 and 00 for independent random candidate sets and rankings, respectively.

  2. Selection divergences are not permanently sustained. After the first divergence, 99.4%99.4\% of trajectories select the same mode as the reference again at a later step. This discrepancy does not accumulate into a persistent drift as the decision-making process unfolds.

  3. The self-selected-mode history improves deployment performance. In the closed-loop ablation, removing this history channel significantly reduces performance (Score change −5.64-5.64, 95% CI [−10.14,−1.47][-10.14,-1.47]). This demonstrates that the history formed from the policy's own selected modes contributes positively to inference-time performance.

Conclusion. The train-inference difference in mode-history construction leads to relatively small changes in candidate sets and evaluation rankings, and selection divergences are not permanently sustained. The self-selected-mode history is beneficial for deployment. We will add this analysis to the appendix.

AuthorsOpenReview ↗

Q2: Inference Latency and Computational Overhead

Thank you for requesting an inference-latency comparison. We have conducted additional measurements and report the mean and 95th-percentile (P95) latency below:

MethodMean (ms)P95 (ms)
USCB0.260.31
CQL1.572.61
IQL1.592.64
BCQ1.922.96
CDT3.064.15
DT3.624.68
PRISM5.476.44
DT-score6.368.41
GAVE6.958.06
DiffBid57.0062.20
GAS84.8994.73

PRISM does not use large language models (LLMs) or large-parameter architectures. Compared with lightweight actor-based baselines, PRISM incurs some additional latency, but its mean and P95 latency remain in the single-digit millisecond range. It is faster than DT-score and GAVE on both metrics, and substantially faster than DiffBid and GAS. These measurements show that the additional mode-selection modules introduce only moderate overhead, while avoiding the much higher inference cost associated with iterative generative methods. We will include this comparison in the appendix.

Dear Reviewer MsVm,

Thank you once again for your valuable comments on our submission. As the discussion period draws to a close, we would be grateful to hear whether our responses have adequately addressed your concerns. Should any questions remain, we would be happy to provide further clarification. Your feedback would be especially helpful to us at this stage.

We sincerely look forward to your reply and thank you again for your time and consideration.

Reviewer_8g4GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PRISM, a framework for auto-bidding in online advertising that optimizes policies over a mode-level decision space (bidding modes) rather than directly over bid values. The motivation is that the behavioral bidding form alone is insufficient and does not carry decision-level semantics. The method first discovers the modes and then performs policy optimization in a batch, multi-pass EM procedure with all components frozen. Evidence is provided through experiments on AuctionNet.

Critical strengths

The paper addresses an important problem in RL-based bidding for online advertising: existing methods typically average over distinct bidding regimes into a single compromise bid. The paper provides extensive empirical analysis in the AuctionNet setting, with 10 seeds and ablation studies. It is is well organized, with clear and consistent evidence.

Critical weaknesses

The evidence for the regime-switching statistic is thin, and the switching modes are not clearly characterized. There is no direct analysis of what the discovered modes are, which is a significant gap. The protocol for selecting 8 clusters is not specified, and there is no validation split. There is no deployment with online or incremental updates. Insufficient evidence of robustness in real-world environments. The paper’s motivation is production non-stationarity, but the main real-world sources of it (traffic shifts, dynamic competition, poorly discriminating and miscalibrated pCVR predictions) are neither tested nor discussed.

Questions for authors

Clarifications: Does the method assume that v_i is true, unbiased, and well-ranked? Which components of the method break if v_i is biased in a context-dependent way? Is there any evidence of the method’s robustness to dynamic competition, traffic shift, or imperfect value-model predictions? There are currently no experiments that test shifted test-time conditions (dynamic competitors, traffic drift, value-model drift).

Additional evidence (would affect my assessment): Can the authors provide results with perturbed values at evaluation time: multiplicative bias, drift over time steps, and rank noise?

9 comments
AuthorsOpenReview ↗

Response Overview

We thank the reviewer for the thoughtful comments and suggestions for additional evidence. We organize our response around the following three questions:

  • Q1: Characterization of the Discovered Modes and Switching Behavior
  • Q2: Selection of the Number of Modes and Validation
  • Q3: Robustness to Value-Model Errors and Test-Time Shifts

To address these questions, we provide additional mode and switching analyses, validation results for the number of modes, and perturbation experiments covering value-model errors, dynamic competition, and traffic shifts.

AuthorsOpenReview ↗

Q1: Characterization of the Discovered Modes and Switching Behavior

We agree that the original submission did not directly characterize the discovered modes, and we have added a full characterization, which we summarize here.

1. Mode Analysis

What each mode is. Importantly, the modes carry no preset semantics: they are not hand-designed but are discovered by the learning process, and the roles below are post-hoc characterizations of what each discovered mode actually does. The table reports each mode's offline bid multiplier α\alpha, per-step spend, win rate, conversions, and CPA, together with its conditional offline α\alpha under stage / budget / market bins:

Moderoleα\alphaspendwinconvCPAstage α\alpha (early/mid/late)budget α\alpha (low/mid/high)market α\alpha (low/mid/high)
m0high-win-rate sprint9.940.01920.12910.826.5217.2 / 9.9 / 9.28.3 / 12.3 / 10.88.9 / 9.2 / 16.6
m1low-bid defense7.080.01190.0757.543.755.9 / 6.6 / 7.45.6 / 9.5 / 8.07.3 / 7.1 / 5.8
m2budget/market-responsive bump9.780.03090.1008.996.6612.8 / 9.4 / 9.27.4 / 13.1 / 11.08.7 / 8.9 / 13.4
m3efficient closing7.950.00590.0424.034.867.2 / 7.3 / 8.97.9 / 9.5 / 7.88.2 / 8.1 / 7.2
m4aggressive scaling14.130.04850.1178.5612.1021.0 / 12.3 / 10.47.3 / 9.8 / 18.110.7 / 11.3 / 20.3
m5high-intensity spender10.860.02830.0585.519.6713.4 / 9.6 / 9.07.5 / 8.0 / 11.19.8 / 9.7 / 12.9
m6mild transition8.620.01190.0242.067.599.0 / 8.0 / 8.45.4 / 7.1 / 8.68.2 / 8.1 / 9.0
m7low-spend baseline8.260.00580.0171.568.078.4 / 7.8 / 8.45.4 / 8.7 / 8.37.8 / 7.9 / 8.5

(Binning: stage = first/middle/last third of the steps; budget = remaining-budget ratio <1/3<1/3, 1/31/3--2/32/3, ≥2/3\ge 2/3; market = terciles of the mean leastWinning-cost over the recent 3 steps.)

The discovered modes span different levels of spending intensity and cost efficiency. Modes m7 and m6 form the low-spend end (per-step spend 0.0058 and 0.0119), while m5 supports higher-intensity spending (0.0283) and m4 represents aggressive scaling, with the highest α\alpha (14.13) and spend (0.0485). Modes m1 and m3 emphasize efficiency: m1 combines the lowest α\alpha (7.08) with the lowest CPA (3.75), while m3 combines low CPA (4.86) with a higher conditional α\alpha in the late stage (8.9 versus 7.2 early). Mode m0 is distinguished by the highest win rate (0.129) and conversions (10.82), whereas m2 shows pronounced budget and market responses, with conditional α\alpha reaching 13.1 in the middle-budget bin and 13.4 in the high-market-cost bin.

These profiles provide the empirical basis for the post-hoc role descriptions, which are further illustrated by the switching patterns below.

AuthorsOpenReview ↗

Q1: Characterization of the Discovered Modes and Switching Behavior

2. Mode-Switching Analysis

Distinct context-response profiles, not static bid levels. The conditional-α\alpha columns show that each mode responds to context differently: m4 is the steepest, firing aggressively early and when budget or market conditions are favorable (early 21.0, budget-high 18.1, market-high 20.3); m3 shows the opposite temporal pattern, raising α\alpha in the late stage for closing (7.2 →\rightarrow 8.9); m0 spikes on a favorable market (market-high 16.6); and m7 is the flattest, staying stable across all contexts (5.4--8.7).

Switching is state-driven, not random jitter. Appendix C.2 (Figure 9) reports the aggregate switching statistics: on 48 dense-market evaluation trajectories, the median per-agent switching rate is 0.15, i.e., roughly one switch every 6--7 active steps. Switching propensity rises when the campaign is in the mid-stage, further behind its budget plan, closer to its CPA constraint, with a higher recent win rate, and under larger market-price jumps (more detailed data will be provided in the appendix). The modes therefore act as a state-driven pacing controller.

Switching direction is structured. Switches to a higher-α\alpha mode account for 58% of all transitions, rising to 65% in the mid-stage (early 55%, late 52%): the mid-stage is dominated by "powering up," while the late stage balances scaling up and down for closing and adjustment.

Modes form a clear stickiness hierarchy. m7 (steady low-spend base) has a leave probability of only 0.05; m5/m4/m3 are around 0.2; m0/m2 are markedly less sticky (0.38/0.47). Switching thus concentrates at a few state-transition points, with m7 as the stable base and m0/m2 as short-lived, context-triggered modes.

Representative switching chains. The following observed sequences illustrate how the discovered modes are combined during deployment:

  • Standard ramp: m7 →\rightarrow m4/m5 →\rightarrow m3, moving from low-spend bidding to higher-intensity spending and then to an efficiency-oriented closing mode.
  • Context-dependent adjustment: m4 →\rightarrow m2 →\rightarrow m5 →\rightarrow m4, moving between aggressive scaling and the budget/market-responsive mode m2, followed by higher-intensity spending through m5 before returning to m4.
  • Defense--transition--sprint: m1 →\rightarrow m6 →\rightarrow m0, moving from low-bid, efficiency-oriented bidding through a moderate transition mode to the mode characterized by the highest win rate and conversion volume.

We will include the complete analysis in the appendix of the revised manuscript.

Reviewer_wkXFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduced PRISM, an offline auto-bidding method that improves policies over a small set of learned discrete “decision modes” rather than directly in the continuous bid space. It first clusters decisions into modes using return-weighted evidence and trains a contextual evaluator of each mode’s outcome. It then trains a mode-conditioned bid decoder with advantage-style weights. At inference, it picks the highest-valued mode among those well supported by the data. Empirically, PRISM reports the best scores across all AuctionNet budget settings and competitive results on D4RL.

Critical strengths
  • The problem is realistically relevant since budget pacing does involve switching between conservative and aggressive regimes.
  • The reported empirical results suggest consistent gains from the proposed method. In particular, improvements hold across all budget levels and in the sparse variant, with the largest gain under the tightest budget. I also find the empirical analysis quite comprehensive. Component ablations, fixed-mode comparisons, mode-utilization analysis, and hyperparameter studies provide evidence that contextual mode selection contributes to performance.
Critical weaknesses
  • The mode evaluator is not clearly aligned with policy improvement. It predicts Ebeh[exp⁡(G/λ)∣c,m]E_{\mathrm{beh}}[\exp(G/\lambda) | c,m] under the logged behavior’s continuation, while deployment uses the learned decoder and switches modes. Consequently, the highest-evaluated mode need not yield the highest expected return under the deployed policy. Exponentiating returns can also favor high-variance outcomes. The paper does not sufficiently justify or test this mismatch.
  • It also seems that several D4RL baseline scores appear substantially below those reported in the original papers, particularly TD3+BC on walker2d-mr and DT on hopper-me. Could the authors clarify whether these baselines were rerun or taken from prior work, and explain any differences in experimental settings? AuctionNet baseline results also lack uncertainty estimates, making the robustness of the reported gains difficult to assess.
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response Overview

We thank the reviewer for the careful reading and constructive comments. We organize our response around the following two questions:

  • Q1: Alignment Between Mode Evaluation and Policy Improvement
  • Q2: Baseline Evaluation

To address these questions, we provide additional theoretical analysis and experimental evidence, and clarify the baseline sources, evaluation settings, and uncertainty estimates.

AuthorsOpenReview ↗

Q1: Alignment Between Mode Evaluation and Policy Improvement

We thank the reviewer for highlighting the need to clarify the connection between the mode evaluator and deployed policy optimization. We address two aspects: whether the evaluator and decoder share a common optimization basis, and how this optimization connects to improvement in the deployed policy's expected return.

1. The evaluator and decoder share a common optimization basis

The connection between the evaluator and MRPI centers on the conditional target action distribution of Eq. (24) in Appendix B, ptilt(a∣c,m)=pbeh(a∣c,m)yˉ(c,a)μq(c,m).(1)p_{\text{tilt}}(a\mid c,m)=\frac{p_{\text{beh}}(a\mid c,m)\bar{y}(c,a)}{\mu_q(c,m)}. \quad (1) Both components are tied to this single target: MRPI fits the decoder to ptiltp_{\text{tilt}}, while the evaluator's log-score log⁡μq(c,m)\log\mu_q(c,m) is the optimal value of the same within-mode KL-regularized log-evidence objective, attained exactly at ptiltp_{\text{tilt}}. We derive each in turn.

a. MRPI fits the target action distribution.

For the idealized derivation, we omit weight clipping and use a consistent mode definition. Substituting Eq. (1) into the conditional weighted MRPI loss gives: −Ebeh[ytρ(c)+ϵlog⁡πξ(at∣s,em)∣c,m]-\mathbb{E}_ {\text{beh}}[\frac{y_ t}{\rho(c)+\epsilon}\log\pi_ \xi(a_ t\mid s,e_ m)\mid c,m] =−1ρ(c)+ϵ∫pbeh(a∣c,m)yˉ(c,a)log⁡πξ(a∣s,em)da=-\frac{1}{\rho(c)+\epsilon}\int p_ {\text{beh}}(a\mid c,m)\bar{y}(c,a)\log\pi_ \xi(a\mid s,e_ m)da =−μq(c,m)ρ(c)+ϵ∫ptilt(a∣c,m)log⁡πξ(a∣s,em)da=-\frac{\mu_ q(c,m)}{\rho(c)+\epsilon}\int p_ {\text{tilt}}(a\mid c,m) \log\pi_ \xi(a\mid s,e_ m)da =μq(c,m)ρ(c)+ϵDKL(ptilt(⋅∣c,m)∥πξ(⋅∣s,em))+C(c,m).(2)=\frac{\mu_ q(c,m)}{\rho(c)+\epsilon}D_ {\text{KL}}(p_ {\text{tilt}}(\cdot\mid c,m)\Vert \pi_ \xi(\cdot\mid s,e_ m))+C(c,m). \quad (2) Here, C(c,m)C(c,m) is independent of the decoder parameters ξ\xi. Thus, minimizing the weighted MRPI loss fits the decoder to ptilt(a∣c,m)p_ {\text{tilt}}(a\mid c,m). When the conditional target is jointly representable by the decoder, πξ⋆(a∣s,em)=ptilt(a∣c,m).(3)\pi_ {\xi^\star}(a\mid s,e_ m)=p_ {\text{tilt}}(a\mid c,m). \quad (3)

b. The evaluator's log-score corresponds to the optimal value of the same within-mode objective.

Consider optimization over action distributions within a fixed context and mode: max⁡π{Ea∼π[log⁡yˉ(c,a)]−DKL(π∥pbeh(⋅∣c,m))}.(4)\max_ \pi\{\mathbb{E}_ {a\sim\pi}[\log\bar{y}(c,a)]-D_ {\text{KL}}(\pi\Vert p_ {\text{beh}}(\cdot\mid c,m))\}. \quad (4) Taking logarithms in Eq. (1) and substituting into DKL(π∥ptilt)D_ {\text{KL}}(\pi\Vert p_ {\text{tilt}}) yields: Ea∼π[log⁡yˉ(c,a)]−DKL(π∥pbeh(⋅∣c,m))\mathbb{E}_ {a\sim\pi}[\log\bar{y}(c,a)]-D_ {\text{KL}}(\pi\Vert p_ {\text{beh}}(\cdot\mid c,m)) =log⁡μq(c,m)−DKL(π∥ptilt(⋅∣c,m)).(5)=\log\mu_ q(c,m)-D_ {\text{KL}}(\pi\Vert p_ {\text{tilt}}(\cdot\mid c,m)). \quad (5) Since KL divergence is nonnegative, the objective is bounded above by log⁡μq(c,m)\log\mu_ q(c,m). This bound is attained at π=ptilt(⋅∣c,m)\pi=p_ {\text{tilt}}(\cdot\mid c,m).

Hence, the optimal action distribution of this behavior-referenced, KL-regularized log-evidence objective is ptilt(a∣c,m)p_ {\text{tilt}}(a\mid c,m), and its optimal value is log⁡μq(c,m)\log\mu_ q(c,m). Ideal MRPI weighted maximum likelihood fits the former, while the evaluator gives the latter. Mode scoring and action realization therefore share a common optimization basis rather than being independent designs.

AuthorsOpenReview ↗

Q1: Alignment Between Mode Evaluation and Policy Improvement

2. Connecting the optimization to deployed expected-return improvement

The evaluator's population target is behavior-continuation evidence, μq(c,m)=Ebeh[eGt/λ∣ct=c,mt=m],\mu_ q(c,m)=\mathbb{E}_ {\text{beh}}[e^{G_ t/\lambda}\mid c_ t=c,m_ t=m], while deployment realizes actions through πξ\pi_ \xi and reselects modes at subsequent steps. We connect the optimization above to deployed expected return through an explicit improvement lower bound.

a. Continuation changes do not invalidate policy improvement.

Behavior-referenced policy improvement constructs a new policy using the reference policy's values; it does not require the new policy to be greedy with respect to its own value function. Related constructions appear in advantage-weighted regression and one-step offline RL (Peng et al., 2019; Brandfonbrener et al., 2021).

Let π\pi denote the complete deployed PRISM policy, and let JJ denote the expected discounted return associated with the training return-to-go GtG_ t. Define Qtbeh(c,a)=E[Gt∣ct=c, execute a, then follow πbeh],Q_ t^{\text{beh}}(c,a)=\mathbb{E}[G_ t\mid c_ t=c,\text{ execute }a,\text{ then follow }\pi_ {\text{beh}}], and Vtbeh(c)=Ea∼πbeh(⋅∣c)[Qtbeh(c,a)].V_ t^{\text{beh}}(c)=\mathbb{E}_ {a\sim\pi_ {\text{beh}}(\cdot\mid c)}[Q_ t^{\text{beh}}(c,a)]. Using a common sufficient state or history representation, the finite-horizon performance-difference identity gives:

[\bar{G}_ \xi(c_ t,m_ t^\star)-V_ t^{\text{beh}}(c_ t)]. \quad (6)$$ Here, $d_ t^\pi$ is the deployed context distribution, and $$\bar{G}_ \xi(c,m)=\mathbb{E}_ {a\sim\pi_ \xi(\cdot\mid s,e_ m)}[Q_ t^{\text{beh}}(c,a)].$$ The advantages are evaluated using the behavior continuation but accumulated over contexts visited by the complete deployed policy. Subsequent mode switching is included in $d_ t^\pi$; deployment is not assumed to continue with $\pi_ {\text{beh}}$. Nonnegative accumulated behavior-referenced advantage is therefore sufficient for improvement, even when behavior and deployment induce different mode-value rankings. **b. The corresponding lower bound for PRISM.** Suppressing the time index in the auxiliary quantities, define $$d(c,a)=\lambda\log\bar{y}(c,a)-Q_ t^{\text{beh}}(c,a),\qquad \eta(c)=\sup_ a d(c,a)-\inf_ a d(c,a).$$ Here, $d(c,a)$ is the exponential score's offset from the mean return, and $\eta(c)$ measures its variation across actions. Let the decoder's realization loss relative to the ideal tilted distribution be bounded by $\delta(c)$: $$\bar{G}_ \xi(c,m^\star) \ge \mathbb{E}_ {a\sim p_ {\text{tilt}}(\cdot\mid c,m^\star)} [Q_ t^{\text{beh}}(c,a)]-\delta(c). \quad (7)$$ Taking logarithms in Eq. (1) and expectations under $p_ {\text{tilt}}(\cdot\mid c,m^\star)$, and applying Jensen's inequality gives $$\lambda\mathbb{E}_ {p_ {\text{tilt}}}[\log\bar{y}] \ge \lambda\log\mu_ q(c,m^\star),$$ $$\lambda\mathbb{E}_ {p_ {\text{beh}}}[\log\bar{y}] \le \lambda\log\rho_ q(c).$$ Substituting $\lambda\log\bar{y}(c,a)=Q_ t^{\text{beh}}(c,a)+d(c,a)$, and using $\eta(c)$ to bound the additional average offset under the tilted action distribution, yields $$\mathbb{E}_ {p_ {\text{tilt}}(\cdot\mid c,m^\star)} [Q_ t^{\text{beh}}(c,a)]-V_ t^{\text{beh}}(c) \ge \lambda\log \frac{\mu_ q(c,m^\star)}{\rho_ q(c)} -\eta(c). \quad (8)$$ Combining Eqs. (7)–(8) and substituting into Eq. (6) gives: $$J(\pi)-J(\pi_ {\text{beh}}) \ge \sum_ t\gamma^t \mathbb{E}_ {d_ t^\pi} [\underbrace{ \lambda\log \frac{\mu_ q(c_ t,m_ t^\star)}{\rho_ q(c_ t)} }_ {\text{evidence margin}}- \underbrace{\eta(c_ t)}_ {\text{risk discrepancy}}- \underbrace{\delta(c_ t)}_ {\text{decoder realization loss}} ]. \quad (9)$$ The bound uses the population quantities $\mu_ q,\rho_ q$ and the actual selected mode $m_ t^\star$. Improvement holds whenever the accumulated evidence margin exceeds the risk discrepancy and decoder realization loss.

Comments on the paper

3 comments

Dear Authors and Reviewers,

Thank you for supporting ICLR'27!

We are now entering the discussion phase. Please make the most of this time to exchange your views. I encourage each Reviewer, if you have not already done so, to read the other reviews and the Authors' responses. Please respond to the Authors and actively participate in the discussion to help clarify any remaining issues.

Thank you,

AC

Dear reviewers,

The authors have submitted their response, and the discussion will be closed in a few days. Please read the response and the other reviews, and update your review, including ratings, if your assessment has changed.

If any concerns remain, please explain whether and why the response has addressed them. Where your assessment differs from that of other reviewers, please discuss the specific points with reference to the paper and the response.

Thank you for your time and contributions.

AC

Dear Reviewers,

Thank you for your efforts for this paper.

As the discussion period draws to a close, I'd especially appreciate hearing from Reviewer 8g4G, Reviewer MsVm and Reviewer wkXF, who have yet to join the discussion. Please post a brief reply to the authors indicating which concerns their response has addressed and which, if any, remain. If all your concerns have been resolved, a short confirmation is sufficient.

Thank you for your time and input.

Thanks & Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.