acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Posterior Sampling for Fast Adaptation in Multi-task Nonlinear Contextual Bandits

Abstract

We study fast adaptation in multi-task nonlinear contextual bandits, where different tasks share the same reward structure but are characterized by distinct model parameters drawn from a common unknown prior distribution. Given data from past tasks, the goal is to rapidly adapt to a new task and minimize regret using only limited online interactions. Thompson Sampling (TS) is a popular approach for solving contextual bandits, maintaining a posterior over the model parameter that is updated each round using a hand-specified conjugate prior (e.g., Gaussian) and the observed rewards. However, such priors cannot capture the rich cross-task structure in multi-task settings, leading to misspecified posteriors and suboptimal exploration. To enable fast adaptation, we train a diffusion model on data from past tasks to learn a flexible transferable prior distribution over task parameters. In a new bandit task, parameters are estimated via a conditional reverse-diffusion process, where each step combines: (i) an unconditional drift from the diffusion prior, (ii) a likelihood-driven drift from the interaction history, and (iii) a noise term enabling randomized exploration. We instantiate this framework in two ways. DLTS integrates history into the diffusion prior at every reverse step to form a conditional posterior, from which approximate samples are drawn. DPSG applies a single history-guided gradient correction at each diffusion step. Theoretically, we formalize oracle TS and its diffusion counterpart, and show they are equivalent when the diffusion prior matches the true prior. Empirically, our methods are competitive with specialized baselines in linear settings and provide strong fast-adaptation performance in challenging nonlinear bandit environments, requiring substantially fewer online interactions to reach a target performance level.

Primary area:
reinforcement learning
Keywords:
diffusion model, fast adaptation, multi-task contextual bandit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vphnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fast adaptation in multi-task nonlinear contextual bandits. The authors train a diffusion model on model parameters from past tasks to learn a transferable prior, and incorporate the interaction history of a new task for approximate posterior sampling. They propose Diffusion Langevin Thompson Sampling (DLTS), which uses Langevin Monte Carlo updates, and Diffusion Posterior Sampling with Guidance (DPSG), which uses likelihood guidance, together with its practical variant, DPSG with Multi-step Projection (DPSG-MP). The theoretical analysis establishes the equivalence between Oracle Diffusion Thompson Sampling (ODTS) and Oracle Thompson Sampling (OTS) under ideal conditions. Experiments cover linear and nonlinear synthetic environments, high-dimensional neural reward models, and bandit tasks constructed from MovieLens.

Critical strengths

1.The research problem is meaningful, as leveraging historical task priors to reduce online interactions on new tasks has practical value.

2.The paper provides theoretical analysis under ideal conditions and detailed descriptions of two concrete algorithms.

3.The experimental evaluation is broad, covering multiple simulation environments, tasks constructed from real-world data, and ablation studies.

4.The authors state that they provide source code, implementation details, and hyperparameters, which support reproducibility.

Critical weaknesses

1.The benefits of using a diffusion model need more direct empirical support. The authors explain how diffusion models can accommodate nonlinear reward models, but relative to existing learned-prior posterior sampling methods, such as MixTS and DiffTS, it remains unclear which additional design choices account for the key improvements.

2.The main baseline comparisons do not fully separate the sources of improvement. The main neural baselines learn from scratch, whereas the proposed methods leverage historical task knowledge. The learned-prior comparisons provide useful evidence, but comparisons with more closely matched reward models and use of historical data would be more convincing.

3.The explanation that the methods improve exploration lacks targeted experimental validation. Lower cumulative regret may also result from more accurate predictions. Comparing posterior sampling with a single-parameter estimation strategy, and examining the respective contributions of stochasticity and likelihood guidance, would help clarify this explanation.

4.Validation under more challenging practical conditions remains limited. The main simulations assume that historical and new tasks share the same prior distribution, whereas new tasks may differ from historical tasks and historical parameter estimates may be inaccurate in practice. Additional real-world datasets and evaluations under these conditions would help establish the scope of applicability.

5.The computational efficiency claim needs broader support. Results from a single environment and performance threshold demonstrate an advantage in that setting, but do not establish that increased per-round latency can generally be offset by fewer online interactions. Evaluations across different settings, together with per-round runtime measurements, would strengthen this claim.

Questions for authors

1.The learned-prior comparisons provide useful evidence. Could the authors additionally include neural baselines pretrained on the same historical data, and a Gaussian mixture prior baseline using the same nonlinear reward model?

2.Relative to existing diffusion posterior sampling methods, which additional design choices are essential to the performance improvements? Could controlled ablation studies isolate their individual contributions?

3.Do the gains of DPSG-MP primarily come from more accurate predictions or more effective exploration? Could the authors compare posterior sampling with a greedy policy based on a single parameter estimate, and provide comparisons among the original DPSG, multi-step updates without projection, and DPSG-MP?

6 comments

Response to Reviewer vphn

We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.


Q1: Which design choices distinguish the proposed methods from existing diffusion posterior sampling methods, and which are essential to the performance improvements?

A1: We would like to clarify that our main contribution is to design diffusion posterior sampling methods for fast adaptation in nonlinear contextual bandits. Our goal is to learn a prior from historical tasks and use it to learn a new task with fewer online interactions.

Compared with the specialized linear and generalized linear updates in DiffTS [1], our methods incorporate general nonlinear reward observations into the reverse diffusion process. DLTS uses Langevin sampling to approximately sample from a conditional posterior at each diffusion step. DPSG first performs an unconditional reverse update and then applies a likelihood correction based on the current interaction history, following DPS [2]. DPSG-MP further uses multiple likelihood updates with projection, following [3]. Eq. (3.1) provides a unified formulation of these approaches.

Appendices H.6 and H.7 already study the effects of the number of diffusion steps and inner updates. To further identify the contribution of each component, we will conduct the comparisons described in A3.


Q2: Could the authors include neural baselines pretrained on the same historical data and a Gaussian-mixture prior baseline using the same nonlinear reward model?

A2: Appendix H.4 already includes a learned-prior comparison using the same historical information and nonlinear reward model. Learned-Gaussian TS fits a full-covariance Gaussian prior to the same 8,000 historical MLP parameter vectors and uses Langevin sampling for posterior updates. Table 6 shows that DLTS and DPSG-MP achieve lower mean regret in all six settings.

We extend this comparison to pretrained neural baselines and a Gaussian-mixture prior fitted to the historical MLP parameters. These comparisons will further examine the choice of prior while giving the methods access to the same historical information.

ProblemDLTSDPSG-MPNonlinear MixTS (M=2)Nonlinear MixTS (M=4)Nonlinear MixTS (M=8)Nonlinear MixTS (M=12)NeuralTSNeuralUCBPretrained NeuralTSPretrained NeuralUCB
Cross44.33 ± 3.0734.87 ± 0.44160.51 ± 15.8487.91 ± 11.3746.63 ± 5.8753.31 ± 12.7157.09 ± 13.7996.00 ± 28.0737.47 ± 9.5653.86 ± 9.88
Rays51.20 ± 2.9036.31 ± 1.09176.57 ± 19.5589.57 ± 27.1050.24 ± 7.1746.96 ± 8.08117.19 ± 32.63200.13 ± 36.3949.30 ± 10.7778.28 ± 14.44
Corners57.92 ± 2.3538.12 ± 1.27425.68 ± 91.1166.34 ± 6.8260.99 ± 8.2348.88 ± 7.09209.13 ± 53.00373.67 ± 64.8857.31 ± 13.1796.68 ± 31.93

Q3: Do the gains of DPSG-MP primarily come from more accurate predictions or more effective exploration?

A3: The learned prior provides information about the new task before many rewards have been observed. Sampling from the updated parameter distribution then allows the policy to explore different actions. To examine the contribution of sampling, we compare our method with a greedy policy that uses the same learned prior and reward model but maintains a single parameter estimate, updated each round.

We also compare the original DPSG, multi-step updates without projection, and DPSG-MP. The comparison between DPSG and multi-step updates examines the benefit of additional likelihood updates. Comparing the two multi-step methods with the same number of updates isolates the effect of projection.

Cumulative regret at T=200T=200 (mean ± std over 8 trials, 64 tasks per trial).

MethodCrossRaysTrianglesSwirlHCorners
DPSG-MP44.3 ± 2.347.4 ± 3.229.3 ± 0.635.6 ± 1.946.4 ± 7.044.4 ± 2.1
Greedy DPSG-MP148.9 ± 24.064.3 ± 10.383.8 ± 20.088.4 ± 24.0101.6 ± 18.0107.5 ± 30.5
DPSG (single update, no projection)141.1 ± 9.9222.7 ± 8.4211.5 ± 9.8258.6 ± 7.4300.4 ± 18.1416.1 ± 6.1
Multi-step without projection288.6 ± 30.5549.9 ± 32.6614.9 ± 28.1603.2 ± 23.2692.4 ± 46.6941.7 ± 7.7

Q4: How robust are the methods to task-distribution shifts and inaccurate historical parameter estimates, and do the benefits extend to additional real-world datasets?

A4: Our main focus is fast adaptation across related tasks drawn from a shared, unknown prior. Robustness to task-distribution shifts introduces an additional challenge: the historical tasks may no longer represent the new task distribution. This is an important extension of our setting, but it is not the main focus of this paper.

For historical parameter estimation and real-data evaluation, Appendix H.3 already includes the MovieLens experiment. The historical MLP parameters are estimated from observed user ratings. Each MLP starts from the same trained anchor before being fitted to an individual user, which helps reduce differences caused by neural-network parameter permutations.

DPSG-MP achieves mean cumulative regret 28.4028.40, compared with 51.2051.20 for NeuralTS and 39.0039.00 for NeuralUCB. These results demonstrate that our method can benefit from estimated historical parameters on real data.


Q5: Does the time-to-target advantage hold across different environments and performance thresholds, and what is the per-round computational cost?

A5: Our efficiency analysis focuses on the time and interactions needed to reach the same performance level. This is relevant to fast adaptation because the cost of posterior sampling can be offset by a reduction in the required interactions.

Table 1 provides a concrete comparison in the sigmoid-gated Triangles setting. To reach an average regret of 0.50.5, DLTS requires 93 rounds and approximately 257.67 seconds, while DPSG-MP requires 54 rounds and approximately 173.63 seconds. NeuralUCB requires 799 rounds and approximately 1967.71 seconds. All methods are evaluated on the same hardware.

The per-round computation depends on the number of diffusion steps and inner updates. DLTS performs Langevin updates at each diffusion level. DPSG-MP performs a reverse diffusion update followed by projected likelihood updates at each level. Their cost also depends on the reward model and the amount of interaction history used to evaluate the likelihood.

The existing timing results demonstrate this trade-off in the setting reported in Table 1. In our revision, we have stated the setting and threshold explicitly when discussing computational efficiency and distinguish the time-to-target result from per-round runtime.


We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?

References:

[1] Kveton, Branislav, et al. Online posterior sampling with a diffusion prior. NeurIPS, 2024

[2] Chung et al. Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR, 2023.

[3] Xu et al. Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior. ICLR, 2025.

Dear Reviewer vphn,

Thank you again for your constructive feedback. In our response, we compared our methods with pretrained neural baselines and nonlinear Gaussian-mixture baselines. We also included a greedy policy comparison and ablation studies to examine how posterior sampling, multiple likelihood updates, and projection contribute to the performance of DPSG-MP.

As the discussion period approaches its end, we would greatly appreciate it if you could let us know whether our response has addressed your concerns or whether there is anything we could further clarify.

Thank you for your time and consideration.

Reviewer_eAU8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies fast adaptation in multi-task nonlinear contextual bandits, where tasks share a reward-model structure but have task-specific parameters drawn from an unknown common prior. The idea is to learn this prior from historical tasks using a diffusion model and then combine observations from a new task through likelihood-guided reverse diffusion.

Critical strengths
  1. Extension of diffusion priors to nonlinear contextual bandits

Prior work on diffusion-based Thompson sampling has largely focused on multi-armed, linear, and generalized linear bandits. This paper extends the approach to nonlinear reward models, including neural networks, where exact posterior inference is typically intractable.

The two proposed samplers reflect distinct computational trade-offs: DLTS uses iterative Langevin updates, whereas DPSG relies on lightweight likelihood guidance. Together, they broaden the potential applicability of diffusion priors to nonlinear decision-making settings.

The motivation is compelling, as transferring knowledge from historical tasks is particularly valuable when data from a new task are costly or limited.

  1. Promising empirical results across diverse settings

The empirical evaluation is reasonably broad, covering structured priors, nonlinear reward functions, sensitivity analyses, and settings beyond the main low-dimensional synthetic experiments.

The methods achieve substantial regret reductions over neural-bandit baselines in several nonlinear environments. Appendix H.4 provides encouraging comparisons with a learned Gaussian prior, while Appendix H.5 considers directly sampled high-dimensional neural parameters.

The computational analysis is also informative. Although diffusion sampling incurs noticeable overhead, the evidence that improved sample efficiency can compensate for this cost strengthens the practical motivation. Overall, the results provide a credible foundation for further study, though some comparisons would benefit from tighter controls.

Critical weaknesses
  1. Limited theoretical support for the posterior-sampling claims

Theorem 4.4 shows that oracle Thompson sampling and an oracle diffusion sampler are equivalent when the latter represents and samples from the exact posterior. This largely follows from Bayes’ rule and leaves the central challenge unresolved: how closely practical DLTS and DPSG approximate the posterior under nonlinear likelihoods, finite diffusion steps, and imperfect learned priors.

The likelihood approximations in Appendices B.2 and B.3 introduce further errors, yet the paper offers neither bounds nor substantial diagnostics to assess them.

Without a clearer account of these approximation errors, it is difficult to distinguish approximate Thompson sampling from effective but biased randomized optimization, limiting confidence in exploration beyond the evaluated settings.

  1. Experiments do not isolate the benefits of diffusion from historical information

The main comparisons often pit DLTS and DPSG-MP, which use historical task information, against neural-bandit methods trained from scratch. These comparisons therefore do not isolate the contribution of diffusion modeling.

The learned-Gaussian comparison is helpful, but several gaps remain:

  • MixTS is misspecified: Appendix H.4 states that it uses a linear reward model in a nonlinear environment.
  • The experiments do not test more flexible alternatives, such as mixtures with additional components.
  • The main nonlinear simulations derive from two-dimensional task parameters, which may make the transferable structure relatively easy to learn.
  • MovieLens results are mixed: DPSG-MP achieves the lowest mean regret, while DLTS underperforms both NeuralTS and NeuralUCB.

These results support the value of historical knowledge but do not convincingly establish that diffusion is necessary or consistently preferable to simpler learned priors.

Questions for authors
  1. Posterior approximation and exploration. Can the authors characterize the errors introduced by Equation (B.2) and the Tweedie-based likelihood guidance? Under what conditions are the resulting samples close enough to the true posterior for effective Thompson sampling? A simple example comparing sampled distributions with exact posteriors would be informative.

  2. Diffusion versus prior transfer. Can the authors evaluate stronger learned-prior baselines using the same historical information, neural reward model, and approximate sampling procedure? A Gaussian-mixture prior fitted directly to historical neural-network parameters would help isolate the value of diffusion modeling. A nonlinear-reward version of MixTS would also offer a more informative comparison.

  3. Posterior initialization and sampler details. What justifies starting the reverse process from an unconditional Gaussian when the exact posterior path requires (p(\theta_L\mid H_t))? Please also clarify DLTS’s temperature and covariance choices and their implications for its posterior-sampling interpretation.

  4. Robustness to historical parameter estimates. How sensitive are the gains to errors in historical task parameter estimates, particularly when each task has few observations? Since the method relies on these estimates, their quality and alignment may substantially affect the learned diffusion prior.

5 comments

Response to Reviewer eAU8

We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.


Q1: Can the authors characterize the posterior-approximation errors and identify conditions under which DLTS and DPSG support effective Thompson sampling?

A1: Our main contribution is algorithm design for fast adaptation in multi-task nonlinear contextual bandits. Theorem 4.4 establishes the connection with oracle Thompson sampling under exact prior recovery and conditional sampling. In Appendix I of the revised paper, we provide additional analysis of the likelihood approximations and finite-step sampling procedures.

First, Theorem I.3, Eq. (I.4) bounds the likelihood approximation error. Let gt(θ)=p(Ht∣θ)g_t(\theta)=p(H_t\mid\theta), mℓ=E[θ0∣θℓ]m_\ell=\mathbb E[\theta_0\mid\theta_\ell], and Cℓ=Cov⁡(θ0∣θℓ)C_\ell=\operatorname{Cov}(\theta_0\mid\theta_\ell). Under the derivative and moment conditions in Assumption I.1, we obtain ∣p(Ht∣θℓ)−gt(cℓ)∣≤Mt2tr⁡(Cℓ)+Gt∥mℓ−cℓ∥2|p(H_t\mid\theta_\ell)-g_t(c_\ell)|\le \frac{M_t}{2}\operatorname{tr}(C_\ell)+G_t\|m_\ell-c_\ell\|_2.

This separates conditional uncertainty from the error of the parameter estimate. Both terms generally contribute for the rescaled estimate in Eq. (B.2). With an exact prior score, the Tweedie estimate equals mℓm_\ell, so the estimation term vanishes. Eq. (I.5) separately bounds the likelihood-score error using conditional second and third moments, while Eq. (I.6) accounts for errors in the estimated mean and its Jacobian.

Second, we connect these errors to the distributions produced by the samplers. For DLTS, Theorem I.7 applies to reverse levels with positive-definite covariance under the strong convexity, smoothness, and regularity conditions in Assumption I.6. It separates finite Langevin iterations, discretization, temperature mismatch, and errors in the likelihood score, reverse-transition score, and implemented gradient.

For DPSG, Theorem I.9 bounds the distance between the final sampling distribution and the exact posterior under Assumption I.8. The bound includes initialization error and accumulated errors from Gaussian reverse approximation, prior-score estimation, likelihood guidance, and the guidance coefficient. An exact prior score removes the prior-score term while retaining the errors from approximate conditioning. The proofs of these likelihood and sampling results are provided in Appendix I.5.

Finally, we relate the posterior-sampling error to the difference in arm-selection probabilities. For a fixed history and context, let π^t\widehat\pi_t and πtTS\pi_t^{\mathrm{TS}} denote the action distributions of our sampler and oracle Thompson sampling.

Suppose each arm's reward function is LfL_f-Lipschitz in the parameter, and the sampled parameter distribution has W2W_2 distance at most ϵt\epsilon_t from the exact posterior. Let Δ(θ)\Delta(\theta) denote the gap between the largest and second-largest predicted rewards under parameter θ\theta. Then, for any r>0r>0,

TV⁡(π^t,πtTS)≤Pr⁡θ∼p(θ∣Ht)(Δ(θ)≤2Lfr)+ϵt2/r2\operatorname{TV}(\widehat\pi_t,\pi_t^{\mathrm{TS}})\le \Pr_{\theta\sim p(\theta\mid H_t)}(\Delta(\theta)\le 2L_fr)+\epsilon_t^2/r^2.

The proof couples the exact and approximate parameter samples so that their expected squared distance is at most ϵt2\epsilon_t^2. If the parameter distance is at most rr, each arm's predicted reward changes by at most LfrL_fr. Therefore, both samples select the same arm whenever the exact sample's reward gap exceeds 2Lfr2L_fr. The probability of selecting different arms is bounded by the probability of a smaller gap plus the probability that the parameter distance exceeds rr. Markov's inequality bounds the latter by ϵt2/r2\epsilon_t^2/r^2, giving the stated total-variation bound.

When most posterior samples yield a clear reward gap between the best two arms, small sampling errors lead to action probabilities close to those of oracle Thompson sampling.


Q2: Can stronger learned-prior baselines isolate the benefit of diffusion under the same historical information, neural reward model, and approximate sampling procedure?

A2: Appendix H.4 already includes a learned-prior comparison using the same 8000 historical MLP parameter vectors and the same nonlinear reward model. Learned-Gaussian TS fits a full-covariance Gaussian prior to these parameters and uses Langevin sampling for posterior updates. Table 6 shows that DLTS and DPSG-MP achieve lower mean regret in all six settings.

We extend this comparison to Gaussian-mixture priors fitted directly to the historical MLP parameters. Using the same nonlinear reward model also addresses the reward-model difference in the current MixTS comparison.

ProblemDLTSDPSG-MPNonlinear MixTS (M=2)Nonlinear MixTS (M=4)Nonlinear MixTS (M=8)Nonlinear MixTS (M=12)
Cross44.33 ± 3.0734.87 ± 0.44160.51 ± 15.8487.91 ± 11.3746.63 ± 5.8753.31 ± 12.71
Rays51.20 ± 2.9036.31 ± 1.09176.57 ± 19.5589.57 ± 27.1050.24 ± 7.1746.96 ± 8.08
Corners57.92 ± 2.3538.12 ± 1.27425.68 ± 91.1166.34 ± 6.8260.99 ± 8.2348.88 ± 7.09

Regarding task dimensionality, Appendix H.5 already includes an experiment where the 1185-dimensional MLP parameters are sampled directly from a Gaussian mixture with full-rank covariance. This setting does not derive its task parameters from a two-dimensional representation. Table 7 shows that both proposed methods achieve lower mean regret than NeuralTS and NeuralUCB.

The results also distinguish the two proposed methods. On MovieLens, DPSG-MP achieves the lowest mean regret, while DLTS performs less well. We will make these differences clear when discussing the empirical findings.


Q3: What justifies starting the reverse process from an unconditional Gaussian when the exact posterior path requires p(θL∣Ht)p(\theta_L|H_t)?

A3: The forward noising process reduces the dependence of the terminal state on the task parameter. In Appendix I of the revised paper, we quantify the error introduced by approximating the terminal conditional distribution with a standard Gaussian.

Let a=αˉLa=\bar\alpha_L. The forward process gives θL=a θ0+1−a z\theta_L=\sqrt a\,\theta_0+\sqrt{1-a}\,z, where θ0∼p(θ0∣Ht)\theta_0\sim p(\theta_0\mid H_t) and z∼N(0,I)z\sim\mathcal N(0,I) are independent. Coupling θL\theta_L with the same Gaussian variable zz yields

W2(p(θL∣Ht),N(0,I))≤a E[∥θ0∥22∣Ht]+d(1−1−a)2W_2(p(\theta_L\mid H_t),\mathcal N(0,I))\le\sqrt{a\,\mathbb E[\|\theta_0\|_2^2\mid H_t]+d(1-\sqrt{1-a})^2}.

This argument is provided in the proof of Theorem I.9 in Appendix I.5. For a finite posterior second moment, the bound converges to zero as a→0a\to0. At a finite noise level, it explicitly characterizes the initialization error through the noise schedule, parameter dimension, and posterior second moment.

DPSG and DPSG-MP use this Gaussian initialization. Theorem I.9, Eq. (I.16) and Theorem I.10, Eq. (I.18) propagate the initialization error through the reverse process and include its contribution to the final posterior-sampling error.

DLTS uses a Gaussian reference (μL+1,ΣL+1)=(0,I)(\mu_{L+1},\Sigma_{L+1})=(0,I) at its initial level. Algorithm 1 initializes the Langevin iterate at μL+1=0\mu_{L+1}=0 and applies history-conditioned Langevin updates at ℓ=L+1\ell=L+1.

For our synthetic experiments, we use L=100L=100 and a linear noise schedule, yielding αˉL≈0.048\bar\alpha_L\approx0.048. The resulting finite-noise initialization error is retained in the bounds above.


Q4: How are DLTS’s temperature and covariance chosen, and how do these choices affect its posterior-sampling interpretation?

A4: There are two different covariance quantities in DLTS. The reverse-transition covariance is determined by the DDPM noise schedule: Σℓ=1−αˉℓ−11−αˉℓβℓI\Sigma_\ell=\frac{1-\bar\alpha_{\ell-1}}{1-\bar\alpha_\ell}\beta_\ell I.It appears in the Gaussian prior term of the conditional target. The noise added in each Langevin update has covariance 2ηℓζℓ−1I2\eta_\ell\zeta_\ell^{-1}I, where ηℓ\eta_\ell is the step size.

Under the convention in Eq. (B.5), ζℓ\zeta_\ell acts as an inverse temperature. If UℓU_\ell is the negative log of the approximate conditional target, the corresponding continuous Langevin process has stationary density proportional to exp⁡(−ζℓUℓ)\exp(-\zeta_\ell U_\ell). Thus, ζℓ=1\zeta_\ell=1 corresponds to the stated target; larger values make it more concentrated, while smaller values make it more spread out. Finite step sizes and update counts introduce the additional sampling error discussed in A1.

Table 3 reports the update counts, step sizes, and noise scales for each setting. In our implementation, each Langevin update adds the noise s2ηℓ ξks\sqrt{2\eta_\ell}\,\xi_k, where ss is the noise scale. The noise scales we use, s∈(0.01,0.05,0.1)s\in(0.01,0.05,0.1). These hyperparameters are selected for each setting by grid search according to the cumulative regret.

Q5: How sensitive are the gains to errors in historical task parameter estimates, particularly when each task has few observations?

A5: Our MovieLens experiment already learns the diffusion prior from estimated task parameters. Each historical user's MLP is fitted to observed ratings. Appendix H.3 initializes these MLPs from the same trained anchor before fitting them to individual tasks. This gives the task models a common starting point and helps reduce differences caused by neural-network parameter permutations.

Errors in the historical estimates affect the learned prior, which in turn affects sampling on a new task. During online learning, our methods incorporate the new task's observed rewards at each reverse step to update the parameter samples. The MovieLens results provide evidence that this approach can benefit from estimated historical parameters: DPSG-MP achieves the lowest mean regret in this setting.


We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?

References:

[1] Chung et al. Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR, 2023.

Reviewer_WBkzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies transfer across nonlinear contextual-bandit tasks whose parameters follow a shared unknown prior. It learns a diffusion prior from parameters obtained from previous tasks, then approximates posterior sampling on a new task using either Langevin updates within reverse-diffusion steps (DLTS) or likelihood guidance (DPSG), with a projected multi-step variant. The theoretical result establishes equivalence between exact Thompson sampling and an oracle diffusion procedure that samples the exact conditional diffusion-path distribution. I focus on the theoretical contribution.

Critical strengths

The framework provides a concrete way to combine a learned task prior with nonlinear reward likelihoods. Distinguishing per-level Langevin sampling from direct likelihood guidance is useful for understanding the algorithmic choices. The oracle result correctly checks consistency with Bayesian posterior sampling when exact conditional inference is available.

Critical weaknesses
  1. The main theorem assumes the inference problem has already been solved. Definition 4.2 explicitly gives ODTS the exact joint posterior over the diffusion path, including its conditional terminal distribution and every conditional reverse transition. Theorem 4.4 then integrates out the auxiliary variables. In Appendix C, the substantive equality is simply

    ∫p(θ0:L∣Ht) dθ1:L=p(Ht∣θ0)p0(θ0)p(Ht).\int p(\theta_{0:L}\mid H_t)\,d\theta_{1:L} =\frac{p(H_t\mid\theta_0)p_0(\theta_0)}{p(H_t)}.

    This is a valid identity for any auxiliary-variable representation of the same prior. It does not establish an additional diffusion-specific or nonlinear-bandit guarantee. The paper's stated obstacle is accurate inference without closed-form updates (Section 1, lines 63–66), but the theorem assumes exact inference. Diffusion priors and per-level conditioning for contextual bandits already appear in Kveton et al. (2024); the present theoretical analysis does not establish what is gained by replacing their specialized updates with the proposed nonlinear procedures.

  2. The practical likelihood approximations introduce an uncontrolled error even with a perfect prior. In Appendix B.2, lines 830–833, the forward noising equation is used to motivate treating θℓ/αˉℓ\theta_\ell/\sqrt{\bar\alpha_\ell} as a conditional mean. For θ0∼N(0,1)\theta_0\sim N(0,1) and θℓ=a θ0+1−a z\theta_\ell=\sqrt a\,\theta_0+\sqrt{1-a}\,z, however,

    θ0∣θℓ∼N(a θℓ,1−a).\theta_0\mid\theta_\ell\sim N(\sqrt a\,\theta_\ell,1-a).

    With an observation y=θ0+ϵy=\theta_0+\epsilon, ϵ∼N(0,σ2)\epsilon\sim N(0,\sigma^2), the exact likelihood is p(y∣θℓ)=N(y;a θℓ,σ2+1−a)p(y\mid\theta_\ell)=N(y;\sqrt a\,\theta_\ell,\sigma^2+1-a). Eq. (B.2)'s proxy has a different mean and variance. Using the correct Tweedie mean, as in DPSG, still leaves a plug-in likelihood that omits the conditional variance. Thus exact unconditional scores alone do not eliminate the relevant inference error. No result controls this bias, finite Langevin sampling, or the multi-step projection's effect on posterior uncertainty and exploration. Extending familiar samplers to a nonlinear likelihood is an implementable proposal, but the only theorem does not substantiate that extension's behavior.

References

The Gaussian example identifies the approximation bias; it does not contradict the oracle identity, which assumes exact inference.

Questions for authors
  1. Which guarantee applies to DLTS, DPSG, or DPSG-MP with their actual likelihood surrogates? In particular, what can be concluded when the prior score is exact but conditional inference uses Eqs. (B.2) or (B.7)?
  2. How should the conditional-mean argument below Eq. (B.2) be corrected? Is there an assumption under which its approximation error is controlled in the limited-data regime emphasized by the paper?
4 comments

Response to Reviewer WBkz

We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.


Q1: Main contributions of the paper and the role of Theorem 4.4

A1: We would like to clarify that our paper is primarily an algorithmic and empirical study of fast adaptation in multi-task nonlinear contextual bandits. Its main contributions are the design of practical algorithms and their experimental validation. The theoretical result provides supporting analysis.

To the best of our knowledge, this is the first work to develop and empirically evaluate diffusion posterior sampling for multi-task contextual bandits with general neural reward models, including high-dimensional task parameters and real-data applications. The central challenge is to combine a prior learned from historical tasks with new nonlinear reward observations when closed-form conditional reverse updates are unavailable. We address this through per-level Langevin sampling in DLTS, likelihood-guided reverse updates in DPSG, and multi-step projected updates in DPSG-MP. Our experiments evaluate these methods across nonlinear reward models, directly sampled 1185-dimensional neural-network parameters, and MovieLens tasks. Comparisons with learned-prior baselines and component ablations further examine the benefits of the proposed designs.

Theorem 4.4 serves as a supporting oracle consistency result. With the true prior and exact conditional sampling, it establishes that the diffusion sampler produces the same posterior and arm-selection distributions as oracle Thompson sampling. Its purpose is to identify the ideal sampling target and connect our framework to Thompson sampling. To address the reviewer's concerns about the practical algorithms, we provide additional analysis of likelihood approximation and finite-step sampling errors under explicit assumptions in Appendix I of the revised paper. We summarize these results in our responses to Q2–Q4.

We therefore hope the reviewer to reconsider the evaluation in light of the paper's main contributions to algorithm design and empirical performance for fast adaptation.


Q2: Clarification about likelihood approximation errors

A2: In Appendix I.1 of the revised paper, we provide bounds for both likelihood-value and likelihood-score approximation errors. Following the likelihood-approximation perspective in DPS [1], we analyze the difference between the exact intermediate likelihood, which averages over θ0\theta_0 given θℓ\theta_\ell, and the likelihood evaluated at a single estimate.

Let gt(θ)=p(Ht∣θ)g_t(\theta)=p(H_t\mid\theta), mℓ=E[θ0∣θℓ]m_\ell=\mathbb E[\theta_0\mid\theta_\ell], and Cℓ=Cov⁡(θ0∣θℓ)C_\ell=\operatorname{Cov}(\theta_0\mid\theta_\ell). Under Assumption I.1, with ∥∇gt∥2≤Gt\lVert \nabla g_t \rVert_2 \le G_t and ∥∇2gt∥op≤Mt\lVert \nabla^2 g_t \rVert_{\mathrm{op}} \le M_t, Theorem I.3, Eq. (I.4) gives ∣p(Ht∣θℓ)−gt(cℓ)∣≤Mt2tr⁡(Cℓ)+Gt∥mℓ−cℓ∥2|p(H_t\mid\theta_\ell)-g_t(c_\ell)|\le \frac{M_t}{2}\operatorname{tr}(C_\ell)+G_t\|m_\ell-c_\ell\|_2.

This follows from a second-order Taylor expansion around mℓm_\ell: the linear term has zero conditional expectation, and the Hessian bound controls the remainder. The Lipschitz bound then accounts for the difference between cℓc_\ell and mℓm_\ell.

The two terms separate conditional uncertainty from estimation error. For the rescaled estimate in Eq. (B.2), both terms generally contribute. With an exact prior score, the Tweedie estimate equals mℓm_\ell, so the estimation term vanishes. The covariance term quantifies the remaining approximation error identified in the reviewer's Gaussian example, where Cℓ=1−αˉℓC_\ell=1-\bar\alpha_\ell.

For likelihood guidance, Theorem I.3, Eq. (I.5) separately bounds the error in ∇θℓlog⁡gt(mℓ)\nabla_{\theta_\ell}\log g_t(m_\ell) using conditional second and third moments and likelihood derivatives. This bound is derived directly through conditional differentiation and Taylor expansion. Eq. (I.6) further accounts for errors in the estimated conditional mean and its Jacobian when using a learned score. The proofs are provided in Appendix I.5, and we explain their connection to the sampling bounds in our response to Q3.

Q3: Which guarantee applies to DLTS, DPSG, or DPSG-MP with their actual likelihood surrogates? In particular, what can be concluded when the prior score is exact but conditional inference uses Eqs. (B.2) or (B.7)?

A3: We provide sampling-error bounds for DLTS, DPSG, and DPSG-MP in Appendix I.2–I.4 of the revised paper. These bounds explicitly account for the likelihood surrogates and finite sampling steps. An exact prior score removes the prior-score estimation error, while the bounds retain the errors from approximate conditioning and sampling.

For DLTS, Theorem I.7, Eq. (I.13) bounds the distance between the distribution after KK Langevin updates and the exact conditional reverse posterior. Consider a level with Σℓ≻0\Sigma_\ell\succ0. Let UℓU_\ell be the potential defined by the approximate likelihood and Gaussian reverse model, and let πℓ,τ∝e−Uℓ/τ\pi_{\ell,\tau}\propto e^{-U_\ell/\tau}, where τ=ζℓ−1\tau=\zeta_\ell^{-1}. Under Assumption I.6, UℓU_\ell is mUm_U-strongly convex and MUM_U-smooth. For 0<η≤1/MU0<\eta\le1/M_U, we obtain W2(νK,Pℓ)≤(1−mUη)KW2(ν0,πℓ,τ)+3MU2mUτdη+dmU∣τ−1∣+ϵℓ,like+ϵℓ,rev+ϵℓ,driftmUW_2(\nu_K,P_\ell)\le(1-m_U\eta)^K W_2(\nu_0,\pi_{\ell,\tau})+\frac{3M_U}{2m_U}\sqrt{\tau d\eta}+\sqrt{\frac{d}{m_U}}|\sqrt{\tau}-1|+\frac{\epsilon_{\ell,\mathrm{like}}+\epsilon_{\ell,\mathrm{rev}}+\epsilon_{\ell,\mathrm{drift}}}{m_U}

Here, νK\nu_K is the distribution after KK updates, and PℓP_\ell is the exact conditional reverse posterior. The first two approximation terms measure the likelihood-score and reverse-transition-score errors in L2(Pℓ)L^2(P_\ell). The third measures the difference between the implemented gradient and ∇Uℓ\nabla U_\ell along the sampling iterates; it vanishes when the full gradient is used.

The bound separates finite iterations, discretization, temperature mismatch, and approximation errors. The analysis in our response to Q2 bounds the plug-in component of the likelihood-score error. Lemma I.11 provides conditions for propagating the per-level errors to the final output distribution.

For DPSG, Theorem I.9, Eq. (I.16) bounds the distance between its final output distribution and the posterior ptp_t, under the smoothness and curvature conditions in Assumption I.8. The bound includes Gaussian initialization error and accumulated per-step errors. Eq. (I.15) separates each step's error into Gaussian reverse approximation, prior-score estimation, likelihood guidance, and the difference between the implemented guidance coefficient and the posterior-score coefficient. With an exact prior score, the prior-score term is zero, while the likelihood-guidance error associated with Eq. (B.7) remains explicitly included.

For DPSG-MP, Theorem I.10, Eq. (I.17) compares its projected update with a single DPSG update at the same input. With the fixed guidance used in Algorithm 3, the additional error is bounded by (Λℓ+∣Λℓ−ηℓ∣)∥u^ℓ(z)∥2(\Lambda_\ell+|\Lambda_\ell-\eta_\ell|)\lVert\widehat u_\ell(z)\rVert_2, where Λℓ\Lambda_\ell is the cumulative inner-loop step size. Eq. (I.18) combines this contribution with the DPSG errors to bound the distance between the final DPSG-MP distribution and ptp_t.

The complete proofs are provided in Appendix I.5.


Q4: How should the conditional-mean argument below Eq. (B.2) be corrected? Is there an assumption under which its approximation error is controlled in the limited-data regime emphasized by the paper?

A4: The quantity θℓ/αˉℓ\theta_\ell/\sqrt{\bar\alpha_\ell} is a rescaled noisy estimate of θ0\theta_0. The forward noising equation gives E[θℓ/αˉℓ∣θ0]=θ0\mathbb E[\theta_\ell/\sqrt{\bar\alpha_\ell}\mid\theta_0]=\theta_0. As stated in Lemma I.2, Eq. (I.1) of the revised paper, the conditional mean given θℓ\theta_\ell is instead mℓ=E[θ0∣θℓ]=θℓ+(1−αˉℓ)∇θℓlog⁡pℓ(θℓ)αˉℓm_\ell=\mathbb E[\theta_0\mid\theta_\ell]=\frac{\theta_\ell+(1-\bar\alpha_\ell)\nabla_{\theta_\ell}\log p_\ell(\theta_\ell)}{\sqrt{\bar\alpha_\ell}}.

We revise the explanation below Eq. (B.2) to make this distinction explicit. For the reviewer's Gaussian example, Tweedie's formula gives mℓ=αˉℓθℓm_\ell=\sqrt{\bar\alpha_\ell}\theta_\ell, consistent with the exact conditional distribution.

For approximation-error control, Lemma I.5, Eq. (I.9) provides sufficient conditions for a fixed finite history. Let a=αˉℓa=\bar\alpha_\ell. Under Assumption I.1, suppose tr⁡(Cℓ)=O(1−a)\operatorname{tr}(C_\ell)=O(1-a) and ∥∇log⁡pℓ(θℓ)∥2=O(1)\lVert\nabla\log p_\ell(\theta_\ell)\rVert_2=O(1) as a↑1a\uparrow1, uniformly over the domain under consideration. Eq. (I.3) then gives ∥mℓ−θℓ/a∥2=O(1−a)\lVert m_\ell-\theta_\ell/\sqrt a\rVert_2=O(1-a). Substituting into the likelihood-value bound in Theorem I.3 yields ∣p(Ht∣θℓ)−gt(θℓ/a)∣=O(1−a)|p(H_t\mid\theta_\ell)-g_t(\theta_\ell/\sqrt a)|=O(1-a). Thus, under these conditions, the approximation error converges to zero as the diffusion noise vanishes.

The dependence on history length and observation noise is made explicit in Lemma I.4, Eq. (I.8). For Gaussian observations with variance σ2\sigma^2 and reward functions satisfying ∥∇θf∥2≤B\lVert\nabla_\theta f\rVert_2\le B and ∥∇θ2f∥op≤D\lVert\nabla_\theta^2f\rVert_{\mathrm{op}}\le D, the unnormalized Gaussian likelihood satisfies Gt=B2λnt/e,Mt=2λntB2+D2λnt/eG_t=B\sqrt{2\lambda n_t/e}, M_t=2\lambda n_tB^2+D\sqrt{2\lambda n_t/e}, where nt=∣Ht∣n_t=|H_t| and λ=1/(2σ2)\lambda=1/(2\sigma^2). History length enters through these constants, while Lemma I.5 specifies the conditions on conditional uncertainty and the prior score needed for the approximation to be accurate.


We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?

References:

[1] Chung et al. Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR, 2023.

Dear Reviewer WBkz,

Thank you again for your constructive feedback. Our paper is primarily an algorithmic and empirical study of fast adaptation in multi-task nonlinear contextual bandits. Its main contributions are the design and experimental evaluation of practical algorithms. In response to your questions, we expanded the supporting theoretical analysis in Appendix I of the revised paper.

As the discussion period draws to a close, we would appreciate it if you could reconsider your assessment in light of these algorithmic and empirical contributions. Please let us know if any concerns remain that we could further clarify.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.