- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies fast adaptation in multi-task nonlinear contextual bandits. The authors train a diffusion model on model parameters from past tasks to learn a transferable prior, and incorporate the interaction history of a new task for approximate posterior sampling. They propose Diffusion Langevin Thompson Sampling (DLTS), which uses Langevin Monte Carlo updates, and Diffusion Posterior Sampling with Guidance (DPSG), which uses likelihood guidance, together with its practical variant, DPSG with Multi-step Projection (DPSG-MP). The theoretical analysis establishes the equivalence between Oracle Diffusion Thompson Sampling (ODTS) and Oracle Thompson Sampling (OTS) under ideal conditions. Experiments cover linear and nonlinear synthetic environments, high-dimensional neural reward models, and bandit tasks constructed from MovieLens.
Critical strengths
1.The research problem is meaningful, as leveraging historical task priors to reduce online interactions on new tasks has practical value.
2.The paper provides theoretical analysis under ideal conditions and detailed descriptions of two concrete algorithms.
3.The experimental evaluation is broad, covering multiple simulation environments, tasks constructed from real-world data, and ablation studies.
4.The authors state that they provide source code, implementation details, and hyperparameters, which support reproducibility.
Critical weaknesses
1.The benefits of using a diffusion model need more direct empirical support. The authors explain how diffusion models can accommodate nonlinear reward models, but relative to existing learned-prior posterior sampling methods, such as MixTS and DiffTS, it remains unclear which additional design choices account for the key improvements.
2.The main baseline comparisons do not fully separate the sources of improvement. The main neural baselines learn from scratch, whereas the proposed methods leverage historical task knowledge. The learned-prior comparisons provide useful evidence, but comparisons with more closely matched reward models and use of historical data would be more convincing.
3.The explanation that the methods improve exploration lacks targeted experimental validation. Lower cumulative regret may also result from more accurate predictions. Comparing posterior sampling with a single-parameter estimation strategy, and examining the respective contributions of stochasticity and likelihood guidance, would help clarify this explanation.
4.Validation under more challenging practical conditions remains limited. The main simulations assume that historical and new tasks share the same prior distribution, whereas new tasks may differ from historical tasks and historical parameter estimates may be inaccurate in practice. Additional real-world datasets and evaluations under these conditions would help establish the scope of applicability.
5.The computational efficiency claim needs broader support. Results from a single environment and performance threshold demonstrate an advantage in that setting, but do not establish that increased per-round latency can generally be offset by fewer online interactions. Evaluations across different settings, together with per-round runtime measurements, would strengthen this claim.
Questions for authors
1.The learned-prior comparisons provide useful evidence. Could the authors additionally include neural baselines pretrained on the same historical data, and a Gaussian mixture prior baseline using the same nonlinear reward model?
2.Relative to existing diffusion posterior sampling methods, which additional design choices are essential to the performance improvements? Could controlled ablation studies isolate their individual contributions?
3.Do the gains of DPSG-MP primarily come from more accurate predictions or more effective exploration? Could the authors compare posterior sampling with a greedy policy based on a single parameter estimate, and provide comparisons among the original DPSG, multi-step updates without projection, and DPSG-MP?
6 comments
Q4: How robust are the methods to task-distribution shifts and inaccurate historical parameter estimates, and do the benefits extend to additional real-world datasets?
A4: Our main focus is fast adaptation across related tasks drawn from a shared, unknown prior. Robustness to task-distribution shifts introduces an additional challenge: the historical tasks may no longer represent the new task distribution. This is an important extension of our setting, but it is not the main focus of this paper.
For historical parameter estimation and real-data evaluation, Appendix H.3 already includes the MovieLens experiment. The historical MLP parameters are estimated from observed user ratings. Each MLP starts from the same trained anchor before being fitted to an individual user, which helps reduce differences caused by neural-network parameter permutations.
DPSG-MP achieves mean cumulative regret , compared with for NeuralTS and for NeuralUCB. These results demonstrate that our method can benefit from estimated historical parameters on real data.
Q5: Does the time-to-target advantage hold across different environments and performance thresholds, and what is the per-round computational cost?
A5: Our efficiency analysis focuses on the time and interactions needed to reach the same performance level. This is relevant to fast adaptation because the cost of posterior sampling can be offset by a reduction in the required interactions.
Table 1 provides a concrete comparison in the sigmoid-gated Triangles setting. To reach an average regret of , DLTS requires 93 rounds and approximately 257.67 seconds, while DPSG-MP requires 54 rounds and approximately 173.63 seconds. NeuralUCB requires 799 rounds and approximately 1967.71 seconds. All methods are evaluated on the same hardware.
The per-round computation depends on the number of diffusion steps and inner updates. DLTS performs Langevin updates at each diffusion level. DPSG-MP performs a reverse diffusion update followed by projected likelihood updates at each level. Their cost also depends on the reward model and the amount of interaction history used to evaluate the likelihood.
The existing timing results demonstrate this trade-off in the setting reported in Table 1. In our revision, we have stated the setting and threshold explicitly when discussing computational efficiency and distinguish the time-to-target result from per-round runtime.
We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?
References:
[1] Kveton, Branislav, et al. Online posterior sampling with a diffusion prior. NeurIPS, 2024
[2] Chung et al. Diffusion Posterior Sampling for General Noisy Inverse Problems. ICLR, 2023.
[3] Xu et al. Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior. ICLR, 2025.
Dear Reviewer vphn,
Thank you again for your constructive feedback. In our response, we compared our methods with pretrained neural baselines and nonlinear Gaussian-mixture baselines. We also included a greedy policy comparison and ablation studies to examine how posterior sampling, multiple likelihood updates, and projection contribute to the performance of DPSG-MP.
As the discussion period approaches its end, we would greatly appreciate it if you could let us know whether our response has addressed your concerns or whether there is anything we could further clarify.
Thank you for your time and consideration.
Response to Reviewer vphn
We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.
Q1: Which design choices distinguish the proposed methods from existing diffusion posterior sampling methods, and which are essential to the performance improvements?
A1: We would like to clarify that our main contribution is to design diffusion posterior sampling methods for fast adaptation in nonlinear contextual bandits. Our goal is to learn a prior from historical tasks and use it to learn a new task with fewer online interactions.
Compared with the specialized linear and generalized linear updates in DiffTS [1], our methods incorporate general nonlinear reward observations into the reverse diffusion process. DLTS uses Langevin sampling to approximately sample from a conditional posterior at each diffusion step. DPSG first performs an unconditional reverse update and then applies a likelihood correction based on the current interaction history, following DPS [2]. DPSG-MP further uses multiple likelihood updates with projection, following [3]. Eq. (3.1) provides a unified formulation of these approaches.
Appendices H.6 and H.7 already study the effects of the number of diffusion steps and inner updates. To further identify the contribution of each component, we will conduct the comparisons described in A3.
Q2: Could the authors include neural baselines pretrained on the same historical data and a Gaussian-mixture prior baseline using the same nonlinear reward model?
A2: Appendix H.4 already includes a learned-prior comparison using the same historical information and nonlinear reward model. Learned-Gaussian TS fits a full-covariance Gaussian prior to the same 8,000 historical MLP parameter vectors and uses Langevin sampling for posterior updates. Table 6 shows that DLTS and DPSG-MP achieve lower mean regret in all six settings.
We extend this comparison to pretrained neural baselines and a Gaussian-mixture prior fitted to the historical MLP parameters. These comparisons will further examine the choice of prior while giving the methods access to the same historical information.
Q3: Do the gains of DPSG-MP primarily come from more accurate predictions or more effective exploration?
A3: The learned prior provides information about the new task before many rewards have been observed. Sampling from the updated parameter distribution then allows the policy to explore different actions. To examine the contribution of sampling, we compare our method with a greedy policy that uses the same learned prior and reward model but maintains a single parameter estimate, updated each round.
We also compare the original DPSG, multi-step updates without projection, and DPSG-MP. The comparison between DPSG and multi-step updates examines the benefit of additional likelihood updates. Comparing the two multi-step methods with the same number of updates isolates the effect of projection.
Cumulative regret at T=200 (mean ± std over 8 trials, 64 tasks per trial).