- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cold-start contextual bandits in a setting where many short-lived bandit instances are encountered repeatedly. The main proposal, BANDHA, is a prior-data fitted network trained on synthetic bandit environments and then deployed as an amortized update rule. The method combines three main ingredients: training with full-feedback teacher targets while conditioning the model only on censored bandit feedback, a permutation-equivariant per-arm recurrent architecture that can operate at different numbers of arms, and an adaptive outer loop that selects action-generating policies used to construct training trajectories. The paper further considers a target-conditioned handoff mechanism, where the warmup policy is trained not only for its own short-horizon regret but also for the continuation performance of a downstream learner.
Empirically, a model trained on (K\in{3,5,10}) is evaluated on both seen and unseen arm counts and outperforms LinUCB and LinTS over the tested synthetic environments. At (T=200), the advantage over LinUCB decreases from 33.6% at (K=3) to 4.2% at (K=20), while the same model also remains competitive on 17 real classification datasets after fixed PCA preprocessing. The paper additionally reports improvements from target-conditioned handoff and several mechanism-oriented analyses and ablations. I find the basic formulation interesting and the manuscript technically careful. My main concern is that the current evaluation does not yet establish that the reported advantage should be attributed to the proposed bandit amortization mechanisms rather than to having a powerful model pretrained on a synthetic prior that is substantially better matched to the evaluation problems than the cold-start baselines.
Critical strengths
-
The training/deployment information boundary is well designed The strongest technical aspect is the separation between the censored history seen by the learner and the full-feedback target available only to the synthetic teacher. This is a clean way to exploit privileged information from a simulator without making the online policy depend on unavailable counterfactual rewards. The paper also provides a useful theoretical account of why this supervision can be substantially more informative than censored feedback. The corresponding controlled experiment is directionally convincing: full-feedback training reduces (T=200) regret by approximately 5 points relative to the censored target across three training runs. Importantly, the authors explicitly acknowledge that this experiment isolates access to full feedback, but does not separately establish the value of soft labels or sharpening.
-
Cross-(K) generalization is a meaningful and practically relevant result The permutation-equivariant per-arm architecture is a natural design for variable action sets, and the empirical behavior is stronger than simply showing architectural compatibility. A single checkpoint is evaluated at several arm counts not encountered during training, including extrapolation to (K=20), without architectural modification. The degradation with increasing (K) is smooth rather than catastrophic. I also view the real-data experiments as useful evidence that the learned update rule is not completely tied to samples from the synthetic generator. The fact that the model is evaluated without fine-tuning is a meaningful positive result, even though I have concerns about how strong a transfer claim can be made from the current baseline set.
Critical weaknesses
- The main evaluation does not sufficiently separate amortization from prior/model-class advantage
This is the main reason I am below the acceptance threshold. On the synthetic experiments, BANDHA is pretrained on the same broad family of formula-tree, MLP, and GP environments from which the evaluation tasks are generated, while the primary comparisons are cold-start LinUCB and LinTS. On nonlinear environments, this is an inherently favorable comparison for a prior-trained nonlinear predictor. The appendix further shows that online NeuralUCB and NeuralTS are unable to learn effectively within (T=200), which does not resolve the issue: these methods are still learning from scratch and therefore do not receive a comparable cross-task training budget. I did not find an empirical comparison against a reasonably competitive amortized or meta-learned baseline that receives the same synthetic training distribution and training budget. This is particularly important because the related-work section positions the contribution relative to meta-bandits and in-context decision models, but the experimental section mostly establishes an advantage over non-amortized algorithms. The real-data experiment does not fully resolve this concern. All datasets are compressed to three dimensions using standardization and PCA, and the principal comparisons remain LinUCB and LinTS. A stronger nonlinear or pretrained transfer baseline would be important to establish whether BANDHA is learning a particularly effective bandit update rule, rather than simply carrying a substantially better inductive bias into the new task. For a top-tier venue, I consider this distinction central rather than cosmetic.
- Several advertised mechanisms are supported by substantially weaker evidence than the headline results
The paper presents full-feedback supervision, adaptive AGP selection, and target-conditioned handoff as major components, but the empirical evidence is uneven. Most notably, the controlled adaptive-AGP experiment is based on a single initialization. The main text itself acknowledges that the observed difference is on the scale of cross-run variation and therefore treats the result only as directional evidence. Given that adaptive AGP selection is presented as one of the three defining mechanisms of BANDHA, I would expect this experiment to be repeated across multiple training seeds before treating the component as established. Similarly, the full-feedback ablation does not isolate the effect of the actual soft/sharpened target used by the final system: the authors explicitly state that the reported experiment does not test soft supervision against sampled one-hot full feedback, nor the effect of sharpening. The handoff experiments are interesting, but presently concentrated on a relatively narrow synthetic (K=10) setting. The result is also somewhat more subtle than the framing suggests: when LinUCB is the eventual downstream learner, a Greedy-targeted prefix is slightly better than the explicitly LinUCB-targeted prefix. This does not invalidate the idea, but it weakens a strong interpretation that accurately targeting the successor is itself the key mechanism. Since the teacher estimates downstream value using tail rollouts, I would also like to see the additional training cost quantified.
Questions for authors
-
Can the authors provide a comparison against at least one amortized/meta-learned baseline that receives the same synthetic training distribution and a comparable offline training budget? This could be a PFN/in-context decision baseline without the proposed state architecture, an appropriate meta-bandit method, or another learned variable-action policy. I would substantially revise my assessment if BANDHA retained a clear advantage under such a comparison.
-
How exactly are standardization and PCA fitted for the 17 real datasets? In particular, are the transformation parameters computed using the complete dataset before constructing the online stream, or are they estimated using only information that would legitimately be available before deployment? The former would introduce transductive information into the real-data evaluation, even though no labels are used by PCA.
-
Can the adaptive-AGP comparison be repeated over multiple independent training seeds? The paper itself states that the controlled single-initialization improvement is comparable to cross-run variation. Since adaptive AGP selection is a headline component, robustness of this effect would materially affect my assessment.
-
For target-conditioned handoff, what is the training-time overhead of computing the tail-rollout target (\Psi_X), and how should the result that Greedy-targeting slightly outperforms LinUCB-targeting before a LinUCB successor be interpreted? In particular, is the intended claim that conditioning on the exact successor is important, or only that the prefix objective should contain some proxy for downstream information value?