acceptodds
Withdrawn from ICLR 2027

Learned Warmup and Target-Conditioned Handoff for Contextual Bandits

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many contextual bandit deployments are streams of short-lived instances: per title, campaign, participant, or item pool. Each instance starts with little local history, so standard online algorithms pay the exploration cost repeatedly, even when aggregate traffic is large. We study prior-data fitted networks (PFNs) as a way to amortize this cold-start phase across related bandit tasks, deploying a model through standard forward passes conditioned on an instance's observed history. We instantiate this idea as Bandha (Bandit Amortization with Neural Decision Handoff and Adaptation), using three mechanisms. First, the sampled PFN environment supplies full-feedback targets for all arms during training, while the model operates only on the censored history available online. Second, a permutation-equivariant per-arm state lets the learned update rule apply across different numbers of arms. Third, an outer Thompson-sampling controller selects the action-generating policy (AGP) used to produce informative training histories. Together, these mechanisms train the model from teacher-augmented synthetic episodes while its deployed state updates only from censored histories, rather than carrying deployment state between instances. We evaluate the construction on fixed-dimensional contextual bandits. Empirically, the learned update rule transfers across both arm counts and datasets. A single model trained on , where is the number of arms, is evaluated at K\in {3,5,7,10,13,15,20\} using cumulative pseudo-regret. At horizon 200, the model reduces this regret by 4.2% to 33.6% relative to linear upper confidence bound (LinUCB), and improves over linear Thompson sampling (LinTS). At horizon 1000, it reduces regret by 21.6% to 54% relative to LinUCB in paired evaluations on the same instances at . On 17 real-world classification datasets, the same model remains useful zero-shot after fixed preprocessing. When another model, such as LinUCB, must take over, a warmup prefix optimized for its downstream continuation (target-conditioned handoff) improves on an untargeted prefix, while unconditioned or misaligned prefixes show where amortized exploration can hurt downstream learners.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
contextual bandits, prior-fitted networks, amortized inference, exploration, warm start, meta-learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Bs2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cold-start contextual bandits in a setting where many short-lived bandit instances are encountered repeatedly. The main proposal, BANDHA, is a prior-data fitted network trained on synthetic bandit environments and then deployed as an amortized update rule. The method combines three main ingredients: training with full-feedback teacher targets while conditioning the model only on censored bandit feedback, a permutation-equivariant per-arm recurrent architecture that can operate at different numbers of arms, and an adaptive outer loop that selects action-generating policies used to construct training trajectories. The paper further considers a target-conditioned handoff mechanism, where the warmup policy is trained not only for its own short-horizon regret but also for the continuation performance of a downstream learner.

Empirically, a model trained on (K\in{3,5,10}) is evaluated on both seen and unseen arm counts and outperforms LinUCB and LinTS over the tested synthetic environments. At (T=200), the advantage over LinUCB decreases from 33.6% at (K=3) to 4.2% at (K=20), while the same model also remains competitive on 17 real classification datasets after fixed PCA preprocessing. The paper additionally reports improvements from target-conditioned handoff and several mechanism-oriented analyses and ablations. I find the basic formulation interesting and the manuscript technically careful. My main concern is that the current evaluation does not yet establish that the reported advantage should be attributed to the proposed bandit amortization mechanisms rather than to having a powerful model pretrained on a synthetic prior that is substantially better matched to the evaluation problems than the cold-start baselines.

Critical strengths
  1. The training/deployment information boundary is well designed The strongest technical aspect is the separation between the censored history seen by the learner and the full-feedback target available only to the synthetic teacher. This is a clean way to exploit privileged information from a simulator without making the online policy depend on unavailable counterfactual rewards. The paper also provides a useful theoretical account of why this supervision can be substantially more informative than censored feedback. The corresponding controlled experiment is directionally convincing: full-feedback training reduces (T=200) regret by approximately 5 points relative to the censored target across three training runs. Importantly, the authors explicitly acknowledge that this experiment isolates access to full feedback, but does not separately establish the value of soft labels or sharpening.

  2. Cross-(K) generalization is a meaningful and practically relevant result The permutation-equivariant per-arm architecture is a natural design for variable action sets, and the empirical behavior is stronger than simply showing architectural compatibility. A single checkpoint is evaluated at several arm counts not encountered during training, including extrapolation to (K=20), without architectural modification. The degradation with increasing (K) is smooth rather than catastrophic. I also view the real-data experiments as useful evidence that the learned update rule is not completely tied to samples from the synthetic generator. The fact that the model is evaluated without fine-tuning is a meaningful positive result, even though I have concerns about how strong a transfer claim can be made from the current baseline set.

Critical weaknesses
  1. The main evaluation does not sufficiently separate amortization from prior/model-class advantage

This is the main reason I am below the acceptance threshold. On the synthetic experiments, BANDHA is pretrained on the same broad family of formula-tree, MLP, and GP environments from which the evaluation tasks are generated, while the primary comparisons are cold-start LinUCB and LinTS. On nonlinear environments, this is an inherently favorable comparison for a prior-trained nonlinear predictor. The appendix further shows that online NeuralUCB and NeuralTS are unable to learn effectively within (T=200), which does not resolve the issue: these methods are still learning from scratch and therefore do not receive a comparable cross-task training budget. I did not find an empirical comparison against a reasonably competitive amortized or meta-learned baseline that receives the same synthetic training distribution and training budget. This is particularly important because the related-work section positions the contribution relative to meta-bandits and in-context decision models, but the experimental section mostly establishes an advantage over non-amortized algorithms. The real-data experiment does not fully resolve this concern. All datasets are compressed to three dimensions using standardization and PCA, and the principal comparisons remain LinUCB and LinTS. A stronger nonlinear or pretrained transfer baseline would be important to establish whether BANDHA is learning a particularly effective bandit update rule, rather than simply carrying a substantially better inductive bias into the new task. For a top-tier venue, I consider this distinction central rather than cosmetic.

  1. Several advertised mechanisms are supported by substantially weaker evidence than the headline results

The paper presents full-feedback supervision, adaptive AGP selection, and target-conditioned handoff as major components, but the empirical evidence is uneven. Most notably, the controlled adaptive-AGP experiment is based on a single initialization. The main text itself acknowledges that the observed difference is on the scale of cross-run variation and therefore treats the result only as directional evidence. Given that adaptive AGP selection is presented as one of the three defining mechanisms of BANDHA, I would expect this experiment to be repeated across multiple training seeds before treating the component as established. Similarly, the full-feedback ablation does not isolate the effect of the actual soft/sharpened target used by the final system: the authors explicitly state that the reported experiment does not test soft supervision against sampled one-hot full feedback, nor the effect of sharpening. The handoff experiments are interesting, but presently concentrated on a relatively narrow synthetic (K=10) setting. The result is also somewhat more subtle than the framing suggests: when LinUCB is the eventual downstream learner, a Greedy-targeted prefix is slightly better than the explicitly LinUCB-targeted prefix. This does not invalidate the idea, but it weakens a strong interpretation that accurately targeting the successor is itself the key mechanism. Since the teacher estimates downstream value using tail rollouts, I would also like to see the additional training cost quantified.

Questions for authors
  1. Can the authors provide a comparison against at least one amortized/meta-learned baseline that receives the same synthetic training distribution and a comparable offline training budget? This could be a PFN/in-context decision baseline without the proposed state architecture, an appropriate meta-bandit method, or another learned variable-action policy. I would substantially revise my assessment if BANDHA retained a clear advantage under such a comparison.

  2. How exactly are standardization and PCA fitted for the 17 real datasets? In particular, are the transformation parameters computed using the complete dataset before constructing the online stream, or are they estimated using only information that would legitimately be available before deployment? The former would introduce transductive information into the real-data evaluation, even though no labels are used by PCA.

  3. Can the adaptive-AGP comparison be repeated over multiple independent training seeds? The paper itself states that the controlled single-initialization improvement is comparable to cross-run variation. Since adaptive AGP selection is a headline component, robustness of this effect would materially affect my assessment.

  4. For target-conditioned handoff, what is the training-time overhead of computing the tail-rollout target (\Psi_X), and how should the result that Greedy-targeting slightly outperforms LinUCB-targeting before a LinUCB successor be interpreted? In particular, is the intended claim that conditioning on the exact successor is important, or only that the prefix objective should contain some proxy for downstream information value?

Reviewer_CacvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes BANDHA, which uses prior-data fitted networks (PFNs) trained on synthetic bandit tasks to reduce the cold-start exploration cost of contextual bandits. To improve cold-start decision-making, the method combines full-feedback supervision, a permutation-equivariant per-arm state architecture, and an adaptive action-generating policy (AGP). The authors also propose a target-conditioned handoff mechanism, which optimizes the interaction data collected during warmup to improve performance after a downstream algorithm takes over.

Experiments show that the model outperforms LinUCB across seven arm-count settings and exhibits zero-shot transfer on 17 real-world classification datasets. These results support the paper's core claims on cold-start performance, transfer across arm counts, and warmup handoff. However, the real-data experiments rely on fixed preprocessing, the controlled experiments on the adaptive AGP are limited in scale, and the long-horizon experiments establish an advantage only over a finite horizon. The method's generalization to real-world settings and its long-term performance guarantees therefore require further investigation.

Critical strengths

S1. Full-feedback supervision under censored feedback. During training, the sampled environment supplies soft full-feedback targets for all arms, while the model conditions only on the censored interaction prefix available online, observing only the selected arm's binary reward, exactly as at deployment. Theoretically, the conditional population minimizer of this soft cross-entropy objective is the posterior mean reward vector. The construction also avoids the information lost to censoring and the AGP-induced target tilt. This is the paper's core novelty relative to prior-data fitted networks (PFNs) and in-context decision models, and the matched full-versus-censored supervision ablation confirms its effectiveness.

S2. Permutation-equivariant per-arm state table. Each arm maintains its own state, updated by a shared gated recurrent unit, and only the pulled arm's state is updated. A Transformer arm mixer without positional embeddings then scores all arms jointly in a set-based manner. This gives the model exact label equivariance and K-independent parameter shapes, so a single checkpoint applies directly across arm counts and beyond its training horizon. This matches the cold-start setting with rotating item pools. Empirically, the model consistently outperforms the linear baselines on held-out arm counts and in zero-shot real-data transfer.

Critical weaknesses

W1. Weak baselines. The main comparators, LinUCB and LinTS, are misspecified for the nonlinear synthetic prior, and NeuralUCB/NeuralTS fail to learn within the short horizon. The paper omits comparisons with meta-bandit methods, e.g., Meta-TS, in-context decision models, e.g., DPT, and a Bayes-optimal reference under the same prior. Since BANDHA exploits a prior shared with the evaluation tasks while the baselines do not, the results show only the benefit of prior knowledge, not the specific contribution of the proposed mechanisms.

W2. Limited external validity and insufficient mechanistic evidence. The synthetic evaluation is in-distribution. The real-data experiments use low-dimensional projections of classification datasets, which neither reflect the motivating deployment scenarios nor test robustness to prior misspecification. Apart from full-feedback supervision, the ablations of the other components are limited or confounded, and the theoretical bound is not empirically examined. The gain from target-conditioned handoff is marginal and partly contradicts the claim that prefixes should be tailored to the downstream learner.

W3. Poor clarity and readability. The abstract is overloaded with experimental details, and the background and related work are insufficiently developed, which leaves the paper's positioning unclear. The relationship between the proposed mechanisms and the stated contributions is ambiguous. Dense acronyms, under-explained equations, and stilted phrasing further hinder comprehension.

Questions for authors

Q1. Baselines. How does BANDHA compare with meta-bandit methods (e.g., Meta-TS), in-context decision models (e.g., DPT), and a Bayes-optimal reference under the same prior? Can the authors show that the gains stem from the proposed mechanisms rather than from access to the prior alone?

Q2. Robustness and mechanisms. How does BANDHA perform under prior misspecification or on real bandit interaction logs with higher-dimensional contexts? Can the authors provide multi-seed ablations that isolate each component?

Q3. Clarity. Can the authors clarify how the proposed mechanisms map onto the stated contributions? Would they consider streamlining the abstract and expanding the background and related work to better position the paper?

Reviewer_UriLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies learned warmup policies for contextual bandits and their handoff to a specified downstream learner. The method combines synthetic full-feedback supervision, censored interaction histories, shared per-arm recurrent states, and a teacher that trades off immediate regret against the successor’s expected future regret. The theoretical analysis provides a prediction-to-regret decomposition, a conditional long-horizon crossover statement, and results concerning feedback information, permutation equivariance, and successor dependence. My assessment focuses on the conceptual and theoretical contributions.

Critical strengths

Optimizing a warmup policy for the learner that subsequently uses its observations is a reasonable objective. The distinction between privileged training targets and the feedback available during interaction is clearly specified. The appendices also explicitly acknowledge several limitations of the theoretical statements.

Critical weaknesses

My main concern is that the analysis does not establish a substantive theoretical advance in exploration or handoff design. The central results formalize familiar observations, while the quantities needed to justify the proposed method remain assumed or uncontrolled.

  1. The successor-conditioned objective largely recovers a standard rollout formulation. With exact quantities and α=1\alpha=1, the score following Eq. (5) becomes

    −Δ(a)+ΨX(a∣s,x)=RJ(X∣s)−[Δ(a)+ErRJ(X∣UX(s,x,a,r))].-\Delta(a)+\Psi_X(a\mid s,x) = R_J(X\mid s) - \left[ \Delta(a)+ \mathbb E_r R_J(X\mid U_X(s,x,a,r)) \right].

    The first term is independent of the action. Thus, action selection minimizes immediate regret plus the continuation regret of a prescribed successor, which is a standard one-step rollout objective. Proposition 6 is the corresponding Bellman telescoping identity. These observations explain the objective, but do not establish a new exploration principle or a guarantee for the learned multi-step prefix.

    Proposition 7 provides only a weak separation: the two constructed successors deliberately ignore observations from different arms, even though either initial action reveals the unknown environment. This demonstrates that unrestricted successors can have incompatible optimal prefixes, but gives little insight into when conditioning on a natural bandit learner is beneficial. The missing step is a quantitative connection between finite-rollout teacher estimation, prefix learning, and an improvement over the successor alone.

  2. The main guarantees leave the statistical and long-horizon difficulties unresolved. Theorem 1 follows from the standard greedy prediction-error inequality, a decomposition into target bias and distribution shift, and Pinsker’s inequality. It assumes the KL-fit condition and introduces a deployment-shift term, without controlling either through the training sample size, model class, or proposed training procedure.

    Moreover, RH(πB)R_H(\pi_B) evaluates posterior-mean greedy actions counterfactually on histories generated by π^\hat\pi; it is neither the regret of an independently executed comparator nor that of the Bayes-optimal exploratory policy. Consequently, even perfect prediction and zero target bias do not resolve the exploration cost of the histories collected by the learned policy. The result is useful as an accounting decomposition, but does not establish a substantive bandit-learning guarantee for BANDHA.

    Theorem 2 assumes the phenomenon that would require explanation. Assuming expected instantaneous regret is at least ρ>0\rho>0 immediately gives RH≥HρR_H\ge H\rho, and comparison with a sublinear bound yields eventual crossover. The mathematical implication is straightforward. The interesting question is whether the proposed policy has such a persistent floor and what determines it. Frozen network parameters do not establish this: the recurrent state continues to incorporate observations and can implement an adaptive learning algorithm. The theorem therefore provides neither a characterization of the proposed policy’s long-horizon behavior nor a principled handoff time.

The additional results on data processing, Rao–Blackwellization, and equivariance are applications of standard facts. The local Fisher-information calculation is more specific to the observation model, but is not connected to a finite-sample or regret guarantee for the trained learner. Taken together, I do not find sufficient theoretical novelty to support acceptance.

Questions for authors
  1. Can the authors establish a quantitative benefit of successor-conditioned prefixes for a natural family of downstream learners, accounting for approximation error in the teacher?
  2. Under what verifiable conditions does the proposed training procedure control the KL-fit and deployment-shift terms sufficiently to obtain a useful regret guarantee?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.