acceptodds
Under review as a conference paper at ICLR 2027

Transfer Learning with Neural UCB

Abstract

We study transfer learning in neural contextual bandits, where labeled data from a source task are available before interaction with a target task whose reward function may differ. We propose Transfer-NeuralUCB, which uses a predictor trained on the source task as the center for target adaptation and transfers source uncertainty through a design-weighted certificate for reward mismatch. The algorithm combines a mismatch-adjusted source UCB with an online UCB learned from target feedback and selects actions using their pointwise minimum. Given an RKHS transfer certificate and finite-width neural tangent kernel conditions, we establish simultaneous confidence guarantees along the source and target training iterates. The resulting regret bound is controlled by the smaller of the cumulative source and online confidence widths. Our analysis identifies residual tangent complexity as the regularization cost of adapting the source predictor and characterizes, through inverse-kernel alignment and scale, when the source prediction is simpler to correct than the target reward is to learn from scratch.Experiments on synthetic and classification-derived bandits show that Transfer-NeuralUCB improves on NeuralUCB with a cold or warm start and on LinUCB in most settings, that the gain comes from source-centered residual learning and persists over rounds, and that it turns into a loss under strong source–target mismatch.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-arm bandit, upper confidence bounds, neural networks, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6BQnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission studies transfer learning in neural contextual bandits. The idea behind transfer learning is that if a neural network has already learned a related task, it may be possible to use this knowledge to help a learner make good decisions on a new task without starting from scratch. This paper asks if a pretrained model can reduce the exploration necessary in a contextual bandit task while accounting for the possibility that the original task significantly differs from the new one.

The authors propose an algorithm called Transfer-NeuralUCB, which works by keeping two upper-confidence bounds (one inherited from the source task and one learned from the target feedback) and combines them by taking their pointwise maximum. Their algorithm then takes the action which maximizes this combined confidence bound, conditioned on the observed context.

They prove regret guarantees under the assumptions that the underlying reward functions belong to the function space associated with the neural network's neural tangent kernel and the kernel matrix is sufficiently well-conditioned. The experiments compare Transfer-NeuralUCB with vanilla NeuralUCB, NeuralUCB warm-started with the source-trained weights, and LinUCB on both synthetic and semi-synthetic datasets. Transfer-NeuralUCB significantly outperforms the baselines on the synthetic tasks. On the semi-synthetic tasks it still performs well, but the gap between it and the baselines is much narrower.

Critical strengths

The paper's idea of modifying NeuralUCB for the transfer learning setting by keeping two confidence intervals is a natural and intuitive idea.

I also appreciated that the authors strove to include both a theoretical analysis and experimental results which went beyond purely synthetic data. I liked the fact that all assumptions were clearly stated before the regret bound was presented.

Critical weaknesses

With that being said, I have some questions about the relationship between the theoretical assumptions and what is being used in practice. In particular, Remark D.5 does not feel like something that should be left for future work in the conclusion, as it appears to be a significant drawback of the current theoretical analysis since (if my understanding is correct) it implies that the main regret bound does not actually establish the claimed benefit of learning a simpler residual.

The conceptual novelty of this submission is also a weakness. The authors are not the first to study transfer learning in neural contextual bandits and the algorithm idea, while intuitive, is based on the very well-studied UCB/optimism-in-the-face-of-uncertainty principle.

Another weakness is the seemingly relatively weak improvement that Transfer-NeuralUCB has over the baselines on the semi-synthetic experiments.

Finally, I would have liked to have seen a more thorough baseline comparison in the empirical results. In particular, Square CB-based methods can also leverage neural networks and are broadly known to outperform UCB-style methods in many settings, and so it would be good to compare against them.

Questions for authors

More of a comment than a question: I feel like it would significantly improve readability if the related work were presented in the main body and the algorithm analysis were deferred to an appendix.

Why did you not also compare against existing work on neural contextual bandits?

Why does Transfer-NeuralUCB perform similarly to the baselines on the semi-synthetic datasets?

2 comments

Thank you for the review. We address the concerns as below.

W1. The certified benefit of transfer comes from the source branch. Theorem 4.5 is a best-of-both-worlds guarantee: by (13), the regret is at most twice the round-by-round minimum of the online width and the source width. The source branch exploits the offline data, while the online branch retains the O~(d~occ+T)\tilde O(\tilde d^+_{occ}\sqrt{T}) form of NeuralUCB in (15). The benefit of learning a simpler residual appears with an oracle norm bound, which NeuralUCB also takes as input (2h⊤H−1h\sqrt{2h^\top H^{-1}h}). With ∣u\*∣\\|u_\*\\| in place of S\*S_\*, the regularization term improves whenever ∣u\*∣\\|u_\*\\| is smaller than the target norm, and Corollary D.4 characterizes when this holds through the alignment condition τ<2ρ\tau < 2\rho. We will move this discussion from the conclusion to Section 4 and clarify the abstract and the introduction accordingly.

W2. We agree that the UCB principle is standard and that transfer in neural bandits has been studied. In this work, we model the shift in the reward, and both the source sample size and the mismatch enter the regret, through αN\alpha_N and ϵ\epsilon. The technical contribution is a confidence analysis around a center that depends on the source rewards rather than on the random initialization, which is more practically relevant (Lemmas 4.7 and 4.8). Its design-weighted certificate (10) recovers the direction-aware condition in the linear case.

W3, Q3. At p≤0.5p \le 0.5, Transfer-NeuralUCB is significantly better than all three baselines on every dataset (Table 5). At p=0p = 0, its regret relative to the cold start is 0.86 on magic, 0.45 on shuttle, 0.82 on covertype, and 0.69 on mnist. The gaps look smaller than on synthetic data for two reasons. First, the rewards are 0/1 class labels on 10 PCA features, so every method pays a misclassification cost that is linear in TT; on magic, all neural methods incur about 0.14 regret per round at T=20000T = 20000 (Table 7). This common term compresses the relative gaps. Second, the 0/1 rewards lie outside the NTK RKHS (Appendix I.1), so these tasks test the method beyond its assumptions.

W4, Q2. Source centering only changes the regression center, so it applies to any method built on a regression oracle. We will add SquareCB with a neural regression oracle and Neural Thompson Sampling, each in a cold-start and a source-centered version. This separates the effect of the transfer mechanism from that of the exploration rule.

Q1. We will move a condensed related-work section into the main text and shorten the proof sketch in Sections 4.2 and 4.3, deferring the details to the appendix.

Thanks for your reply. I will maintain my score, as while I think that these changes will improve the paper, they are still hypothetical at this point and have not been implemented.

Reviewer_7ujzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies transfer learning in neural contextual bandits. In this setting, labeled data from a source task are available before online interaction with a target task whose reward function may differ. It proposes Transfer-NeuralUCB, which transfers a source-trained predictor and an estimate of its uncertainty. The algorithm first trains a ReLU network on source data to obtain parameters bb. During target interaction, it updates the network using a ridge penalty centered at bb. This helps it to learn the residual between the target reward and the source prediction. It also maintains a fixed source-based upper confidence bound, enlarged by a supplied RKHS transfer certificate ϵ\epsilon to account for source–target mismatch. This is sort of standard in these types of neural bandits. At each round, the algorithm selects the action maximizing the pointwise minimum of the source-based and online confidence bounds.

Under neural tangent kernel (NTK) assumptions and finite-width conditions, Theorem 4.5 bounds regret by twice the smaller of the cumulative online confidence width and the cumulative source confidence width plus a mismatch term linear in TT. The analysis introduces a residual-complexity quantity S∗S_* and characterizes conditions under which correcting the source predictor may be easier than learning the target reward from scratch. Experiments on synthetic NTK rewards and four classification-derived bandit tasks compare Transfer-NeuralUCB with warm-start and cold-start NeuralUCB and LinUCB. The method achieves the lowest regret in 26 of 32 reported settings.

Critical strengths

The strengths are as follows:

  1. The technical analysis is good. It uses fixed source-trained features ϕb\phi_b in the self-normalized bound, keeping the source phase separate from the online martingale argument. It explicitly handles the measurability of the residual representer. It also recovers a direction-aware condition related to Han et al. in the linear case. These are nice contributions in neural bandits.

  2. The paper addresses an interesting problem that combines transfer learning with uncertainty-aware action selection. Reusing a model trained on related historical data while accounting for source–target mismatch is an important deployment setting.

  3. The authors explicitly discuss several important limitations: the source-based confidence bound does not determine action selection in the reported selected configurations. Also, the warm-start baseline loses much of its source information after refitting. Remark D.5 acknowledges that the certified residual bound does not capture the empirical gain.

Critical weaknesses

The weaknesses are as follows:

  1. It is not clear where the exact gains come from. The proposed method has two potential sources of improvement: online adaptation regularized toward the source-trained parameters, and a source-based confidence bound combined with the online bound through a pointwise minimum. The authors need to clarify which contributes the most.

  2. The source-based confidence bound is constructed from source information and does not incorporate target observations. Consequently, its uncertainty does not tighten through online target feedback, while the regret bound also includes a mismatch contribution that grows linearly with the horizon. The authors need to clarify when the frozen branch should be expected to improve on the online branch.

  3. The theoretical regime appears to be very different from the experiments. The finite-width conditions are very strict, and the paper does not adequately connect them to the experimental configurations.

  4. The online term inherits an effective-dimension form from NeuralUCB, with the dimension measured over candidate-context occurrences. The paper does not report the realized log-determinant ΓT\Gamma_T or bound the relevant occurrence-based effective dimension for its NTK and input domain.

  5. The empirical comparisons do not sufficiently isolate the benefit of the proposed method from simpler transfer-learning approaches. More ablations are needed on this.

Questions for authors

Here are the questions for the authors. If they are addressed, I am willing to raise my score.

  1. Could you confirm or refute the following chain of inequalities under the assumptions used in Appendix F?

S∗≥(αN+ϵ)/λsrcS_* \geq (\alpha_N+\epsilon)/\sqrt{\lambda_{\rm src}}, by (76);

αN≥λsrcBsrc≥λsrc∣rsrc∣H\alpha_N \geq \sqrt{\lambda_{\rm src}}B_{\rm src} \geq \sqrt{\lambda_{\rm src}}|r_{\rm src}|_{\mathcal H}, by (70);

ϵ≥λsrc∣d∣H\epsilon \geq \sqrt{\lambda_{\rm src}}|d|_{\mathcal H}, by (10);

Using rtar=rsrc+dr_{\rm tar}=r_{\rm src}+d and the relevant minimum-norm interpolation inequality, this appears to imply a lower bound on S∗S_* in terms of the target reward's RKHS complexity.

If this chain is valid, what does the online part of Theorem 4.5 guarantee beyond the standard cold-start bound?

  1. Why freeze the source branch? Han et al. use an offline-informed branch that incorporates online data. Given that the source width does not tighten with target feedback and the bound includes a linear-in-TT mismatch term, under what conditions or horizons should the frozen branch improve on the online branch?

  2. Why does cold-start NeuralUCB vary across pp? In Tables 5–7, including untuned Table 6, why do cold-start results vary across label-permutation levels when the method does not use source labels?

  3. Is the effective-dimension term sublinear in the experiments? Can you bound IoccI_{\rm occ} for the NTK on the input domain used, and report the realized ΓT\Gamma_T and d~occ+\tilde d^+_{\rm occ}? This would help establish whether the effective-dimension term is sublinear in the regimes tested.

  4. Does Corollary D.4 predict empirical gains?

  5. Can you add stronger transfer baselines? In particular, can you compare against NeuralUCB initialized at bb with weaker regularization or early stopping, pooled source-and-target training, Neural-LinUCB on the source representation, Neural Thompson Sampling, biased-regularization LinUCB?

1 comment

Thank you for the careful reading and detailed questions. We answer them in order.

Q1. Yes, the chain is valid. Theorem 4.5 is a best-of-both-worlds guarantee: by (13), the regret is at most twice the round-by-round minimum of the online width and the source width. The online part retains the O~(d~occ+T)\tilde O(\tilde d^+_{occ}\sqrt{T}) form of the cold-start bound in (15), and the source part exploits the offline data, so transfer improves the bound whenever the source branch is tighter. In addition, NeuralUCB takes the oracle norm 2h⊤H−1h\sqrt{2h^\top H^{-1}h} as input. With the analogous oracle norm ∣u\*∣\\|u_\*\\| in place of S\*S_\*, the online part improves on the cold start whenever ∣u\*∣\\|u_\*\\| is smaller, which Corollary D.4 characterizes through the alignment condition τ<2ρ\tau < 2\rho.

Q2, W2. The bound (13) takes the minimum in every round, so the source width is paid only in rounds where it is smaller; the linear term TΔsrcT\Delta_{src} appears only in the coarser bound (14). Let wsrc=max⁡z(ρsrcssrc(z)+Δsrc)w_{src} = \max_z (\rho_{src} s_{src}(z) + \Delta_{src}). The average online width over the first tt rounds is at most βt2Γt/t\beta_t\sqrt{2\Gamma_t/t}, so the two bounds cross at about t≈2βt2Γt/wsrc2t \approx 2\beta_t^2\Gamma_t/w_{src}^2. This horizon is long when the source coverage is good and the mismatch is small. Target data enter through the online branch, which is centered at bb. A branch that pools source and target data would need a confidence analysis for a network trained on biased pooled data; we will discuss this and add a pooled baseline (Q6).

Q3. Cold-start NeuralUCB does not use source labels, and all settings of a dataset share the split, the permutation, and the arm orders; accordingly, LinUCB is identical across pp. The variation comes from the network initialization θ0\theta_0, which is drawn per setting after the source labels are generated, and, in the tuned tables, from the selected multiplier. We will fix θ0\theta_0 across pp in the revision.

Q4, W4. Yes. In the synthetic experiments, M=100M = 100 or 10001000 and Q=10TQ = 10T, where MM is the number of distinct candidate actions and QQ the number of their occurrences. Since HoccH_{occ} has rank at most MM, Iocc≤Mlog⁡(1+Qκ2/(Mλon))I_{occ} \le M\log(1 + Q\kappa^2/(M\lambda_{on})), which is logarithmic in TT, so the effective-dimension term is sublinear. On real data, MM grows with the stream, so we will report the realized ΓT\Gamma_T, which we accumulate exactly (Appendix I.1), and d~occ+\tilde d^+_{occ} for every setting.

Q5. All quantities in Corollary D.4 are computable on synthetic instances: hh and gg on Z\*Z_\*, and H\*H_\* from the exact NTK. We will report the ratio ∣q\*∣H\*−1/∣h∣H\*−1=1−2ρτ+τ2\\|q_\*\\|_{H_\*^{-1}}/\\|h\\|_{H_\*^{-1}} = \sqrt{1 - 2\rho\tau + \tau^2}. The corollary predicts gains through ∣u\*∣\\|u_\*\\|, not the certified S\*S_\*, consistent with Q1.

Q6, W1, W5. Thank you for the suggestions. We will add the baselines you list: NeuralUCB initialized at bb with weaker regularization or early stopping, pooled source-and-target training, Neural-LinUCB on the source representation, Neural Thompson Sampling, and biased-regularization LinUCB. We will also add ablations that separate the source-centered online model from the source UCB.

W3. The width conditions (160) are sufficient conditions, polynomial in N,M,Q,TN, M, Q, T, and 1/λ01/\lambda_0, as in NeuralUCB. The experiments test the qualitative predictions: the effect of the center bb and the dependence on ϵ\epsilon and NN.

Reviewer_Rt4COpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies transfer learning for neural contextual bandits using labeled source data. The proposed method, Transfer-NeuralUCB, uses the source-trained network for initialization and regularizes subsequent online training toward its parameters. It selects actions using the minimum of a source-based and an online upper confidence bound. Under an RKHS reward-mismatch assumption and finite-width neural network conditions, the authors establish confidence and regret guarantees and evaluate the method on synthetic and classification-derived bandits.

Critical strengths
  1. The reviewer is interested in the careful theoretical analysis. Specifically, Lemmas 4.7 and 4.8 establish a residual representation and confidence guarantees for a source-trained network. Since the pretrained parameters depend on the source data, extending the original NeuralUCB analysis requires additional steps in controlling both source training and subsequent online updates.

  2. The experimental evaluation is also reasonably thorough. The authors consider different levels of task mismatch, source sample sizes, noise levels, and time horizons, including cases where transfer hurts performance. The use of paired instance and comparable tuning budgets also make the comparisons ore informative.

Critical weaknesses
  1. Several key ideas have been studied before, including prior-centered regularization in BIAS-OFUL (Cella et al., 2020), minimum UCB aggregation (Cheung and Lyu, 2024), and using pretrained neural networks for bandits (Terekhov, 2025). While these papers consider different settings, the main new contribution here seems to be extending NeuralUCB analysis to a source-trained network. It would be helpful to compare the proposed method with a kernel-based residual learning using fixed features, to better understand the benefit of updating the neural network in an online manner.

  2. The current guarantee does not clearly quantify all the proposed benefits of transfer. Specifically, Corollary D.4 characterizes when the actual residual is simpler than the target reward, but the certified complexity need not reflect this gain (Remark D.5). It remains unclear when the online regret bound improves over standard NeuralUCB. Moreover, the theoretical results requires a valid mismatch certificate, while the real experiments generally use an uncertified estimate.

  3. The experiments compare against standard and warm start NeuralUCB, but do not separately evaluate source-centered regularization without the source UCB. Such a comparison would help clarify which part of the proposed method contributes to the performance improvements.

Questions for authors
  1. Can the authors clarify when the stated residual compelxity bound leads to a better online regret guarantee than standard NeuralUCB, given the limitation discussed in Remark D.5?

  2. How does the proposed method compare with source-centered NeuralUCB without the source UCB, or with a kernel-based method that keeps the neural features fixed? How often does the source UCB actually change the selected action?

  3. Appendix I.1 describes estimating the mismatch using source and online contexts. Does this require target reward labels before online learning, and how sensitive is the method to errors in this estimate?

1 comment

Thank you for the careful reading and the concrete suggestions.

W1. We agree that each ingredient has precedents: prior-centered regularization, the minimum of two UCBs, and pretrained networks in bandits. Our contribution is the analysis that keeps both branches valid along finite-width neural training when the center bb is trained on the source labels. NeuralUCB centers at a random initialization that is independent of all data; here the center depends on the source rewards. We will cite them and state in the introduction which parts are inherited.

W2, Q1. The certified benefit of transfer comes from the source branch. Theorem 4.5 is a best-of-both-worlds guarantee: by (13), the regret is at most twice the round-by-round minimum of the online width and the source width. The source branch exploits the offline data, while the online branch retains the O~(d~occ+T)\tilde O(\tilde d^+_{occ}\sqrt{T}) form of NeuralUCB in (15). The residual gain appears with an oracle norm bound, which NeuralUCB also takes as input (2h⊤H−1h\sqrt{2h^\top H^{-1}h}). With ∣u\*∣\\|u_\*\\| in place of S\*S_\*, our regularization term improves on that of NeuralUCB whenever ∣u\*∣\\|u_\*\\| is smaller than the target norm, and Corollary D.4 characterizes when this holds. We will state both facts in Section 4 and clarify the abstract accordingly.

W3, Q2. Thank you for this suggestion. We will add ablation studies, including source-centered NeuralUCB without the source UCB and a kernel-based method with fixed source features, to clarify which part of the proposed method contributes to the performance improvements.

Q3. Yes, for the real-data experiments only. The algorithm and the theory take the certificate ϵ\epsilon as an input, and on synthetic data it is known by construction. On real data, we compute a plug-in value from (10), with ∣d∣H\\|d\\|_{\mathcal H} replaced by the norm of the minimum-norm NTK interpolant of dd on 300 source and 300 online contexts (Appendix I.1). This plug-in is a lower bound on the true ϵ\epsilon, so our real-data results are obtained with an underestimated mismatch, which is the unfavorable direction for the source UCB. A correct, larger ϵ\epsilon only raises UtsrcU^{src}_t, so the minimum in (9) relies more on the online UCB, which is still centered at the source predictor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.