- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission studies transfer learning in neural contextual bandits. The idea behind transfer learning is that if a neural network has already learned a related task, it may be possible to use this knowledge to help a learner make good decisions on a new task without starting from scratch. This paper asks if a pretrained model can reduce the exploration necessary in a contextual bandit task while accounting for the possibility that the original task significantly differs from the new one.
The authors propose an algorithm called Transfer-NeuralUCB, which works by keeping two upper-confidence bounds (one inherited from the source task and one learned from the target feedback) and combines them by taking their pointwise maximum. Their algorithm then takes the action which maximizes this combined confidence bound, conditioned on the observed context.
They prove regret guarantees under the assumptions that the underlying reward functions belong to the function space associated with the neural network's neural tangent kernel and the kernel matrix is sufficiently well-conditioned. The experiments compare Transfer-NeuralUCB with vanilla NeuralUCB, NeuralUCB warm-started with the source-trained weights, and LinUCB on both synthetic and semi-synthetic datasets. Transfer-NeuralUCB significantly outperforms the baselines on the synthetic tasks. On the semi-synthetic tasks it still performs well, but the gap between it and the baselines is much narrower.
Critical strengths
The paper's idea of modifying NeuralUCB for the transfer learning setting by keeping two confidence intervals is a natural and intuitive idea.
I also appreciated that the authors strove to include both a theoretical analysis and experimental results which went beyond purely synthetic data. I liked the fact that all assumptions were clearly stated before the regret bound was presented.
Critical weaknesses
With that being said, I have some questions about the relationship between the theoretical assumptions and what is being used in practice. In particular, Remark D.5 does not feel like something that should be left for future work in the conclusion, as it appears to be a significant drawback of the current theoretical analysis since (if my understanding is correct) it implies that the main regret bound does not actually establish the claimed benefit of learning a simpler residual.
The conceptual novelty of this submission is also a weakness. The authors are not the first to study transfer learning in neural contextual bandits and the algorithm idea, while intuitive, is based on the very well-studied UCB/optimism-in-the-face-of-uncertainty principle.
Another weakness is the seemingly relatively weak improvement that Transfer-NeuralUCB has over the baselines on the semi-synthetic experiments.
Finally, I would have liked to have seen a more thorough baseline comparison in the empirical results. In particular, Square CB-based methods can also leverage neural networks and are broadly known to outperform UCB-style methods in many settings, and so it would be good to compare against them.
Questions for authors
More of a comment than a question: I feel like it would significantly improve readability if the related work were presented in the main body and the algorithm analysis were deferred to an appendix.
Why did you not also compare against existing work on neural contextual bandits?
Why does Transfer-NeuralUCB perform similarly to the baselines on the semi-synthetic datasets?
2 comments
Thanks for your reply. I will maintain my score, as while I think that these changes will improve the paper, they are still hypothetical at this point and have not been implemented.
Thank you for the review. We address the concerns as below.
W1. The certified benefit of transfer comes from the source branch. Theorem 4.5 is a best-of-both-worlds guarantee: by (13), the regret is at most twice the round-by-round minimum of the online width and the source width. The source branch exploits the offline data, while the online branch retains the O~(d~occ+T) form of NeuralUCB in (15). The benefit of learning a simpler residual appears with an oracle norm bound, which NeuralUCB also takes as input (2h⊤H−1h). With ∣u\*∣ in place of S\*, the regularization term improves whenever ∣u\*∣ is smaller than the target norm, and Corollary D.4 characterizes when this holds through the alignment condition τ<2ρ. We will move this discussion from the conclusion to Section 4 and clarify the abstract and the introduction accordingly.
W2. We agree that the UCB principle is standard and that transfer in neural bandits has been studied. In this work, we model the shift in the reward, and both the source sample size and the mismatch enter the regret, through αN and ϵ. The technical contribution is a confidence analysis around a center that depends on the source rewards rather than on the random initialization, which is more practically relevant (Lemmas 4.7 and 4.8). Its design-weighted certificate (10) recovers the direction-aware condition in the linear case.
W3, Q3. At p≤0.5, Transfer-NeuralUCB is significantly better than all three baselines on every dataset (Table 5). At p=0, its regret relative to the cold start is 0.86 on magic, 0.45 on shuttle, 0.82 on covertype, and 0.69 on mnist. The gaps look smaller than on synthetic data for two reasons. First, the rewards are 0/1 class labels on 10 PCA features, so every method pays a misclassification cost that is linear in T; on magic, all neural methods incur about 0.14 regret per round at T=20000 (Table 7). This common term compresses the relative gaps. Second, the 0/1 rewards lie outside the NTK RKHS (Appendix I.1), so these tasks test the method beyond its assumptions.
W4, Q2. Source centering only changes the regression center, so it applies to any method built on a regression oracle. We will add SquareCB with a neural regression oracle and Neural Thompson Sampling, each in a cold-start and a source-centered version. This separates the effect of the transfer mechanism from that of the exploration rule.
Q1. We will move a condensed related-work section into the main text and shorten the proof sketch in Sections 4.2 and 4.3, deferring the details to the appendix.