- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies robust in-context learning through adversarial pretraining. The authors theoretically show that sufficiently deep adversarially trained linear transformers can approach the robust Bayes error on unseen tasks, while standard training cannot. Synthetic and real-world experiments are provided to support the theoretical results.
Critical strengths
- The paper provides a solid theoretical analysis of robust in-context learning and clearly distinguishes adversarial from standard pretraining.
- The theoretical results cover multiple aspects, including Bayes optimality, optimization, and robustness–accuracy trade-offs.
Critical weaknesses
-
The empirical evaluation is relatively limited. The real-world experiments mainly compare standard and adversarial training on three datasets, without stronger alternative robust pretraining or adaptation baselines.
-
The theoretical results rely on several structured assumptions, particularly the Gaussian-mixture setting with shared mean and covariance across tasks. This raises some concerns about how broadly the theoretical conclusions apply beyond the considered setting.
Questions for authors
-
The empirical evaluation mainly compares standard training with adversarial training. Could the authors clarify why other relevant robust pretraining or adaptation baselines are not included?
-
How restrictive are the assumptions underlying the theoretical results, particularly the shared mean/covariance structure across tasks and the condition ? To what extent would the main conclusions hold if these assumptions were relaxed?