Towards Understanding How Deep Transformers Perform Classification with Unlabeled Context

Transformers achieve remarkable performance in various tasks in natural language processing and computer vision. Despite the empirical success of transformers, the impact of model depth on performance remains not fully understood. In this paper, we consider a classification task with an unlabeled context, and theoretically investigate the performance of a deep transformer model trained by gradient descent in learning this task. Our analysis demonstrates that deeper transformer layers can produce features with higher signal-to-noise ratio (SNR), and as a result, linear classification based on features of deeper layers can yield higher prediction accuracy. Experiments on both synthetic and real-world datasets are conducted to validate our theory. Our study sheds light on the benefits and impact of transformer model depth, and provides insights into how unlabeled context can enhance transformer performance on supervised tasks.