Dissecting DNN Overfitting: The Interplay Between Inference Patterns and Overfitting

The overfitting of deep neural networks (DNNs) has rarely been studied from the perspective of their detailed inference patterns. In this paper, we explore whether DNN overfitting can be explained through the generalizability of interactions between input variables encoded by the network. Specifically, we explore this question from four complementary perspectives: (1) We find that overfitted DNNs tend to encode non-generalizable interactions. (2) We discover that removing non-transferable interactions from DNNs effectively mitigates overfitting. (3) This improvement arises because enhancing interaction transferability between two DNNs substantially improves the generalizability of these interactions. (4) We find that among various types of feature consistency across DNNs, improving interaction transferability is the most effective in mitigating overfitting.