Abstract: When a large-scale neural network is trained on a small training set, it typically yields “overfitting”, i.e., the model performs poorly on held-out test data. Therefore, various Dropout techniques have been proposed to alleviate this problem. However, the aforementioned methods cannot directly encourage the model to learn the less discriminative parts, which is also important to reducing overfitting. To address this problem, we proposed an attention-guided Dropout (AD), which utilized the self-attention mechanism to alleviate the co-adaptation of feature detectors more effectively. The AD comprised two distinctive components, the importance measurement mechanism for feature maps and the Dropout with a learnable probability. The importance measurement mechanism calculated the degree of importance for each feature map in whole by a Squeeze-and-Excitation block. The Dropout with a learnable probability can force the “bad” neurons to learn a better representation by dropping the “good” neurons. Therefore, it will diminish the co-adaptation and encourage models to learn the less discriminative part. The experimental results show that the proposed method can be easily applied to various convolutional neural network (CNN) architectures, thus yielding better performance. 

Key words:  , deep neural network, overfitting, Dropout, self-attention mechanism, image classification