Thanks to the large-scale labeled datasets available, deep learning has made a great breakthrough in computer vision. However, due to the ambiguity of emotion semantics, it is hard to annotate the emotional labels for images. Thus, only a few small-scale image emotion datasets are open and available, restricting the performance of image emotion analysis based on deep learning. The semantics of emotions have unique characteristics, such as order and polarity, but few studies have paid attention to these essential characteristics in image emotion analysis. Thus, in the paper, based on domain adaptation, considering the essential characteristics of emotion semantics, that is, the ordered and grouped polarity, we proposed to measure emotion semantic differences through earth mover’s distance (EMD). The goal is to better transfer the trained model with labeled emotion dataset to unlabeled emotion dataset and complete the unsupervised image emotion analysis. The Yunnan Heavy Color Paintings Emotion dataset was created in this paper, and was applied to our proposed method. The experimental results demonstrate that the proposed method can effectively align the emotional semantics between the source domain and the target domain, realizing the unsupervised automatic annotation of emotion dataset, thus expanding the size of the image emotion dataset.