Offline mathematical symbol recognition is the premise of offline mathematical expression recognition. The existing offline symbol recognition methods can only recognize symbols, but is of no help to other steps of offline expression recognition, even restricting expression recognition. Thus, an improved YOLOv5s offline symbol recognition method was proposed. Firstly, considering the small size of symbolic image, generative adversarial network (GAN) was employed to enhance the data. Secondly, from the point of view of symbolic categories, the spatial attention mechanism was introduced to YOLOv5s model, and the global maximum and global mean were pooled to enlarge the differences between categories. Finally, from the point of view of the symbol itself, the bidirectional long-short-term memory network (BiLSTM) was utilized to process the symbol feature matrix, so that the symbol feature could possess the upper and lower related information. Experimental results show that the improved YOLOv5s achieves better offline symbol recognition, with a recognition rate of 92.47%. Compared with other methods, the proposed method is effective and robust. At the same time, it can effectively avoid the problem of error accumulation in offline mathematical expression recognition and provide an effective basis for expression structure analysis.