In China, there are frequent poisoning events caused by ingestion of inedible wild mushrooms every summer, especially in Southwest China, such as Yunnan. This is due to the slight differences in inter-class characteristics of wild mushrooms and the complex image backgrounds in actual scenarios, making it difficult to distinguish only by naked eyes. At present, although there are many methods to classify wild mushrooms, and the most reliable way is molecular identification, the relevant techniques are time-consuming and require a high threshold, so they are not suitable for real-time classification and detection. To solve this problem, an approach based on deep learning was proposed. This approach employed the attention mechanism convolution block attention module (CBAM), was combined with multi-scale fusion, and added the anchor layer. The hyperparameter evolution idea was adopted to adjust the hyperparameter during the model training, so as to improve the recognition accuracy. Compared with standard target detection networks, such as SSD, Faster Rcnn, and Yolo series, the proposed model can classify and detect wild mushrooms more accurately. Compared with the original Yolov5, after the proposed model was improved, Map was improved by 3.7% and reached 93.2%, precision by 1.3%, Recall by 1.0%, and model detection speed by 2.3%. Compared with SSD, Map was improved by 14.3%. Finally, the model was simplified and deployed on Android devices to increase its practicability, thus solving the current problem of poisoning caused by eating inedible wild mushrooms because of the difficulty of identification.