To address the problems of low efficiency, poor accuracy, and insufficient robustness of detection methods due to the variety and complex distribution of weeds in the field, the seven kinds of common vegetables and field weeds in the seedling stage in the seedling field were taken as the research objects, the weed detection was reversely converted into crop detection, and a weed detection algorithm in vegetable seedling fields based on optimized YOLOv4 and image processing was proposed. Based on the YOLOv4 object detection algorithm, the backbone feature extraction network was embedded in SA module to enhance the feature extraction capability, the Transformer module was introduced to construct the long-distance global semantic information of the feature map, and the detection head and loss function were improved to increase the detection and positioning accuracy. The improved model’s average recognition time for a single image was 0.261 s, and the average recognition accuracy rate was 97.49%. Under the same training samples and system environment settings, the improved method was compared with the mainstream target detection algorithms Faster RCNN, SSD, and YOLOv4. The results show that the improved YOLOv4 model is of evident advantages in the identification of various vegetables in the seedling stage. The improved YOLOv4 target detection algorithm was used to detect crops: the vegetation outside the crop area is weeds, and the excess-green feature was combined with the OTSU threshold segmentation algorithm to obtain the weed foreground. Finally, the connected component of the weed foreground was marked to output the weed centroid coordinates and the position of the detection frame. In doing so, weed can be effectively detected in vegetable seedling fields.