Abstract:

Recently, open-vocabulary detection (OVD) has become a research focus in the field of computer vision due to its potential to recognize objects from unknown categories. As a representative approach in this domain, YOLO-World possesses powerful real-time detection capabilities; however, security issues stemming from the vulnerabilities of deep learning networks cannot be overlooked. Against this backdrop, a white-box adversarial examples generation method was proposed, targeting the YOLO-World algorithm, providing insights into identifying and quantifying vulnerabilities in large models. The method utilized gradient data generated during backpropagation in the YOLO-World network to optimize predefined perturbations, which were then added to original examples to form adversarial examples. Initially, confidence scores and bounding box information from model outputs served as a basis for preliminary optimization, resulting in adversarial examples with a certain level of attack effectiveness. This was further enhanced by a visually-textual fusion loss designed according to the RepVL-PAN structure in the YOLO-World model, to increase the destructiveness of adversarial examples against the model. Finally, perturbation magnitude loss was integrated to constrain the total amount of perturbation, generating adversarial examples with limited disturbance. The adversarial examples generated by this method were capable of achieving attack objectives such as confidence reduction and bounding box displacement according to practical needs. Experimental results demonstrated that the proposed method significantly impaired the YOLO-World model, with mean average precision dropping below 5% after testing on the LIVS dataset.

Key words: open vocabulary detection, YOLO-World, adversarial examples, visually-textual fusion loss, sparse perturbations