| 引用本文： |
-
黄赞,袁得嵛,苗博,杨懿,吴钦.特征缝合后门：利用分块特征对齐攻击视觉语言模型[J].信息安全学报,已采用 [点击复制]
- huangzan,yuandeyu,miaobo,yangyi,wuqin.Feature-Stitching Backdoor: Attacking Visual Language Models via Patch-Wise Feature Manipulation[J].Journal of Cyber Security,Accept [点击复制]
|
|
| 摘要: |
| 视觉语言模型（VLMs）依赖开放来源的大规模图像—文本数据进行训练，使其面临数据投毒和后门攻击风险。现有VLM后门攻击多依赖显式像素触发器、整体图像扰动或图文语义错配，较少关注视觉Transformer（ViT）分块表示机制可能带来的安全隐患。本文提出一种特征缝合后门攻击方法（FSB），通过分块级特征对齐优化，对训练图像块施加视觉上较难察觉的微扰动，使中毒patch的深层视觉表示向攻击者指定的目标语义特征靠近，并在视觉指令微调阶段诱导模型学习错误的视觉—文本关联。与依赖固定像素模式的传统触发器不同，FSB在推理阶段不需要额外添加显式触发器。实验以LLaVA-1.5-7B为主要基准模型，并与BadNet、Blended、ISSBA、AnyDoor、ShadowCast和VL-Trojan等方法进行对比。结果表明，在2%投毒率下，FSB达到94.0%的攻击成功率，并保持94.6%的良性准确率；相较于未投毒干净模型，其良性准确率下降值为5.4个百分点，体现出攻击效能与模型效用之间的权衡。进一步实验显示，FSB在PSNR、SSIM和特征对齐均值等指标上具有较高隐蔽性，并在STRIP、Activation Clustering和Neural Cleanse-style检测设置下表现出较低可检测性。表示空间可视化结果表明，poisoned patch相较clean patch更接近目标语义特征，为分块特征对齐参与错误关联学习提供了直观证据。本文结果揭示，VLM中的局部视觉特征表示可能被用于构造隐式后门攻击，后续防御需关注分块级特征异常和跨模态关联异常。 |
| 关键词: 视觉Transformer 视觉语言模型 后门攻击 特征对齐 视觉缝合 |
| DOI： |
| 投稿时间：2026-05-02修订日期：2026-07-22 |
| 基金项目:公安部技术研究计划重点项目 |
|
| Feature-Stitching Backdoor: Attacking Visual Language Models via Patch-Wise Feature Manipulation |
|
huangzan1,2,3, yuandeyu1,2,3, miaobo1,2,3, yangyi1,2,3, wuqin1,2,3
|
| (1.People'2.'3.s Public Security University of China) |
| Abstract: |
| Vision-language models (VLMs) are commonly trained on large-scale image-text data collected from open sources, which makes them vulnerable to data poisoning and backdoor attacks. Existing backdoor attacks against VLMs mainly rely on explicit pixel triggers, whole-image perturbations, or semantic mismatches between images and texts, while the security risks introduced by the patch-wise representation mechanism of Vision Transformers (ViTs) remain insufficiently explored. In this paper, we propose Feature-Stitching Backdoor (FSB), a patch-wise feature alignment attack against VLMs. FSB adds subtle perturbations to training image patches so that the deep visual representations of poisoned patches move toward an attacker-specified target semantic feature. These poisoned patches are then paired with the target text during visual instruction tuning, inducing the model to learn a spurious visual-textual association. Unlike conventional trigger-based attacks, FSB does not require an explicit pixel trigger at inference time.We evaluate FSB mainly on LLaVA-1.5-7B and compare it with representative backdoor methods, including BadNet, Blended, ISSBA, AnyDoor, ShadowCast, and VL-Trojan. On the Pisa-to-Big-Ben attack task, FSB achieves a 94.0% attack success rate with a 2% poisoning rate while maintaining a benign accuracy of 94.6%. Compared with the clean model, the benign accuracy drops by 5.4 percentage points, indicating a trade-off between attack effectiveness and model utility. In terms of stealthiness, FSB achieves favorable visual quality and feature alignment metrics, with a PSNR of 33.25 dB, an SSIM of 0.9248, and a feature alignment mean of 0.899. Moreover, FSB shows low detectability under STRIP, Activation Clustering, and Neural Cleanse-style detection settings. Representation-space visualization further shows that poisoned patches are closer to the target semantic feature than clean patches, providing direct evidence for the targeted feature shift induced by FSB. These results suggest that local visual representations in VLMs can be exploited to construct implicit backdoor behaviors, motivating future defenses against patch-level feature anomalies and abnormal cross-modal associations. |
| Key words: Vision Transformer ,Vision-Language Models ,backdoor attack ,feature alignment ,visual stitching |
