Attention-Sink-Aware Hallucination Attacks and Defenses on Multimodal LLMs

Course. AI Security (Spring 2026), POSTECH. — Slides (PDF)

My prior research centered on AI privacy through privacy-enhancing technologies (PETs) such as homomorphic encryption. I took this course to broaden my perspective toward non-PET AI security — in particular, adversarial attacks and defenses.

Project. This project studies attention-sink-aware adversarial hallucination attacks and defenses on multimodal large language models (MLLMs). While exploring attention-sink-aware adversarial misclassification attacks, I found Mirage in the Eyes [1] (USENIX Security ‘25), which proposes a closely related idea: it induces hallucinations by stealthily forming adversarial attention sinks during the decoding stage. This motivated the central question of the project — can the attention-sink mitigation techniques developed in the quantization [2] and model-improvement [3] literature serve as an effective defense?

Method. Building on the Attention Calibration Technique (ACT) from the model-improvement literature [3], I proposed DASH (Decoding-stage Attention-Sink Head mitigation). DASH identifies attention heads whose decoding-stage sinks are amplified by the attack yet remain safe to recalibrate on clean inputs, and redistributes their attention away from the adversarial sink tokens.

Result. Owing to limited compute (API rate limits), I could not run the full experimental suite, but DASH showed a stronger defense than the mitigation reported alongside Mirage. For example, on LLaVA-1.5 (white-box), the hallucination success rate rises from a clean baseline of 33.9% to 39.0% under Mirage; the existing decoding-based defense (OPERA) fails to suppress it (42.7%), whereas DASH brings it back down to 36.1% — close to the clean baseline.

Takeaway. The project gave me hands-on grounding in the fundamentals of non-PET adversarial attack and defense research.

References

[1] Wang, Yining, et al. “Mirage in the eyes: Hallucination attack on multi-modal large language models with only attention sink.” 34th USENIX Security Symposium (USENIX Security 25). 2025.

[2] Son, Seungwoo, et al. “Prefixing attention sinks can mitigate activation outliers for large language model quantization.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

[3] Yu, Zhongzhi, et al. “Unveiling and harnessing hidden attention sinks: enhancing large language models without training through attention calibration.” Proceedings of the 41st International Conference on Machine Learning. 2024.