Fusion Strategies for Vision-Language Generative Systems

Abstract

Vision-language generative systems -- models that jointly process visual and textual inputs to produce coherent multimodal or unimodal outputs -- are a central class of multimodal AI systems with diverse applications in image captioning, visual question answering, text-to-image generation, and visual dialogue. The fusion strategy -- the architectural mechanism by which visual and textual representations are combined -- is a critical design decision that substantially impacts generation quality, cross-modal coherence, and computational efficiency. While individual fusion approaches have been extensively studied, a systematic comparative evaluation of fusion strategies across multiple generation tasks, model scales, and evaluation dimensions remains absent. This paper presents a comprehensive comparative study of seven vision-language fusion strategies -- early fusion, late fusion, cross-attention fusion, query-based fusion (Q-Former), gated fusion, mixture-of-modalities fusion, and hierarchical dual-stream fusion -- evaluated across five vision-language generation benchmarks at three model scales (3B, 7B, 13B). Results demonstrate significant fusion strategy effects across tasks: cross-attention fusion achieves best visual question answering performance (VQAv2 accuracy = 82.4%); Q-Former fusion achieves best image captioning (CIDEr = 148.6); hierarchical dual-stream achieves best text-to-image generation (FID = 19.2); and early fusion achieves best computational efficiency (training time = 68.4% of cross-attention baseline). The study identifies a generation task-fusion strategy alignment principle: optimal fusion strategy depends on whether the task requires visual grounding (favouring cross-attention/Q-Former) or generation independence from vision (favouring early/late fusion). The paper contributes a fusion strategy taxonomy, a comprehensive benchmark evaluation suite, and design guidelines for vision- language generative system architects.