Multimodal Generative Models for Text-Image-Audio Integration

Abstract

Generative AI has advanced rapidly in individual modality domains -- text generation through large language models, image synthesis through diffusion models, audio generation through neural codec language models -- but the integration of all three modalities within a unified generative architecture capable of cross-modal conditioning, joint generation, and coherent multimodal output remains an open and technically challenging problem. This paper proposes the Trimodal Generative Integration (TGI) architecture, a unified framework for text-image-audio joint generation and cross-modal conditioning based on a shared latent space representation with modality- specific encoders, a cross-modal attention transformer core, and modality- specific decoders. TGI supports four generation modes: text-conditioned image-audio generation (given a text description, generate coherent image and audio); image-conditioned text-audio generation; audio-conditioned text-image generation; and unconditioned trimodal joint generation. The TGI architecture is trained on a curated trimodal dataset of 2.1M aligned text-image-audio triplets (TIA-2M) constructed from captioned video content with automatic audio transcription and alignment. Evaluation on four multimodal generation benchmarks demonstrates that TGI achieves state-of-the-art cross-modal coherence scores: cross-modal semantic alignment (CLIP-Audio similarity) of 0.74 (SD = 0.04) versus 0.61 for best competing bimodal system; image generation quality (FID = 18.4, SD = 2.1) competitive with specialist text-to-image models; and audio generation naturalness (MOS = 4.2/5.0, SD = 0.3) competitive with specialist TTS systems. The study contributes the TGI architecture, the TIA-2M trimodal dataset, and a Trimodal Generation Quality (TGQ) evaluation suite covering seven cross-modal coherence and generation quality dimensions.