Distributed Training Frameworks for Large-Scale Generative Models

Abstract

Training large-scale generative AI models at frontier quality requires distributed compute infrastructure spanning hundreds to thousands of GPUs, with sophisticated parallelism strategies that coordinate computation and communication across this infrastructure with maximal efficiency. The dominant distributed training paradigms -- data parallelism, tensor parallelism, pipeline parallelism, and their combinations -- each impose distinct communication patterns, memory constraints, and load balancing requirements that must be carefully co-designed with the model architecture and hardware topology for optimal training throughput. This paper proposes the Adaptive Distributed Generative Training (ADGT) framework, a systematic methodology for co-designing distributed training strategies with model architecture for large-scale generative model training, integrating four adaptive parallelism components: topology-aware tensor parallelism (TATP) that matches tensor parallelism degree to the hardware interconnect bandwidth topology; dynamic pipeline scheduling (DPS) that adapts micro-batch scheduling to minimise pipeline bubble based on observed computation and communication profiles; heterogeneous expert routing for MoE parallelism (HERMP) that assigns mixture-of-experts routing to heterogeneous GPU capacities; and adaptive gradient communication compression (AGCC) that dynamically selects gradient compression ratios based on gradient noise levels and bandwidth availability. ADGT is evaluated on training runs of 7B, 13B, 34B, and 70B parameter generative models on 32, 64, 128, and 256 GPU clusters with standard InfiniBand networking. ADGT achieves a mean model FLOP utilisation (MFU) of 58.4% (SD = 3.2%) versus 41.6% for the Megatron-LM baseline -- a 40.4% MFU improvement -- and reduces total training wall-clock time by 28.8% at 70B scale on 256 GPUs. The study contributes the ADGT specification, a Training Efficiency Index (TEI), and empirical characterisation of parallelism strategy interactions across model scales and cluster sizes.