Scalable Architectures for Training Large Language Models under Resource Constraints

Abstract

Training large language models (LLMs) at frontier scale requires compute infrastructure accessible to only a small number of well- resourced organisations, creating a concentration of generative AI capability that has significant scientific, economic, and governance implications. Research on scalable training architectures that reduce the computational footprint of high-quality LLM training without proportionate quality degradation is therefore a high- priority direction for democratising access to frontier AI capabilities. This paper proposes the Resource-Constrained LLM Training (RCLT) framework, an integrated system of four architectural and algorithmic techniques -- sparse mixture-of-experts routing with adaptive capacity, gradient checkpointing with selective recomputation, dynamic precision scheduling across training phases, and pipeline parallelism with asynchronous micro-batch processing -- designed to reduce peak GPU memory consumption and wall-clock training time for LLMs in the 7B-70B parameter range. The RCLT framework is evaluated on three LLM training benchmarks across 7B, 13B, and 70B parameter scales on clusters of 8, 32, and 128 A100 GPUs respectively. Results demonstrate that RCLT achieves a 38.4% reduction in peak GPU memory consumption (SD = 3.1%) and a 29.7% reduction in wall-clock training time (SD = 2.8%) relative to standard dense transformer training baselines, while maintaining 97.3% of baseline model quality as measured by perplexity and downstream MMLU performance. The 7B parameter RCLT model trained on a single 8xA100 node achieves performance competitive with dense baseline models requiring 3x the compute budget. The study contributes the RCLT framework specification, open-source implementation, and an empirical characterisation of quality-efficiency trade-offs across scales.