Cost-Aware Scheduling and Resource Optimization for Generative AI Pipelines

Abstract

Generative AI pipelines -- multi-stage workflows that chain document ingestion, embedding, retrieval, generation, and post-processing steps -- have become the dominant deployment architecture for enterprise RAG (Retrieval-Augmented Generation) applications, scientific knowledge synthesis, and complex document processing systems. The resource consumption of these pipelines is substantial and highly variable: a single pipeline execution may invoke LLM inference multiple times, trigger vector database queries, perform embedding model inference, and execute document processing operations, with total cost per pipeline execution ranging from fractions of a cent to several dollars depending on input complexity and model choices. Cost-aware scheduling -- the optimisation of when and how pipeline stages are executed based on computational cost, quality requirements, and resource availability -- is a critical but understudied aspect of generative AI operations. This paper proposes the Generative Pipeline Cost Optimisation (GPCO) framework, a systematic methodology for cost-aware scheduling and resource optimisation of generative AI pipelines, comprising five optimisation strategies: adaptive stage skipping (ASS) that bypasses expensive stages when simpler alternatives suffice; quality-cost frontier navigation (QCFN) that selects the minimum-cost pipeline configuration achieving a target quality level; asynchronous stage parallelism (ASP) that overlaps independent pipeline stages for throughput; cost-aware caching (CAC) that prioritises expensive stage results for caching based on cost-amortisation potential; and dynamic resource allocation (DRA) that right-sizes compute allocation for each pipeline stage based on input complexity. GPCO is evaluated on three enterprise RAG pipeline deployments serving 48,000 daily requests over 60 days. GPCO achieves a 54.2% pipeline execution cost reduction (SD = 6.4%) while maintaining 96.8% of baseline output quality, and a 38.4% reduction in mean pipeline latency. The study contributes the GPCO specification, a Pipeline Cost- Quality Index (PCQI), and empirical evidence that coordinated cost-aware scheduling substantially outperforms naive cost-blind pipeline execution for enterprise generative AI workloads.