Auto-Tuning Framework for Efficient Large Language Model Inference on Cloud Accelerators

Abstract

Large language model inference on cloud accelerators has become a central systems problem because it couples model architecture, hardware heterogeneity, serving policies, energy constraints, and organizational governance. This paper develops a system-level account of an auto-tuning framework for efficient inference in cloud environments. The framework is not treated merely as a collection of parameter search routines but as a socio-technical infrastructure that continuously observes workloads, predicts latency and cost behavior, selects execution configurations, and subjects those decisions to governance and policy constraints. The analysis begins with the architectural foundations of transformer-based inference and accelerator-oriented serving, including attention optimization, memory management, quantization, adapters, speculative decoding, and distributed inference. It then examines trade-offs among throughput, latency, accuracy, robustness, fairness, and sustainability. Particular attention is given to telemetry and machine-learning-based delay prediction as an input to tuning decisions, while avoiding the reduction of auto-tuning to prediction alone. The paper argues that efficient inference requires co-design across accelerator scheduling, model compression, request routing, multi-tenant isolation, observability, and carbon-aware deployment. It also considers deployment risks such as performance instability, unfair resource allocation, vendor lock-in, and hidden environmental costs. Case illustrations from inference serving systems, adaptive model serving, and distributed transformer serving are used to show how auto-tuning frameworks can be evaluated beyond benchmark accuracy. The conclusion proposes that future auto-tuning infrastructures should be auditable, accountable, and adaptive, with explicit metrics for robustness, fairness, energy intensity, and operational resilience rather than latency alone.