Machine Learning-Based Performance Modeling for Efficient Large Language Model Serving

Abstract

The rapid proliferation of large language models has created unprecedented demands on computing infrastructure, necessitating sophisticated performance modeling techniques to ensure efficient serving. This paper examines the role of machine learning-based performance modeling in optimizing large language model serving systems. We analyze the architectural trade-offs inherent in serving these models, including latency, throughput, memory management, and energy consumption. The paper discusses how supervised and reinforcement learning approaches can predict inference delays, resource utilization, and quality-of-service metrics, enabling dynamic scheduling and resource allocation. We explore data requirements, feature engineering, and model selection challenges, emphasizing the need for robust, generalizable models that can adapt to heterogeneous hardware and evolving workloads. The discussion extends to deployment considerations, including edge-cloud continuum, sustainability, and governance frameworks. We highlight the importance of fairness, transparency, and policy interventions to mitigate risks of algorithmic bias and environmental impact. Through case illustrations and cross-domain comparisons, we demonstrate that machine learning-based performance modeling is not merely a technical optimization but a socio-technical infrastructure requiring careful governance. The paper concludes with forward-looking perspectives on automated orchestration, federated learning for performance prediction, and the need for standardized benchmarks and auditing mechanisms.