AI-Based Congestion Prediction and Traffic Engineering for High-Performance Computing Interconnects
Keywords:
high-performance computing, interconnect congestion, traffic engineering, congestion prediction, machine learning, data center networks, socio-technical systems, sustainabilityAbstract
High-performance computing interconnects have become critical substrates for artificial intelligence training, scientific simulation, and data-intensive service workloads. As these systems scale, congestion ceases to be a marginal performance issue and becomes a structural challenge that couples network architecture, workload behavior, scheduling policy, energy use, and organizational governance. This paper examines artificial intelligence based congestion prediction and traffic engineering for high-performance computing interconnects from a system-level perspective. It argues that prediction models cannot be understood merely as algorithmic improvements; they are embedded in instrumentation regimes, control loops, topology choices, and operational policies that determine whether congestion management is effective, fair, and sustainable. The discussion first situates interconnect congestion within the evolution of data center and supercomputing fabrics, then analyzes workload characteristics that produce burstiness, incast, and tail latency. It subsequently evaluates how machine learning can be positioned across endpoints, switches, and centralized controllers, and how predictive signaling can inform routing, scheduling, and flow control. The paper also considers deployment trade-offs involving telemetry overhead, model drift, multi-tenant fairness, robustness under failure, energy consumption, and policy constraints. Case illustrations from cloud data centers, wide-area traffic engineering, and large-scale scientific computing are used to compare governance models and operational incentives. The conclusion emphasizes that AI-based congestion prediction is most credible when treated as a socio-technical capability rather than a standalone model, requiring observability, accountability, fallback mechanisms, and careful alignment with the physical and institutional limits of high-performance computing infrastructure.
References
1. Al-Fares, M., Loukissas, A., & Vahdat, A. (2008). A scalable, commodity data center network architecture. ACM SIGCOMM Computer Communication Review, 38(4), 63-74. https://doi.org/10.1145/1402946.1402967
2. Benson, T., Akella, A., & Maltz, D. A. (2010). Network traffic characteristics of data centers in the wild. Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, 267-280. https://doi.org/10.1145/1879141.1879175
3. Kandula, S., Sengupta, S., Greenberg, A., Patel, P., & Chaiken, R. (2009). The nature of data center traffic: Measurements and analysis. Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, 202-208. https://doi.org/10.1145/1644893.1644918
4. Greenberg, A., Hamilton, J., Maltz, D. A., & Patel, P. (2008). The cost of a cloud: Research problems in data center networks. ACM SIGCOMM Computer Communication Review, 39(1), 68-73. https://doi.org/10.1145/1496091.1496103
5. Farrington, N., Porter, G., Radhakrishnan, S., Bazzaz, H. H., Subramanya, V., Fainman, Y., Papen, G., & Vahdat, A. (2010). Helios: A hybrid electrical/optical switch architecture for modular data centers. ACM SIGCOMM Computer Communication Review, 40(4), 339-350. https://doi.org/10.1145/1851182.1851223
6. Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., & Sridharan, M. (2010). Data center TCP (DCTCP). ACM SIGCOMM Computer Communication Review, 40(4), 63-74. https://doi.org/10.1145/1851182.1851192
7. Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M. H., & Zhang, M. (2015). Congestion control for large-scale RDMA deployments. ACM SIGCOMM Computer Communication Review, 45(4), 523-536. https://doi.org/10.1145/2785956.2787484
8. Mittal, R., Shpiner, A., Panda, A., Zahavi, E., Krishnamurthy, A., Ratnasamy, S., & Shenker, S. (2015). TIMELY: RTT-based congestion control for the datacenter. ACM SIGCOMM Computer Communication Review, 45(4), 537-550. https://doi.org/10.1145/2785956.2787510
9. Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74-80. https://doi.org/10.1145/2408776.2408794
10. Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., ... & Yoon, D. H. (2017). In-datacenter performance analysis of a tensor processing unit. ACM SIGARCH Computer Architecture News, 45(2), 1-12. https://doi.org/10.1145/3079856.3080246
11. Reed, D. A., & Dongarra, J. (2015). Exascale computing and big data. Communications of the ACM, 58(7), 56-68. https://doi.org/10.1145/2699414
12. Ben-Nun, T., & Hoefler, T. (2019). Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys, 52(4), 1-43. https://doi.org/10.1145/3320060
13. Kurth, T., Treichler, S., Romero, J., Mudigonda, M., Luehr, N., Phillips, E., Mahesh, A., Mattingly, M., Podhorszki, N., Ruiter, J., & Houston, M. (2018). Exascale deep learning for climate analytics. SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, 649-660. https://doi.org/10.1109/SC.2018.00054
14. Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018). A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities. Journal of Internet Services and Applications, 9(1), 1-99. https://doi.org/10.1186/s13174-018-0087-2
15. Wang, M., Cui, Y., Wang, X., Xiao, S., & Jiang, J. (2018). Machine learning for networking: Workflow, advances and opportunities. IEEE Network, 32(2), 92-99. https://doi.org/10.1109/MNET.2017.1700200
16. Fadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017). State-of-the-art deep learning: Evolving machine intelligence toward tomorrow's intelligent network traffic control systems. IEEE Communications Surveys & Tutorials, 19(4), 2432-2455. https://doi.org/10.1109/COMST.2017.2707140
17. Zhou, Zichuan, et al. "Low-latency wavelength-switched clock-synchronized intra-data center interconnects enabled by hollow core fiber." Optics Express 31.15 (2023): 24739-24748.
18. Zhang, C., Patras, P., & Haddadi, H. (2019). Deep learning in mobile and wireless networking: A survey. IEEE Communications Surveys & Tutorials, 21(3), 2224-2287. https://doi.org/10.1109/COMST.2019.2904897
19. Hoefler, T., & Belli, R. (2015). Scientific benchmarking of parallel computing systems: Twelve ways to tell the masses when reporting performance results. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1-12. https://doi.org/10.1145/2807591.2807644
20. Kreutz, D., Ramos, F. M. V., Veríssimo, P. E., Rothenberg, C. E., Azodolmolky, S., & Uhlig, S. (2015). Software-defined networking: A comprehensive survey. Proceedings of the IEEE, 103(1), 14-76. https://doi.org/10.1109/JPROC.2014.2371999
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Computer Science and Engineering Transactions
This work is licensed under a Creative Commons Attribution 4.0 International License.