Multimodal Learning for Robust Trajectory Prediction under Diverse Autonomous Driving Scenarios

Abstract

Trajectory prediction has become a central functional layer in autonomous driving systems, where vehicles must anticipate the movement of other road users under uncertainty, occlusion, interaction, and distribution shift. Multimodal learning offers a powerful route to robustness because it can combine cameras, lidar, radar, high-definition maps, vehicle telemetry, and infrastructure-side signals into shared or coordinated representations. Yet the promise of multimodal prediction cannot be reduced to model architecture. It is also a system-level problem involving data curation, sensor calibration, edge-cloud orchestration, lifecycle maintenance, fairness, governance, and sustainability. This paper examines multimodal learning for robust trajectory prediction under diverse autonomous driving scenarios through a socio-technical and infrastructural lens. It analyzes how curatorial discipline, fusion strategy, uncertainty management, and policy-aware assurance shape real-world performance. It considers structural trade-offs among latency, redundancy, coverage, cost, and equity, and it discusses how deployment choices influence robustness across urban, highway, adverse-weather, and mixed-traffic environments. The paper argues that robust prediction is achieved when multimodal architectures are embedded within accountable institutions, maintained data pipelines, resilient compute infrastructures, and transparent evaluation regimes. It concludes with a forward-looking research and governance agenda for resilient, fair, and sustainable autonomous mobility.