Transformer-Based Singing Voice Separation and Pitch Estimation in Complex Musical Scenes

Abstract

Singing voice separation and pitch estimation are central tasks in music information retrieval, yet they become particularly difficult in complex musical scenes where vocals overlap with dense instrumentation, reverberation, expressive pitch variation, and heterogeneous recording conditions. Recent transformer-based architectures have produced notable gains in both areas by modeling long-range dependencies more effectively than earlier convolutional or recurrent systems. This paper provides a system-level examination of transformer-based singing voice separation and pitch estimation, focusing on architectural trade-offs, deployment infrastructure, robustness, fairness, sustainability, and policy implications. Rather than presenting a single model, the paper analyzes how separation and pitch estimation systems interact within larger sociotechnical infrastructures, including streaming platforms, music education tools, archival workflows, and assistive applications. It discusses the shift from spectrogram masking to hybrid waveform and time-frequency modeling, the role of attention in capturing harmonic and temporal structure, the challenges of maintaining pitch continuity in polyphonic contexts, and the consequences of dataset design for model behavior. Particular attention is given to system-level concerns such as latency, computational cost, energy consumption, model versioning, data provenance, and evaluation across culturally diverse singing traditions. The paper argues that robust deployment requires not only improved architectures but also stronger governance mechanisms, transparent evaluation protocols, and careful consideration of fairness and consent. By connecting technical design choices to broader infrastructural and policy questions, the paper offers an interdisciplinary perspective on next-generation music analysis systems.