Joint Singing Voice–Music Score Representation Learning for Multimodal Music Foundation Models
Keywords:
multimodal representation learning; music foundation models; singing voice; symbolic music scores; audio-symbolic alignment; robustness; governance; sustainabilityAbstract
Recent advances in foundation models have transformed multimodal learning by aligning heterogeneous data sources in shared representational spaces. In music, however, the alignment of singing voice audio with symbolic music scores remains an open systems problem. This paper examines joint singing voice and music score representation learning for multimodal music foundation models from an interdisciplinary systems perspective. It argues that robust audio-symbolic alignment cannot be reduced to a single modeling objective but must be understood as a socio-technical infrastructure comprising architectural choices, data governance, training strategies, evaluation regimes, deployment constraints, and policy instruments. The discussion begins by situating joint representation learning within broader foundation model research and music information retrieval. It then analyzes architectural trade-offs between two-tower contrastive encoders, cross-attention fusion, and sequence-to-sequence decoding. Attention is given to the structural challenges of temporal alignment, score sparsity, phoneme dependence, and singing style variability. The paper further examines data infrastructure, including corpus construction, licensing, provenance, and consent, with a brief reference to a recently released global multi-technique singing corpus. Training and evaluation are considered in relation to contrastive objectives, curriculum design, perceptual metrics, and robustness. Fairness, interpretability, and accountability are treated as first-class system requirements rather than post hoc audits. Finally, the paper discusses deployment sustainability, computational cost, and policy implications for music foundation models. The overall contribution is a conceptual and architectural framework for building multimodal music systems that are not only accurate but also governable, fair, and sustainable.
References
1. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., ... & Liang, P. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
2. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, 8748-8763.
3. Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., & Frank, C. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
4. Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., & Defossez, A. (2024). Simple and controllable music generation. Advances in Neural Information Processing Systems, 36.
5. Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2018). Music Transformer: Generating music with long-term structure. arXiv preprint arXiv:1809.04281.
6. Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., & Eck, D. (2019). Enabling factorized piano music modeling with the MAESTRO dataset. International Conference on Learning Representations.
7. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460.
8. McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., & Nieto, O. (2015). librosa: Audio and music signal analysis in Python. Proceedings of the 14th Python in Science Conference, 18-25.
9. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
10. Raffel, C., McFee, B., Humphrey, E. J., Salamon, J., Nieto, O., Liang, D., & Ellis, D. P. W. (2014). mir_eval: A transparent implementation of common MIR metrics. Proceedings of the 15th International Society for Music Information Retrieval Conference, 367-372.
11. Oore, S., Simon, I., Dieleman, S., Eck, D., & Simonyan, K. (2020). This time with feeling: Learning expressive musical performance. Neural Computing and Applications, 32(4), 955-967.
12. Donahue, C., Mao, H. H., Li, Y. E., Cottrell, G. W., & McAuley, J. (2019). LakhNES: Improving multi-instrumental music generation with cross-domain pre-training. Proceedings of the 20th International Society for Music Information Retrieval Conference, 685-692.
13. Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017). Convolutional recurrent neural networks for music classification. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, 2392-2396.
14. Bittner, R. M., Salamon, J., Tierney, M., Mauch, M., Cannam, C., & Bello, J. P. (2017). Deep salience representations for f0 estimation in polyphonic music. Proceedings of the 18th International Society for Music Information Retrieval Conference, 63-70.
15. Simonetta, F., Ntalampiras, S., & Avanzini, F. (2019). A perceptual measure for evaluating the resynthesis of automatic music transcriptions. Multimedia Tools and Applications, 78(19), 27745-27764.
16. Sturm, B. L., Ben-Tal, O., Monaghan, U., Collins, N., Herremans, D., Chew, E., Hadjeres, G., Deruty, E., & Pachet, F. (2019). Machine learning research that matters for music creation: A case study. Journal of New Music Research, 48(1), 36-55.
17. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623.
18. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229.
19. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, 618-626.
20. Lhoest, Q., Villanova del Moral, A., Jernite, Y., Ramesh, A., et al. (2021). Datasets: A community library for natural language processing. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 175-184.
21. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021). Zero-shot text-to-image generation. Proceedings of the 38th International Conference on Machine Learning, 8821-8831.
22. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021). Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350.
23. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., & Dubnov, S. (2023). Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing.
24. Hawthorne, C., Simon, I., Roberts, A., Zeghidour, N., Gardner, J., Manilow, E., & Engel, J. (2022). Multi-instrument music synthesis with spectrogram diffusion. Proceedings of the 23rd International Society for Music Information Retrieval Conference, 598-606.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Computer Science and Engineering Transactions
This work is licensed under a Creative Commons Attribution 4.0 International License.