Personalized Singing Voice Generation via Singing Style Transfer and Voice Cloning
Keywords:
personalized singing voice generation; singing style transfer; voice cloning; neural vocoding; self-supervised audio representation; data governance; fairness; deploymentAbstract
Personalized singing voice generation sits at the intersection of speech synthesis, music information retrieval, biometric identity modeling, and generative artificial intelligence. The task requires a system to synthesize novel vocal performances that preserve a target singer's timbral identity while transferring expressive attributes such as vibrato, dynamics, ornamentation, and rhythmic phrasing from a reference performance or score. This paper examines the problem as an infrastructure-level challenge rather than a narrowly defined modeling exercise. It discusses the architectural trade-offs between end-to-end and modular pipelines; the role of self-supervised, discrete, and spectral representations; the design of large-scale singing corpora; training strategies for disentanglement, robustness, and cross-lingual generalization; and the sociotechnical obligations that arise when voice cloning becomes widely deployable. The analysis draws on advances in neural text-to-speech, voice conversion, neural vocoding, music generation, and biometric privacy. It argues that sustainable progress requires coordinated attention to model architecture, data governance, evaluation beyond acoustic fidelity, identity protection, equity across languages and genres, and deployment efficiency. The paper closes with a forward-looking research agenda centered on privacy-preserving personalization, transparent provenance, and robust evaluation. Throughout the discussion, personalized singing voice generation is treated as a coupled sociotechnical system in which technical design and institutional safeguards must co-evolve.
References
1. Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., ... & Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135.
2. Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., ... & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.
3. Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33, 17022–17033.
4. Nagrani, A., Chung, J. S., & Zisserman, A. (2017). VoxCeleb: A large-scale speaker identification dataset. In Proceedings of Interspeech 2017 (pp. 2616–2620).
5. Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., ... & Wu, Y. (2018). Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in Neural Information Processing Systems, 31, 4485–4495.
6. Casanova, E., Weber, J., Shulby, C., Junior, A. C., Gölge, E., Ponti, M. A., ... & Barbosa, L. (2022). YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Proceedings of the 39th International Conference on Machine Learning (pp. 2709–2720). PMLR.
7. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.
8. Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451–3460.
9. Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., ... & Zeghidour, N. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
10. Qian, K., Zhang, Y., Chang, S., Yang, X., & Hasegawa-Johnson, M. (2019). AutoVC: Zero-shot voice style transfer with only autoencoder loss. In Proceedings of the 36th International Conference on Machine Learning (pp. 5210–5219). PMLR.
11. Qian, K., Zhang, Y., Chang, S., Cox, D., & Hasegawa-Johnson, M. (2020). Unsupervised speech decomposition via triple information bottleneck. In Proceedings of the 37th International Conference on Machine Learning (pp. 7836–7846). PMLR.
12. Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., ... & Dupoux, E. (2021). Speech resynthesis from discrete disentangled self-supervised representations. In Proceedings of Interspeech 2021 (pp. 3615–3619).
13. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2019). FastSpeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, 32, 3171–3180.
14. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., & Tagliasacchi, M. (2021). SoundStream: An end-to-end neural audio codec. arXiv preprint arXiv:2107.03312.
15. Zhang, Y., Pan, C., Guo, W., Li, R., Zhu, Z., Wang, J., ... & Zhao, Z. (2024). Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks. Advances in Neural Information Processing Systems, 37, 1117-1140.
16. Liu, J., Li, C., Ren, Y., Chen, F., & Zhao, Z. (2022). DiffSinger: Singing voice synthesis via shallow diffusion mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11020–11028.
17. Dhariwal, P., Jun, H., Payne, C., Kim, J., Radford, A., & Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341.
18. Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., ... & Zeghidour, N. (2023). AudioLM: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 2523–2533.
19. Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., ... & Adi, Y. (2023). Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 47704–47720.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Computer Science and Engineering Transactions
This work is licensed under a Creative Commons Attribution 4.0 International License.