Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis

Jiatong Shi; Shuai Guo; Tao Qian; Nan Huo; Tomoki Hayashi; Yuning Wu; Frank Xu; Xuankai Chang; Huazhe Li; Peter Wu; Shinji Watanabe; Qin Jin

doi:10.21437/Interspeech.2022-10039

Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis

Jiatong Shi, Shuai Guo, Tao Qian, Nan Huo, Tomoki Hayashi, Yuning Wu, Frank Xu, Xuankai Chang, Huazhe Li, Peter Wu, Shinji Watanabe, Qin Jin^*

^*Corresponding author for this work

Research output: Contribution to journal › Conference article › peer-review

Abstract

This paper introduces a new open-source platform named Muskits for end-to-end music processing, which mainly focuses on end-to-end singing voice synthesis (E2E-SVS). Muskits supports state-of-the-art SVS models, including RNN SVS, transformer SVS, and XiaoiceSing. The design of Muskits follows the style of widely-used speech processing toolkits, ESPnet and Kaldi, for data prepossessing, training, and recipe pipelines. To the best of our knowledge, this toolkit is the first platform that allows a fair and highly-reproducible comparison between several published works in SVS. In addition, we also demonstrate several advanced usages based on the toolkit functionalities, including multilingual training and transfer learning. This paper describes the major framework of Muskits, its functionalities, and experimental results in single-singer, multi-singer, multilingual, and transfer learning scenarios. The toolkit is publicly available at https://github.com/SJTMusicTeam/Muskits.

Original language	English
Pages (from-to)	4277-4281
Number of pages	5
Journal	Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Volume	2022-September
DOIs	https://doi.org/10.21437/Interspeech.2022-10039
Publication status	Published - 2022
Externally published	Yes
Event	23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022 - Incheon, Korea, Republic of Duration: 2022 Sep 18 → 2022 Sep 22

Keywords

end-to-end
open-source toolkit
Singing voice synthesis

ASJC Scopus subject areas

Language and Linguistics
Human-Computer Interaction
Signal Processing
Software
Modelling and Simulation

Access to Document

10.21437/Interspeech.2022-10039

Cite this

Shi, J., Guo, S., Qian, T., Huo, N., Hayashi, T., Wu, Y., Xu, F., Chang, X., Li, H., Wu, P., Watanabe, S., & Jin, Q. (2022). Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2022-September, 4277-4281. https://doi.org/10.21437/Interspeech.2022-10039

Shi, J, Guo, S, Qian, T, Huo, N, Hayashi, T, Wu, Y, Xu, F, Chang, X, Li, H, Wu, P, Watanabe, S & Jin, Q 2022, 'Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis', Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2022-September, pp. 4277-4281. https://doi.org/10.21437/Interspeech.2022-10039

@article{8324908fae2a4a24881c42e605be3030,

title = "Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis",

abstract = "This paper introduces a new open-source platform named Muskits for end-to-end music processing, which mainly focuses on end-to-end singing voice synthesis (E2E-SVS). Muskits supports state-of-the-art SVS models, including RNN SVS, transformer SVS, and XiaoiceSing. The design of Muskits follows the style of widely-used speech processing toolkits, ESPnet and Kaldi, for data prepossessing, training, and recipe pipelines. To the best of our knowledge, this toolkit is the first platform that allows a fair and highly-reproducible comparison between several published works in SVS. In addition, we also demonstrate several advanced usages based on the toolkit functionalities, including multilingual training and transfer learning. This paper describes the major framework of Muskits, its functionalities, and experimental results in single-singer, multi-singer, multilingual, and transfer learning scenarios. The toolkit is publicly available at https://github.com/SJTMusicTeam/Muskits.",

keywords = "end-to-end, open-source toolkit, Singing voice synthesis",

author = "Jiatong Shi and Shuai Guo and Tao Qian and Nan Huo and Tomoki Hayashi and Yuning Wu and Frank Xu and Xuankai Chang and Huazhe Li and Peter Wu and Shinji Watanabe and Qin Jin",

note = "Funding Information: This work was supported by the National Natural Science Foundation of China (No. 62072462) and the National Key R&D Program of China (No. 2020AAA0108600). Publisher Copyright: Copyright {\textcopyright} 2022 ISCA.; 23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022 ; Conference date: 18-09-2022 Through 22-09-2022",

year = "2022",

doi = "10.21437/Interspeech.2022-10039",

language = "English",

volume = "2022-September",

pages = "4277--4281",

journal = "Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH",

issn = "2308-457X",

}

TY - JOUR

T1 - Muskits

T2 - 23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022

AU - Shi, Jiatong

AU - Guo, Shuai

AU - Qian, Tao

AU - Huo, Nan

AU - Hayashi, Tomoki

AU - Wu, Yuning

AU - Xu, Frank

AU - Chang, Xuankai

AU - Li, Huazhe

AU - Wu, Peter

AU - Watanabe, Shinji

AU - Jin, Qin

N1 - Funding Information: This work was supported by the National Natural Science Foundation of China (No. 62072462) and the National Key R&D Program of China (No. 2020AAA0108600). Publisher Copyright: Copyright © 2022 ISCA.

PY - 2022

Y1 - 2022

N2 - This paper introduces a new open-source platform named Muskits for end-to-end music processing, which mainly focuses on end-to-end singing voice synthesis (E2E-SVS). Muskits supports state-of-the-art SVS models, including RNN SVS, transformer SVS, and XiaoiceSing. The design of Muskits follows the style of widely-used speech processing toolkits, ESPnet and Kaldi, for data prepossessing, training, and recipe pipelines. To the best of our knowledge, this toolkit is the first platform that allows a fair and highly-reproducible comparison between several published works in SVS. In addition, we also demonstrate several advanced usages based on the toolkit functionalities, including multilingual training and transfer learning. This paper describes the major framework of Muskits, its functionalities, and experimental results in single-singer, multi-singer, multilingual, and transfer learning scenarios. The toolkit is publicly available at https://github.com/SJTMusicTeam/Muskits.

AB - This paper introduces a new open-source platform named Muskits for end-to-end music processing, which mainly focuses on end-to-end singing voice synthesis (E2E-SVS). Muskits supports state-of-the-art SVS models, including RNN SVS, transformer SVS, and XiaoiceSing. The design of Muskits follows the style of widely-used speech processing toolkits, ESPnet and Kaldi, for data prepossessing, training, and recipe pipelines. To the best of our knowledge, this toolkit is the first platform that allows a fair and highly-reproducible comparison between several published works in SVS. In addition, we also demonstrate several advanced usages based on the toolkit functionalities, including multilingual training and transfer learning. This paper describes the major framework of Muskits, its functionalities, and experimental results in single-singer, multi-singer, multilingual, and transfer learning scenarios. The toolkit is publicly available at https://github.com/SJTMusicTeam/Muskits.

KW - end-to-end

KW - open-source toolkit

KW - Singing voice synthesis

UR - http://www.scopus.com/inward/record.url?scp=85140046757&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85140046757&partnerID=8YFLogxK

U2 - 10.21437/Interspeech.2022-10039

DO - 10.21437/Interspeech.2022-10039

M3 - Conference article

AN - SCOPUS:85140046757

VL - 2022-September

SP - 4277

EP - 4281

JO - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

JF - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

SN - 2308-457X

Y2 - 18 September 2022 through 22 September 2022

ER -

Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this