A unified accent estimation method based on multi-task learning for Japanese text-to-speech

Accepted to Interspeech 2022

Authors

• Byeongseon Park
• Ryuichi Yamamoto
• Kentaro Tachibana

Abstract

We propose a unified accent estimation method for Japanese text-to-speech (TTS). Unlike the conventional two-stage methods, which separately train two models for predicting accent phrase boundaries and accent nucleus positions, our method merges the two models and jointly optimizes the entire model in a multi-task learning framework. Furthermore, considering the hierarchical linguistic structure of intonation phrases (IPs), accent phrases, and accent nuclei, we generalize the proposed approach to simultaneously model the IP boundaries with accent information. Objective evaluation results reveal that the proposed method achieves an accent estimation accuracy of 80.4%, which is 6.67% higher than the conventional two-stage method. When the proposed method is incorporated into a neural TTS framework, the system achieves a 4.29 mean opinion score with respect to prosody naturalness.

Demo

TTS setup

• Acoustic model : FastSpeech2 [1]
• Vocoder : Parallel WaveGAN [2]

The detailed model structure and training conditions of these two models were the same as those in [3].

Target tasks

• IPs : Intonation phrases
• APs : Accent phrases
• ANs : Accent nuclei

Systems used for comparision

Models

Systems

Audio samples (Japanese)

States used for comparision

• Recording : Recorded speech.
• X: The speech synthesised using wrong IP, AP, and AN information.
• IP: The speech synthesised using correct IP information, with wrong AP and AN.
• AP: The speech synthesised using correct AP information, with wrong IP and AN.
• AN: The speech synthesised using correct AN information, with wrong IP and AP.
• IP+AP: The speech synthesised using correct IP and AP information, with wrong AN.
• IP+AN: The speech synthesised using correct IP and AN information, with wrong AP.
• AP+AN: The speech synthesised using correct AP and AN information, with wrong IP.
• IP+AP+AN: The speech synthesised using correct IP, AP, and AN information.

Audio samples (Japanese)

References

• [1]: Y. Ren, C. Hu, X. Tan, S. Zhao, Z. Zhao, and T.Y. Liu, “FastSpeech 2: Fast and high-quality end-to-end text-to-speech,” in Proc. ICLR, 2021 (arXiv).
• [2]: R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in Proc. ICASSP, 2020, pp. 6199–6203 (arXiv).
• [3]: R. Yamamoto, E. Song, M.-J. Hwang, and J.-M. Kim, “Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discriminators,” in Proc. ICASSP, 2021, pp. 6039–6043 (arXiv).
• [4]: M. Suzuki, R. Kuroiwa, K. Innami, S. Kobayashi, S. Shimizu, N. Minematsu, and K. Hirose, “Accent sandhi estimation of Tokyo dialect of Japanese using conditional random fields,” IEICE Trans., vol. E100-D, no. 4, pp. 655–661, 2017 (IEICE).

Acknowledgements

This work was supported by Clova Voice, NAVER Corp., Seongnam, Korea. The authors would like to thank Yuma Shirahata and Kosuke Futamata at LINE Corp., Tokyo, Japan, for their support.