Microsoft announces 'NaturalSpeech 2,' a speech synthesis AI that can reproduce conversations and singing voices from just a few seconds of sample data
On April 18, 2023, a research team led by Kai Shen of
[2304.09116] NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
https://doi.org/10.48550/arXiv.2304.09116
NaturalSpeech 2
While existing TTS systems have achieved high speech quality with datasets recorded by a single speaker, these datasets are unable to fully capture diverse human identities, accents, and styles. Furthermore, when scaling to large-scale, multi-speaker datasets, current TTS systems typically quantize speech into individual tokens and generate each token individually using a language model, resulting in unstable prosody, word skipping/repeating, and poor speech quality.
However, 'NaturalSpeech 2,' developed by Shen and his research team, uses latent diffusion models to create a text-to-speech model that is highly expressive and reproducible, faithfully reproducing the sample voice.
NaturalSpeech 2 uses a neural network-based
A paper published by Shen and his research team presents an example of speech synthesis using NaturalSpeech 2. The following shows the process of reading aloud
The 'Ground Truth' audio is a text read by the same person as the sample audio, and this is the target 'correct audio.'
The 'Baseline' below is a synthetic voice created using a conventional AI model. It sounds monotonous, somewhat unnatural, and mechanical, but it produces a voice that is close to human speech.
Below is the audio output using NaturalSpeech 2. You can see that the audio output is comparable to a 'human voice' in terms of breathing and accents.
Below is an example of training using ' VCTK ' by a research team at the University of Edinburgh. To generate the speech 'We will turn the corner,' first input an unrelated speech.
Below is the audio for 'Ground Truth.'
The audio output for 'Baseline' is as follows.
Below is the audio of 'Turning the corner' output using NaturalSpeech 2.
The research team also compared their model with Microsoft's VALL-E speech synthesis AI model, which outputs the following sentence: 'Thus did this humane and right-minded father comfort his unhappy daughter, and her mother embracing her again, did all she could to soothe her feelings.'
Below is the audio output using NaturalSpeech 2.
NaturalSpeech 2 also allows you to input and output singing voices. Below is an example of singing the lyrics 'So listen very carefully.' First, input an unrelated sentence.
This will output a singing voice with intonation and rhythm.
NaturalSpeech 2 also supports input from singing voice. In the example below, the same ' BINGO ' is sung.
In the example below, even if you input vocals, the output will be vocals as well.
The Microsoft research team warns that 'NaturalSpeech 2 is capable of faithful representation, which poses a risk of misuse, such as speaker imitation and impersonation.' To avoid these ethical and potential problems, the research team argues, 'We urge developers not to misuse this technology and to develop countermeasures to detect AI-synthesized speech.' Furthermore, they state, 'When developing these AI models, we always adhere to Microsoft's Core Principles for Responsible AI .'
The source code for NaturalSpeech 2 is available on GitHub.
GitHub - lucidrains/naturalspeech2-pytorch: Implementation of Natural Speech 2, Zero-shot Speech and Singing Synthesizer, in Pytorch
https://github.com/lucidrains/naturalspeech2-pytorch
Related Posts:
