| --- |
| license: apache-2.0 |
| --- |
| <div align="center"> |
| <h1>FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications</h1> |
| </div> |
|
|
| #### 👉🏻 [FireRedTTS Paper](https://arxiv.org/abs/2409.03283) 👈🏻 |
|
|
| #### 👉🏻 [FireRedTTS Demos](https://fireredteam.github.io/demos/firered_tts/) 👈🏻 |
|
|
| #### 👉🏻 [FireRedTTS Space (Interactive Demo)](https://huggingface.co/spaces/fireredteam/FireRedTTS) 👈🏻 |
|
|
| ## News |
|
|
| - [2024/10/17] 🔥 We release [new rich-punctuation model](https://huggingface.co/fireredteam/FireRedTTS/tree/main), offering expanded punctuation coverage and enhanced audio production consistency. In addition, we have strengthened the capabilities of the text front-end and enhanced the stability of synthesis. |
| - [2024/09/26] 🔥 Our model is already available on [huggingface space](https://huggingface.co/spaces/fireredteam/FireRedTTS)，try it through the interactive interface. |
| - [2024/09/20] 🔥 We release the pre-trained checkpoints and inference code. |
| - [2024/09/06] 🔥 We release the [technical report](https://arxiv.org/abs/2409.03283) and [project page](https://fireredteam.github.io/demos/firered_tts/) |
|
|
| ## Roadmap |
|
|
| - [ ] 2024/09 |
| - [x] Release the pre-trained checkpoints and inference code. |
| - [ ] Release testing set. |
|
|
| - [ ] 2024/10 |
| - [x] Release rich punctuation version. |
| - [ ] Release finetuned checkpoints for controllable human-like speech generation. |
|
|
| ## Usage |
|
|
| #### Clone and install |
|
|
| - Clone the repo |
|
|
| ```shell |
| https://github.com/FireRedTeam/FireRedTTS.git |
| cd FireRedTTS |
| ``` |
|
|
| - Create conda env |
|
|
| ```shell |
| # step1.create env |
| conda create --name redtts python=3.10 |
| |
| # stpe2.install torch （pytorch should match the cuda-version on your machine） |
| # CUDA 11.8 |
| conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia |
| # CUDA 12.1 |
| conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia |
| |
| # step3.install fireredtts form source |
| pip install -e . |
| |
| # step4.install other requirements |
| pip install -r requirements.txt |
| ``` |
|
|
| #### Download models |
|
|
| Download the required model files from [**Model_Lists**](https://huggingface.co/fireredteam/FireRedTTS/tree/main) and place them in the folder `pretrained_models` |
|
|
| #### Basic Usage |
|
|
| ```python |
| import os |
| import torchaudio |
| from fireredtts.fireredtts import FireRedTTS |
| |
| tts = FireRedTTS( |
| config_path="configs/config_24k.json", |
| pretrained_path=<pretrained_models_dir>, |
| ) |
| |
| #same language |
| rec_wavs = tts.synthesize( |
| prompt_wav="examples/prompt_1.wav", |
| text="小红书，是中国大陆的网络购物和社交平台，成立于二零一三年六月。", |
| lang="zh", |
| ) |
| |
| rec_wavs = rec_wavs.detach().cpu() |
| out_wav_path = os.path.join("./example.wav") |
| torchaudio.save(out_wav_path, rec_wavs, 24000) |
| |
| ``` |
|
|
| ## Tips |
|
|
| - Removing the long silence (>1s) in the middle of prompt_wav may bring better stability. If there are too many long silences in your prompt_wav and it causes stability problems, it is recommended to use our tool(`tools/process_prompts.py`) to remove the silence. |
|
|
| ## Acknowledgements |
|
|
| - [**Tortoise-tts**](https://github.com/neonbjb/tortoise-tts) and [**XTTS-v2**](https://github.com/coqui-ai/TTS) offer invaluable insights for constructing an autoregressive-style system. |
| - [**Matcha-TTS**](https://github.com/shivammehta25/Matcha-TTS) and [**CosyVoice**](https://github.com/FunAudioLLM/CosyVoice) demonstrate the excellent ability of flow-matching in converting audio code to mel. |
| |
| - [**BigVGAN-v2**](https://github.com/NVIDIA/BigVGAN), utilized for vocoding. |
| - We referred to [**whisper’s**](https://github.com/openai/whisper) text tokenizer solution. |
