Spaces:
Running
Running
A Le Thanh Son
commited on
Commit
·
e02c9de
1
Parent(s):
a6403d5
fix
Browse files- README.md +39 -39
- app.py +195 -139
- generator.py +10 -10
- test_model.py +22 -22
README.md
CHANGED
@@ -11,35 +11,35 @@ pinned: false
|
|
11 |
|
12 |
# CSM-1B Text-to-Speech Demo
|
13 |
|
14 |
-
|
15 |
|
16 |
-
##
|
17 |
|
18 |
-
- **
|
19 |
-
- **
|
20 |
-
- **
|
21 |
|
22 |
-
##
|
23 |
|
24 |
-
###
|
25 |
|
26 |
-
|
27 |
|
28 |
- [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B)
|
29 |
- [sesame/csm-1b](https://huggingface.co/sesame/csm-1b)
|
30 |
|
31 |
-
###
|
32 |
|
33 |
-
1.
|
34 |
-
2.
|
35 |
-
3.
|
36 |
-
4.
|
37 |
```bash
|
38 |
export HF_TOKEN=your_token_here
|
39 |
```
|
40 |
-
5.
|
41 |
|
42 |
-
###
|
43 |
|
44 |
```bash
|
45 |
git clone https://github.com/yourusername/csm-1b-gradio.git
|
@@ -47,54 +47,54 @@ cd csm-1b-gradio
|
|
47 |
pip install -r requirements.txt
|
48 |
```
|
49 |
|
50 |
-
##
|
51 |
|
52 |
-
1.
|
53 |
```bash
|
54 |
python app.py
|
55 |
```
|
56 |
-
2.
|
57 |
-
3.
|
58 |
-
4.
|
59 |
-
5.
|
60 |
-
6.
|
61 |
|
62 |
-
##
|
63 |
|
64 |
-
CSM-1B
|
65 |
|
66 |
## ZeroGPU
|
67 |
|
68 |
-
|
69 |
|
70 |
```python
|
71 |
import spaces
|
72 |
|
73 |
@spaces.GPU
|
74 |
def my_gpu_function():
|
75 |
-
#
|
76 |
-
#
|
77 |
pass
|
78 |
```
|
79 |
|
80 |
-
|
81 |
|
82 |
-
##
|
83 |
|
84 |
-
-
|
85 |
-
-
|
86 |
-
-
|
87 |
|
88 |
-
##
|
89 |
|
90 |
-
|
91 |
|
92 |
-
1.
|
93 |
-
2.
|
94 |
-
3.
|
95 |
-
4.
|
96 |
|
97 |
-
##
|
98 |
|
99 |
- [GitHub Repository](https://github.com/SesameAILabs/csm-1b)
|
100 |
- [Hugging Face Model](https://huggingface.co/sesame/csm-1b)
|
|
|
11 |
|
12 |
# CSM-1B Text-to-Speech Demo
|
13 |
|
14 |
+
This application uses the CSM-1B (Collaborative Speech Model) to convert text to high-quality speech.
|
15 |
|
16 |
+
## Features
|
17 |
|
18 |
+
- **Simple Audio Generation**: Convert text to speech with options for speaker ID, duration, temperature, and top-k.
|
19 |
+
- **Audio Generation with Context**: Provide audio clips and text as context to help the model generate more appropriate speech.
|
20 |
+
- **GPU Optimization**: Uses Hugging Face Spaces' ZeroGPU to optimize GPU usage.
|
21 |
|
22 |
+
## Installation and Configuration
|
23 |
|
24 |
+
### Access Requirements
|
25 |
|
26 |
+
To use the CSM-1B model, you need access to the following models on Hugging Face:
|
27 |
|
28 |
- [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B)
|
29 |
- [sesame/csm-1b](https://huggingface.co/sesame/csm-1b)
|
30 |
|
31 |
+
### Hugging Face Token Configuration
|
32 |
|
33 |
+
1. Create a Hugging Face account if you don't have one.
|
34 |
+
2. Go to [Hugging Face Settings](https://huggingface.co/settings/tokens) to create a token.
|
35 |
+
3. Request access to the models if needed.
|
36 |
+
4. Set the `HF_TOKEN` environment variable with your token:
|
37 |
```bash
|
38 |
export HF_TOKEN=your_token_here
|
39 |
```
|
40 |
+
5. Or you can enter your token directly in the "Configuration" tab of the application.
|
41 |
|
42 |
+
### Installation
|
43 |
|
44 |
```bash
|
45 |
git clone https://github.com/yourusername/csm-1b-gradio.git
|
|
|
47 |
pip install -r requirements.txt
|
48 |
```
|
49 |
|
50 |
+
## How to Use
|
51 |
|
52 |
+
1. Start the application:
|
53 |
```bash
|
54 |
python app.py
|
55 |
```
|
56 |
+
2. Open a web browser and go to the displayed address (usually http://127.0.0.1:7860).
|
57 |
+
3. Enter the text you want to convert to speech.
|
58 |
+
4. Choose a speaker ID (from 0-10).
|
59 |
+
5. Adjust parameters like maximum duration, temperature, and top-k.
|
60 |
+
6. Click the "Generate Audio" button to create speech.
|
61 |
|
62 |
+
## About the Model
|
63 |
|
64 |
+
CSM-1B is an advanced text-to-speech model developed by Sesame AI Labs. This model can generate natural speech from text with various voices.
|
65 |
|
66 |
## ZeroGPU
|
67 |
|
68 |
+
This application uses Hugging Face Spaces' ZeroGPU to optimize GPU usage. ZeroGPU helps free up GPU memory when not in use, saving resources and improving performance.
|
69 |
|
70 |
```python
|
71 |
import spaces
|
72 |
|
73 |
@spaces.GPU
|
74 |
def my_gpu_function():
|
75 |
+
# This function will only use GPU when called
|
76 |
+
# and release GPU after completion
|
77 |
pass
|
78 |
```
|
79 |
|
80 |
+
When deployed on Hugging Face Spaces, ZeroGPU will automatically manage GPU usage, making the application more efficient.
|
81 |
|
82 |
+
## Notes
|
83 |
|
84 |
+
- This model uses watermarking to mark audio generated by AI.
|
85 |
+
- Audio generation time depends on text length and hardware configuration.
|
86 |
+
- You need access to the CSM-1B model on Hugging Face to use this application.
|
87 |
|
88 |
+
## Deployment on Hugging Face Spaces
|
89 |
|
90 |
+
To deploy this application on Hugging Face Spaces:
|
91 |
|
92 |
+
1. Create a new Space on Hugging Face with Gradio SDK.
|
93 |
+
2. Upload all project files.
|
94 |
+
3. In the Space settings, add the `HF_TOKEN` environment variable with your token.
|
95 |
+
4. Choose appropriate hardware configuration (GPU recommended).
|
96 |
|
97 |
+
## Resources
|
98 |
|
99 |
- [GitHub Repository](https://github.com/SesameAILabs/csm-1b)
|
100 |
- [Hugging Face Model](https://huggingface.co/sesame/csm-1b)
|
app.py
CHANGED
@@ -11,64 +11,82 @@ from dataclasses import dataclass
|
|
11 |
from generator import Segment, load_csm_1b
|
12 |
from huggingface_hub import login
|
13 |
|
14 |
-
#
|
15 |
torch._dynamo.config.suppress_errors = True
|
16 |
|
17 |
-
#
|
18 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
19 |
-
print(f"
|
20 |
|
21 |
-
#
|
22 |
def login_huggingface():
|
23 |
hf_token = os.environ.get("HF_TOKEN")
|
24 |
if hf_token:
|
25 |
-
print("
|
26 |
login(token=hf_token)
|
27 |
-
print("
|
28 |
else:
|
29 |
-
print("
|
30 |
|
31 |
-
#
|
32 |
login_huggingface()
|
33 |
|
34 |
-
#
|
35 |
generator = None
|
36 |
model_loaded = False
|
37 |
|
38 |
-
#
|
39 |
-
@spaces.GPU
|
40 |
def initialize_model():
|
41 |
global generator, model_loaded
|
42 |
if not model_loaded:
|
43 |
-
print("
|
44 |
generator = load_csm_1b(device="cuda")
|
45 |
model_loaded = True
|
46 |
-
print("
|
47 |
return generator
|
48 |
|
49 |
-
#
|
50 |
-
@spaces.GPU
|
51 |
def get_model():
|
52 |
global generator, model_loaded
|
53 |
if not model_loaded:
|
54 |
return initialize_model()
|
55 |
return generator
|
56 |
|
57 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
58 |
def audio_to_tensor(audio_path: str) -> Tuple[torch.Tensor, int]:
|
59 |
waveform, sample_rate = torchaudio.load(audio_path)
|
60 |
-
waveform = waveform.mean(dim=0) #
|
61 |
return waveform, sample_rate
|
62 |
|
63 |
-
#
|
64 |
def save_audio(audio_tensor: torch.Tensor, sample_rate: int) -> str:
|
65 |
temp_dir = tempfile.gettempdir()
|
66 |
output_path = os.path.join(temp_dir, f"csm1b_output_{int(time.time())}.wav")
|
67 |
torchaudio.save(output_path, audio_tensor.unsqueeze(0), sample_rate)
|
68 |
return output_path
|
69 |
|
70 |
-
#
|
71 |
-
@spaces.GPU
|
72 |
def generate_speech(
|
73 |
text: str,
|
74 |
speaker_id: int,
|
@@ -83,49 +101,62 @@ def generate_speech(
|
|
83 |
top_k: int = 50,
|
84 |
progress=gr.Progress()
|
85 |
) -> str:
|
86 |
-
|
87 |
-
|
88 |
-
|
89 |
-
|
90 |
-
|
91 |
-
|
92 |
-
|
93 |
-
|
94 |
-
|
95 |
-
|
96 |
-
|
97 |
-
|
98 |
-
|
99 |
-
|
100 |
-
|
101 |
-
|
102 |
-
|
103 |
-
|
104 |
-
|
105 |
-
|
106 |
-
|
107 |
-
|
108 |
-
|
109 |
-
|
110 |
-
|
111 |
-
|
112 |
-
|
113 |
-
|
114 |
-
|
115 |
-
|
116 |
-
|
117 |
-
|
118 |
-
|
119 |
-
|
120 |
-
|
121 |
-
|
122 |
-
|
123 |
-
|
124 |
-
|
125 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
126 |
|
127 |
-
#
|
128 |
-
@spaces.GPU
|
129 |
def generate_speech_simple(
|
130 |
text: str,
|
131 |
speaker_id: int,
|
@@ -134,43 +165,56 @@ def generate_speech_simple(
|
|
134 |
top_k: int = 50,
|
135 |
progress=gr.Progress()
|
136 |
) -> str:
|
137 |
-
|
138 |
-
|
139 |
-
|
140 |
-
|
141 |
-
|
142 |
-
|
143 |
-
|
144 |
-
|
145 |
-
|
146 |
-
|
147 |
-
|
148 |
-
|
149 |
-
|
150 |
-
|
151 |
-
|
152 |
-
|
153 |
-
|
154 |
-
|
155 |
-
|
156 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
157 |
|
158 |
-
#
|
159 |
def create_demo():
|
160 |
with gr.Blocks(title="CSM-1B Text-to-Speech") as demo:
|
161 |
gr.Markdown("# CSM-1B Text-to-Speech Demo")
|
162 |
-
gr.Markdown("
|
163 |
|
164 |
-
with gr.Tab("
|
165 |
with gr.Row():
|
166 |
with gr.Column():
|
167 |
text_input = gr.Textbox(
|
168 |
-
label="
|
169 |
-
placeholder="
|
170 |
lines=5
|
171 |
)
|
172 |
speaker_id = gr.Number(
|
173 |
-
label="ID
|
174 |
value=0,
|
175 |
precision=0,
|
176 |
minimum=0,
|
@@ -179,7 +223,7 @@ def create_demo():
|
|
179 |
|
180 |
with gr.Row():
|
181 |
max_duration = gr.Slider(
|
182 |
-
label="
|
183 |
minimum=1000,
|
184 |
maximum=90000,
|
185 |
value=30000,
|
@@ -200,38 +244,38 @@ def create_demo():
|
|
200 |
step=1
|
201 |
)
|
202 |
|
203 |
-
generate_btn = gr.Button("
|
204 |
|
205 |
with gr.Column():
|
206 |
-
output_audio = gr.Audio(label="
|
207 |
|
208 |
-
with gr.Tab("
|
209 |
-
gr.Markdown("
|
210 |
|
211 |
with gr.Row():
|
212 |
with gr.Column():
|
213 |
-
context_text1 = gr.Textbox(label="
|
214 |
-
context_audio1 = gr.Audio(label="
|
215 |
-
context_speaker1 = gr.Number(label="ID
|
216 |
|
217 |
-
context_text2 = gr.Textbox(label="
|
218 |
-
context_audio2 = gr.Audio(label="
|
219 |
-
context_speaker2 = gr.Number(label="ID
|
220 |
|
221 |
text_input_context = gr.Textbox(
|
222 |
-
label="
|
223 |
-
placeholder="
|
224 |
lines=3
|
225 |
)
|
226 |
speaker_id_context = gr.Number(
|
227 |
-
label="ID
|
228 |
value=0,
|
229 |
precision=0
|
230 |
)
|
231 |
|
232 |
with gr.Row():
|
233 |
max_duration_context = gr.Slider(
|
234 |
-
label="
|
235 |
minimum=1000,
|
236 |
maximum=90000,
|
237 |
value=30000,
|
@@ -252,27 +296,27 @@ def create_demo():
|
|
252 |
step=1
|
253 |
)
|
254 |
|
255 |
-
generate_context_btn = gr.Button("
|
256 |
|
257 |
with gr.Column():
|
258 |
-
output_audio_context = gr.Audio(label="
|
259 |
|
260 |
-
#
|
261 |
-
with gr.Tab("
|
262 |
-
gr.Markdown("###
|
263 |
gr.Markdown("""
|
264 |
-
|
265 |
|
266 |
-
|
267 |
-
1.
|
268 |
-
2.
|
269 |
|
270 |
-
|
271 |
""")
|
272 |
|
273 |
hf_token_input = gr.Textbox(
|
274 |
-
label="Hugging Face Token (
|
275 |
-
placeholder="
|
276 |
type="password"
|
277 |
)
|
278 |
|
@@ -280,57 +324,69 @@ def create_demo():
|
|
280 |
if token:
|
281 |
os.environ["HF_TOKEN"] = token
|
282 |
login(token=token)
|
283 |
-
return "
|
284 |
-
return "
|
285 |
|
286 |
-
set_token_btn = gr.Button("
|
287 |
-
token_status = gr.Textbox(label="
|
288 |
|
289 |
set_token_btn.click(fn=set_token, inputs=hf_token_input, outputs=token_status)
|
290 |
|
291 |
-
#
|
292 |
-
with gr.Tab("
|
293 |
-
gr.Markdown("###
|
294 |
gr.Markdown("""
|
295 |
-
|
|
|
|
|
|
|
|
|
296 |
|
297 |
-
|
|
|
|
|
|
|
|
|
|
|
298 |
|
299 |
-
|
|
|
|
|
|
|
300 |
|
301 |
-
|
302 |
""")
|
303 |
|
304 |
-
@spaces.GPU
|
305 |
def check_gpu():
|
306 |
if torch.cuda.is_available():
|
307 |
gpu_name = torch.cuda.get_device_name(0)
|
308 |
gpu_memory = torch.cuda.get_device_properties(0).total_memory / (1024**3)
|
309 |
-
return f"GPU: {gpu_name}\
|
310 |
else:
|
311 |
-
return "
|
312 |
|
313 |
-
check_gpu_btn = gr.Button("
|
314 |
-
gpu_info = gr.Textbox(label="
|
315 |
|
316 |
check_gpu_btn.click(fn=check_gpu, inputs=None, outputs=gpu_info)
|
317 |
|
318 |
-
#
|
319 |
-
load_model_btn = gr.Button("
|
320 |
-
model_status = gr.Textbox(label="
|
321 |
|
322 |
-
@spaces.GPU
|
323 |
def load_model_and_report():
|
324 |
global model_loaded
|
325 |
if model_loaded:
|
326 |
-
return "
|
327 |
else:
|
328 |
initialize_model()
|
329 |
-
return "
|
330 |
|
331 |
load_model_btn.click(fn=load_model_and_report, inputs=None, outputs=model_status)
|
332 |
|
333 |
-
#
|
334 |
generate_btn.click(
|
335 |
fn=generate_speech_simple,
|
336 |
inputs=[
|
@@ -363,7 +419,7 @@ def create_demo():
|
|
363 |
|
364 |
return demo
|
365 |
|
366 |
-
#
|
367 |
if __name__ == "__main__":
|
368 |
demo = create_demo()
|
369 |
demo.queue().launch()
|
|
|
11 |
from generator import Segment, load_csm_1b
|
12 |
from huggingface_hub import login
|
13 |
|
14 |
+
# Disable torch compile feature to avoid triton error
|
15 |
torch._dynamo.config.suppress_errors = True
|
16 |
|
17 |
+
# Check if GPU is available and configure the device
|
18 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
19 |
+
print(f"Using device: {device}")
|
20 |
|
21 |
+
# Login to Hugging Face Hub if token is available
|
22 |
def login_huggingface():
|
23 |
hf_token = os.environ.get("HF_TOKEN")
|
24 |
if hf_token:
|
25 |
+
print("Logging in to Hugging Face Hub...")
|
26 |
login(token=hf_token)
|
27 |
+
print("Login successful!")
|
28 |
else:
|
29 |
+
print("HF_TOKEN not found in environment variables. Some models may not be accessible.")
|
30 |
|
31 |
+
# Login at startup
|
32 |
login_huggingface()
|
33 |
|
34 |
+
# Global variables to track model state
|
35 |
generator = None
|
36 |
model_loaded = False
|
37 |
|
38 |
+
# Function to load model in ZeroGPU
|
39 |
+
@spaces.GPU(duration=30)
|
40 |
def initialize_model():
|
41 |
global generator, model_loaded
|
42 |
if not model_loaded:
|
43 |
+
print("Loading CSM-1B model in GPU...")
|
44 |
generator = load_csm_1b(device="cuda")
|
45 |
model_loaded = True
|
46 |
+
print("Model loaded successfully!")
|
47 |
return generator
|
48 |
|
49 |
+
# Function to get the loaded model
|
50 |
+
@spaces.GPU(duration=30)
|
51 |
def get_model():
|
52 |
global generator, model_loaded
|
53 |
if not model_loaded:
|
54 |
return initialize_model()
|
55 |
return generator
|
56 |
|
57 |
+
# Preload model if environment variable is set
|
58 |
+
def preload_model_if_needed():
|
59 |
+
if os.environ.get("PRELOAD_MODEL", "").lower() in ("true", "1", "yes"):
|
60 |
+
print("PRELOAD_MODEL is set. Attempting to preload model...")
|
61 |
+
try:
|
62 |
+
# We can't directly call initialize_model() here because it's decorated with @spaces.GPU
|
63 |
+
# Instead, we'll set a flag that will be checked when the first request comes in
|
64 |
+
global model_loaded
|
65 |
+
model_loaded = False
|
66 |
+
print("Model will be loaded on first request.")
|
67 |
+
except Exception as e:
|
68 |
+
print(f"Error during model preloading setup: {e}")
|
69 |
+
else:
|
70 |
+
print("PRELOAD_MODEL is not set. Model will be loaded on demand.")
|
71 |
+
|
72 |
+
# Call preload function at startup
|
73 |
+
preload_model_if_needed()
|
74 |
+
|
75 |
+
# Function to convert audio to tensor
|
76 |
def audio_to_tensor(audio_path: str) -> Tuple[torch.Tensor, int]:
|
77 |
waveform, sample_rate = torchaudio.load(audio_path)
|
78 |
+
waveform = waveform.mean(dim=0) # Convert stereo to mono if needed
|
79 |
return waveform, sample_rate
|
80 |
|
81 |
+
# Function to save audio tensor to file
|
82 |
def save_audio(audio_tensor: torch.Tensor, sample_rate: int) -> str:
|
83 |
temp_dir = tempfile.gettempdir()
|
84 |
output_path = os.path.join(temp_dir, f"csm1b_output_{int(time.time())}.wav")
|
85 |
torchaudio.save(output_path, audio_tensor.unsqueeze(0), sample_rate)
|
86 |
return output_path
|
87 |
|
88 |
+
# Function to generate speech from text using ZeroGPU
|
89 |
+
@spaces.GPU(duration=30)
|
90 |
def generate_speech(
|
91 |
text: str,
|
92 |
speaker_id: int,
|
|
|
101 |
top_k: int = 50,
|
102 |
progress=gr.Progress()
|
103 |
) -> str:
|
104 |
+
try:
|
105 |
+
# Get the loaded model
|
106 |
+
generator = get_model()
|
107 |
+
|
108 |
+
# Prepare context
|
109 |
+
context = []
|
110 |
+
progress(0.1, "Processing context...")
|
111 |
+
|
112 |
+
# Process context 1
|
113 |
+
if context_audio_path1 and context_text1:
|
114 |
+
waveform, sample_rate = audio_to_tensor(context_audio_path1)
|
115 |
+
# Resample if needed
|
116 |
+
if sample_rate != generator.sample_rate:
|
117 |
+
waveform = torchaudio.functional.resample(waveform, orig_freq=sample_rate, new_freq=generator.sample_rate)
|
118 |
+
context.append(Segment(speaker=context_speaker1, text=context_text1, audio=waveform))
|
119 |
+
|
120 |
+
# Process context 2
|
121 |
+
if context_audio_path2 and context_text2:
|
122 |
+
waveform, sample_rate = audio_to_tensor(context_audio_path2)
|
123 |
+
# Resample if needed
|
124 |
+
if sample_rate != generator.sample_rate:
|
125 |
+
waveform = torchaudio.functional.resample(waveform, orig_freq=sample_rate, new_freq=generator.sample_rate)
|
126 |
+
context.append(Segment(speaker=context_speaker2, text=context_text2, audio=waveform))
|
127 |
+
|
128 |
+
progress(0.3, "Generating audio...")
|
129 |
+
# Generate audio from text
|
130 |
+
audio = generator.generate(
|
131 |
+
text=text,
|
132 |
+
speaker=speaker_id,
|
133 |
+
context=context,
|
134 |
+
max_audio_length_ms=max_duration_ms,
|
135 |
+
temperature=temperature,
|
136 |
+
topk=top_k
|
137 |
+
)
|
138 |
+
|
139 |
+
progress(0.8, "Saving audio...")
|
140 |
+
# Save audio to file
|
141 |
+
output_path = save_audio(audio, generator.sample_rate)
|
142 |
+
|
143 |
+
progress(1.0, "Completed!")
|
144 |
+
return output_path
|
145 |
+
except spaces.zero.gradio.HTMLError as e:
|
146 |
+
# Handle ZeroGPU quota exceeded error
|
147 |
+
error_message = str(e)
|
148 |
+
if "GPU quota exceeded" in error_message:
|
149 |
+
# Extract wait time from error message
|
150 |
+
import re
|
151 |
+
wait_time_match = re.search(r"Try again in (\d+:\d+:\d+)", error_message)
|
152 |
+
wait_time = wait_time_match.group(1) if wait_time_match else "some time"
|
153 |
+
return f"GPU quota exceeded. Please try again in {wait_time}."
|
154 |
+
return f"GPU error: {error_message}"
|
155 |
+
except Exception as e:
|
156 |
+
return f"Error generating speech: {str(e)}"
|
157 |
|
158 |
+
# Function to generate simple speech without context
|
159 |
+
@spaces.GPU(duration=30)
|
160 |
def generate_speech_simple(
|
161 |
text: str,
|
162 |
speaker_id: int,
|
|
|
165 |
top_k: int = 50,
|
166 |
progress=gr.Progress()
|
167 |
) -> str:
|
168 |
+
try:
|
169 |
+
# Get the loaded model
|
170 |
+
generator = get_model()
|
171 |
+
|
172 |
+
progress(0.3, "Generating audio...")
|
173 |
+
# Generate audio from text
|
174 |
+
audio = generator.generate(
|
175 |
+
text=text,
|
176 |
+
speaker=speaker_id,
|
177 |
+
context=[], # No context
|
178 |
+
max_audio_length_ms=max_duration_ms,
|
179 |
+
temperature=temperature,
|
180 |
+
topk=top_k
|
181 |
+
)
|
182 |
+
|
183 |
+
progress(0.8, "Saving audio...")
|
184 |
+
# Save audio to file
|
185 |
+
output_path = save_audio(audio, generator.sample_rate)
|
186 |
+
|
187 |
+
progress(1.0, "Completed!")
|
188 |
+
return output_path
|
189 |
+
except spaces.zero.gradio.HTMLError as e:
|
190 |
+
# Handle ZeroGPU quota exceeded error
|
191 |
+
error_message = str(e)
|
192 |
+
if "GPU quota exceeded" in error_message:
|
193 |
+
# Extract wait time from error message
|
194 |
+
import re
|
195 |
+
wait_time_match = re.search(r"Try again in (\d+:\d+:\d+)", error_message)
|
196 |
+
wait_time = wait_time_match.group(1) if wait_time_match else "some time"
|
197 |
+
return f"GPU quota exceeded. Please try again in {wait_time}."
|
198 |
+
return f"GPU error: {error_message}"
|
199 |
+
except Exception as e:
|
200 |
+
return f"Error generating speech: {str(e)}"
|
201 |
|
202 |
+
# Create Gradio interface
|
203 |
def create_demo():
|
204 |
with gr.Blocks(title="CSM-1B Text-to-Speech") as demo:
|
205 |
gr.Markdown("# CSM-1B Text-to-Speech Demo")
|
206 |
+
gr.Markdown("CSM-1B (Collaborative Speech Model) is an advanced text-to-speech model capable of generating natural-sounding speech from text.")
|
207 |
|
208 |
+
with gr.Tab("Simple Audio Generation"):
|
209 |
with gr.Row():
|
210 |
with gr.Column():
|
211 |
text_input = gr.Textbox(
|
212 |
+
label="Text to convert to speech",
|
213 |
+
placeholder="Enter the text you want to convert to speech...",
|
214 |
lines=5
|
215 |
)
|
216 |
speaker_id = gr.Number(
|
217 |
+
label="Speaker ID",
|
218 |
value=0,
|
219 |
precision=0,
|
220 |
minimum=0,
|
|
|
223 |
|
224 |
with gr.Row():
|
225 |
max_duration = gr.Slider(
|
226 |
+
label="Maximum Duration (ms)",
|
227 |
minimum=1000,
|
228 |
maximum=90000,
|
229 |
value=30000,
|
|
|
244 |
step=1
|
245 |
)
|
246 |
|
247 |
+
generate_btn = gr.Button("Generate Audio")
|
248 |
|
249 |
with gr.Column():
|
250 |
+
output_audio = gr.Audio(label="Output Audio", type="filepath")
|
251 |
|
252 |
+
with gr.Tab("Audio Generation with Context"):
|
253 |
+
gr.Markdown("This feature allows you to provide audio clips and text as context to help the model generate more appropriate speech.")
|
254 |
|
255 |
with gr.Row():
|
256 |
with gr.Column():
|
257 |
+
context_text1 = gr.Textbox(label="Context Text 1", lines=2)
|
258 |
+
context_audio1 = gr.Audio(label="Context Audio 1", type="filepath")
|
259 |
+
context_speaker1 = gr.Number(label="Speaker ID 1", value=0, precision=0)
|
260 |
|
261 |
+
context_text2 = gr.Textbox(label="Context Text 2", lines=2)
|
262 |
+
context_audio2 = gr.Audio(label="Context Audio 2", type="filepath")
|
263 |
+
context_speaker2 = gr.Number(label="Speaker ID 2", value=1, precision=0)
|
264 |
|
265 |
text_input_context = gr.Textbox(
|
266 |
+
label="Text to convert to speech",
|
267 |
+
placeholder="Enter the text you want to convert to speech...",
|
268 |
lines=3
|
269 |
)
|
270 |
speaker_id_context = gr.Number(
|
271 |
+
label="Speaker ID",
|
272 |
value=0,
|
273 |
precision=0
|
274 |
)
|
275 |
|
276 |
with gr.Row():
|
277 |
max_duration_context = gr.Slider(
|
278 |
+
label="Maximum Duration (ms)",
|
279 |
minimum=1000,
|
280 |
maximum=90000,
|
281 |
value=30000,
|
|
|
296 |
step=1
|
297 |
)
|
298 |
|
299 |
+
generate_context_btn = gr.Button("Generate Audio with Context")
|
300 |
|
301 |
with gr.Column():
|
302 |
+
output_audio_context = gr.Audio(label="Output Audio", type="filepath")
|
303 |
|
304 |
+
# Add Hugging Face configuration tab
|
305 |
+
with gr.Tab("Configuration"):
|
306 |
+
gr.Markdown("### Hugging Face Token Configuration")
|
307 |
gr.Markdown("""
|
308 |
+
To use the CSM-1B model, you need access to the model on Hugging Face.
|
309 |
|
310 |
+
You can configure your token by:
|
311 |
+
1. Create a token at [Hugging Face Settings](https://huggingface.co/settings/tokens)
|
312 |
+
2. Set the `HF_TOKEN` environment variable with your token value
|
313 |
|
314 |
+
Note: In Hugging Face Spaces, you can set environment variables in the Space Settings.
|
315 |
""")
|
316 |
|
317 |
hf_token_input = gr.Textbox(
|
318 |
+
label="Hugging Face Token (Only for this session)",
|
319 |
+
placeholder="Enter your token...",
|
320 |
type="password"
|
321 |
)
|
322 |
|
|
|
324 |
if token:
|
325 |
os.environ["HF_TOKEN"] = token
|
326 |
login(token=token)
|
327 |
+
return "Token set successfully! You can now load the model."
|
328 |
+
return "Invalid token. Please enter a valid token."
|
329 |
|
330 |
+
set_token_btn = gr.Button("Set Token")
|
331 |
+
token_status = gr.Textbox(label="Status", interactive=False)
|
332 |
|
333 |
set_token_btn.click(fn=set_token, inputs=hf_token_input, outputs=token_status)
|
334 |
|
335 |
+
# Add GPU information tab
|
336 |
+
with gr.Tab("GPU Information"):
|
337 |
+
gr.Markdown("### About ZeroGPU")
|
338 |
gr.Markdown("""
|
339 |
+
This application uses Hugging Face Spaces' ZeroGPU to optimize GPU usage.
|
340 |
+
|
341 |
+
ZeroGPU helps free up GPU memory when not in use, saving resources and improving performance.
|
342 |
+
|
343 |
+
When you generate audio, the GPU will be used automatically and released after completion.
|
344 |
|
345 |
+
Note: In the ZeroGPU environment, CUDA is not initialized in the main process, but only in functions with the @spaces.GPU decorator.
|
346 |
+
""")
|
347 |
+
|
348 |
+
gr.Markdown("### GPU Quota Information")
|
349 |
+
gr.Markdown("""
|
350 |
+
Hugging Face Spaces has GPU quota limitations:
|
351 |
|
352 |
+
- Each GPU operation has a default duration of 60 seconds
|
353 |
+
- We've reduced this to 30 seconds for audio generation and 10 seconds for GPU checks
|
354 |
+
- If you exceed your quota, you'll need to wait for it to reset (usually a few hours)
|
355 |
+
- For better performance, try generating shorter audio clips
|
356 |
|
357 |
+
If you encounter a "GPU quota exceeded" error, please wait for the specified time and try again.
|
358 |
""")
|
359 |
|
360 |
+
@spaces.GPU(duration=10)
|
361 |
def check_gpu():
|
362 |
if torch.cuda.is_available():
|
363 |
gpu_name = torch.cuda.get_device_name(0)
|
364 |
gpu_memory = torch.cuda.get_device_properties(0).total_memory / (1024**3)
|
365 |
+
return f"GPU: {gpu_name}\nMemory: {gpu_memory:.2f} GB"
|
366 |
else:
|
367 |
+
return "No GPU found. The application will run on CPU."
|
368 |
|
369 |
+
check_gpu_btn = gr.Button("Check GPU")
|
370 |
+
gpu_info = gr.Textbox(label="GPU Information", interactive=False)
|
371 |
|
372 |
check_gpu_btn.click(fn=check_gpu, inputs=None, outputs=gpu_info)
|
373 |
|
374 |
+
# Add model loading button
|
375 |
+
load_model_btn = gr.Button("Load Model")
|
376 |
+
model_status = gr.Textbox(label="Model Status", interactive=False)
|
377 |
|
378 |
+
@spaces.GPU(duration=10)
|
379 |
def load_model_and_report():
|
380 |
global model_loaded
|
381 |
if model_loaded:
|
382 |
+
return "Model has already been loaded!"
|
383 |
else:
|
384 |
initialize_model()
|
385 |
+
return "Model loaded successfully!"
|
386 |
|
387 |
load_model_btn.click(fn=load_model_and_report, inputs=None, outputs=model_status)
|
388 |
|
389 |
+
# Connect components
|
390 |
generate_btn.click(
|
391 |
fn=generate_speech_simple,
|
392 |
inputs=[
|
|
|
419 |
|
420 |
return demo
|
421 |
|
422 |
+
# Launch the application
|
423 |
if __name__ == "__main__":
|
424 |
demo = create_demo()
|
425 |
demo.queue().launch()
|
generator.py
CHANGED
@@ -10,7 +10,7 @@ from tokenizers.processors import TemplateProcessing
|
|
10 |
from transformers import AutoTokenizer
|
11 |
from watermarking import CSM_1B_GH_WATERMARK, load_watermarker, watermark
|
12 |
|
13 |
-
#
|
14 |
torch._dynamo.config.suppress_errors = True
|
15 |
|
16 |
@dataclass
|
@@ -167,19 +167,19 @@ class Generator:
|
|
167 |
|
168 |
def load_csm_1b(device: str = "cuda") -> Generator:
|
169 |
"""
|
170 |
-
|
171 |
|
172 |
Args:
|
173 |
-
device:
|
174 |
|
175 |
Returns:
|
176 |
-
Generator:
|
177 |
"""
|
178 |
try:
|
179 |
-
#
|
180 |
-
#
|
181 |
if 'cuda' in device and not torch.cuda.is_initialized():
|
182 |
-
#
|
183 |
model = Model.from_pretrained("sesame/csm-1b")
|
184 |
else:
|
185 |
model = Model.from_pretrained("sesame/csm-1b")
|
@@ -188,7 +188,7 @@ def load_csm_1b(device: str = "cuda") -> Generator:
|
|
188 |
generator = Generator(model)
|
189 |
return generator
|
190 |
except Exception as e:
|
191 |
-
print(f"
|
192 |
-
print("
|
193 |
-
print("
|
194 |
raise e
|
|
|
10 |
from transformers import AutoTokenizer
|
11 |
from watermarking import CSM_1B_GH_WATERMARK, load_watermarker, watermark
|
12 |
|
13 |
+
# Disable torch compile feature to avoid triton error
|
14 |
torch._dynamo.config.suppress_errors = True
|
15 |
|
16 |
@dataclass
|
|
|
167 |
|
168 |
def load_csm_1b(device: str = "cuda") -> Generator:
|
169 |
"""
|
170 |
+
Load the CSM-1B model from Hugging Face Hub.
|
171 |
|
172 |
Args:
|
173 |
+
device: Device to run the model on (cuda or cpu)
|
174 |
|
175 |
Returns:
|
176 |
+
Generator: Generator object to create audio from text
|
177 |
"""
|
178 |
try:
|
179 |
+
# In ZeroGPU, CUDA should not be initialized in the main process
|
180 |
+
# Only move the model to GPU when called in a function with the @spaces.GPU decorator
|
181 |
if 'cuda' in device and not torch.cuda.is_initialized():
|
182 |
+
# Use CPU for the main process
|
183 |
model = Model.from_pretrained("sesame/csm-1b")
|
184 |
else:
|
185 |
model = Model.from_pretrained("sesame/csm-1b")
|
|
|
188 |
generator = Generator(model)
|
189 |
return generator
|
190 |
except Exception as e:
|
191 |
+
print(f"Error loading model: {e}")
|
192 |
+
print("Please check if you are logged in to Hugging Face Hub.")
|
193 |
+
print("You may need to request access to the model at: https://huggingface.co/sesame/csm-1b")
|
194 |
raise e
|
test_model.py
CHANGED
@@ -6,29 +6,29 @@ from generator import Segment, load_csm_1b
|
|
6 |
from huggingface_hub import login
|
7 |
|
8 |
def login_huggingface():
|
9 |
-
"""
|
10 |
hf_token = os.environ.get("HF_TOKEN")
|
11 |
|
12 |
if not hf_token:
|
13 |
-
print("
|
14 |
-
hf_token = input("
|
15 |
|
16 |
if hf_token:
|
17 |
-
print("
|
18 |
login(token=hf_token)
|
19 |
-
print("
|
20 |
return True
|
21 |
else:
|
22 |
-
print("
|
23 |
return False
|
24 |
|
25 |
@spaces.GPU
|
26 |
def generate_test_audio(text, speaker_id, device):
|
27 |
-
"""
|
28 |
generator = load_csm_1b(device=device)
|
29 |
-
print("
|
30 |
|
31 |
-
print(f"
|
32 |
audio = generator.generate(
|
33 |
text=text,
|
34 |
speaker=speaker_id,
|
@@ -41,33 +41,33 @@ def generate_test_audio(text, speaker_id, device):
|
|
41 |
return audio, generator.sample_rate
|
42 |
|
43 |
def test_model():
|
44 |
-
print("
|
45 |
|
46 |
-
#
|
47 |
login_huggingface()
|
48 |
|
49 |
-
#
|
50 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
51 |
-
print(f"
|
52 |
|
53 |
-
#
|
54 |
-
print("
|
55 |
try:
|
56 |
-
#
|
57 |
-
text = "
|
58 |
speaker_id = 0
|
59 |
|
60 |
audio, sample_rate = generate_test_audio(text, speaker_id, device)
|
61 |
|
62 |
-
#
|
63 |
output_path = "test_output.wav"
|
64 |
torchaudio.save(output_path, audio.unsqueeze(0), sample_rate)
|
65 |
-
print(f"
|
66 |
|
67 |
-
print("
|
68 |
except Exception as e:
|
69 |
-
print(f"
|
70 |
-
print("
|
71 |
|
72 |
if __name__ == "__main__":
|
73 |
test_model()
|
|
|
6 |
from huggingface_hub import login
|
7 |
|
8 |
def login_huggingface():
|
9 |
+
"""Login to Hugging Face Hub using token from environment variable or user input"""
|
10 |
hf_token = os.environ.get("HF_TOKEN")
|
11 |
|
12 |
if not hf_token:
|
13 |
+
print("HF_TOKEN not found in environment variables.")
|
14 |
+
hf_token = input("Please enter your Hugging Face token: ")
|
15 |
|
16 |
if hf_token:
|
17 |
+
print("Logging in to Hugging Face Hub...")
|
18 |
login(token=hf_token)
|
19 |
+
print("Login successful!")
|
20 |
return True
|
21 |
else:
|
22 |
+
print("No token provided. Some models may not be accessible.")
|
23 |
return False
|
24 |
|
25 |
@spaces.GPU
|
26 |
def generate_test_audio(text, speaker_id, device):
|
27 |
+
"""Generate test audio using ZeroGPU"""
|
28 |
generator = load_csm_1b(device=device)
|
29 |
+
print("Model loaded successfully!")
|
30 |
|
31 |
+
print(f"Generating audio for text: '{text}'")
|
32 |
audio = generator.generate(
|
33 |
text=text,
|
34 |
speaker=speaker_id,
|
|
|
41 |
return audio, generator.sample_rate
|
42 |
|
43 |
def test_model():
|
44 |
+
print("Testing CSM-1B model...")
|
45 |
|
46 |
+
# Login to Hugging Face Hub
|
47 |
login_huggingface()
|
48 |
|
49 |
+
# Check if GPU is available and configure the device
|
50 |
device = "cuda" if torch.cuda.is_available() else "cpu"
|
51 |
+
print(f"Using device: {device}")
|
52 |
|
53 |
+
# Load CSM-1B model and generate audio
|
54 |
+
print("Loading CSM-1B model...")
|
55 |
try:
|
56 |
+
# Use ZeroGPU to generate audio
|
57 |
+
text = "Hello, this is a test of the CSM-1B model."
|
58 |
speaker_id = 0
|
59 |
|
60 |
audio, sample_rate = generate_test_audio(text, speaker_id, device)
|
61 |
|
62 |
+
# Save audio to file
|
63 |
output_path = "test_output.wav"
|
64 |
torchaudio.save(output_path, audio.unsqueeze(0), sample_rate)
|
65 |
+
print(f"Audio saved to file: {output_path}")
|
66 |
|
67 |
+
print("Test completed!")
|
68 |
except Exception as e:
|
69 |
+
print(f"Error testing model: {e}")
|
70 |
+
print("Please check your token and access permissions.")
|
71 |
|
72 |
if __name__ == "__main__":
|
73 |
test_model()
|