Spanish TTS Model with Emotions and Multiple Voices

This repository contains a fine-tuned Spanish Text-to-Speech (TTS) model based on canopylabs/3b-es_it-pretrain-research_release. The model supports multiple voices and nuanced emotions, trained using Unsloth and SNAC for audio tokenization.

➡️ Try it online: https://huggingface.co/spaces/sirekist98/orpheustts_spanish_tuned

👨‍💻 Model Summary

Base model: canopylabs/3b-es_it-pretrain-research_release
Fine-tuned with: LoRA adapters (64 rank, alpha 64)
Audio tokenization: SNAC (24kHz)
Input format: source (emotion): text
Dataset: ~109k samples, 11 emotions × 11 speakers
Training framework: Unsloth + Hugging Face Transformers

🚀 Training Overview

The model was trained on a curated subset of the dataset sirekist98/spanish_tts_noauddataset_24khz. We selected combinations of speaker (source) and emotion with at least 1000 samples, resulting in a balanced dataset of over 109,000 examples.

Each sample was tokenized using SNAC and embedded in a prompt structured as:

source (emotion): text

This prompt was then used to generate audio tokens, enabling the model to learn nuanced emotional prosody and voice control.

We trained the model for 1 epoch using gradient accumulation (batch size 8 × 4 steps) with 4-bit quantization on an NVIDIA L4 GPU.

🔊 Inference

You can run inference using the demo space: Orpheus TTS Spanish Fine-Tuned.

To run inference locally with full control:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
from snac import SNAC

# --- Minimal config ---
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
BASE  = "canopylabs/3b-es_it-pretrain-research_release"
LORA  = "sirekist98/orpheustts_spanish_finetuned"
SNAC_ID = "hubertsiuzdak/snac_24khz"

VOICE = "alloy"
EMOTION_ID = "intense_fear_dread_apprehension_horror_terror_panic"
TEXT = "Estoy atrapado, por favor ayúdame."
prompt = f"{VOICE} ({EMOTION_ID}): {TEXT}"

# --- Load models ---
tokenizer  = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE,
    torch_dtype=torch.float16 if device.type == "cuda" else torch.float32
)
model      = PeftModel.from_pretrained(base_model, LORA).to(device).eval()
snac_model = SNAC.from_pretrained(SNAC_ID).to(device)

# --- Prepare input (same as your Space) ---
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
start_tok = torch.tensor([[128259]], dtype=torch.long).to(device)
end_toks  = torch.tensor([[128009, 128260]], dtype=torch.long).to(device)

input_ids = torch.cat([start_tok, input_ids, end_toks], dim=1)
MAX_LEN   = 4260
pad_len   = MAX_LEN - input_ids.shape[1]
pad       = torch.full((1, pad_len), 128263, dtype=torch.long).to(device)
input_ids = torch.cat([pad, input_ids], dim=1)
attention_mask = torch.cat(
    [torch.zeros((1, pad_len), dtype=torch.long),
     torch.ones((1, input_ids.shape[1] - pad_len), dtype=torch.long)],
    dim=1
).to(device)

# --- Generate ---
generated = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    max_new_tokens=1200,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    repetition_penalty=1.1,
    num_return_sequences=1,
    eos_token_id=128258,
    use_cache=True
)

# --- Post-process (find 128257, remove 128258, multiple of 7, subtract 128266) ---
AUDIO_TOKEN_OFFSET = 128266
token_to_find      = 128257
token_to_remove    = 128258

idxs = (generated == token_to_find).nonzero(as_tuple=True)
cropped = generated[:, idxs[1][-1].item() + 1:] if len(idxs[1]) > 0 else generated
cleaned = cropped[cropped != token_to_remove]
codes   = cleaned[: (len(cleaned) // 7) * 7].tolist()
codes   = [int(t) - AUDIO_TOKEN_OFFSET for t in codes]

# --- SNAC decode (same layout as your Space) ---
layer_1, layer_2, layer_3 = [], [], []
for i in range((len(codes) + 1) // 7):
    b = 7 * i
    if b + 6 >= len(codes):
        break
    layer_1.append(codes[b + 0])
    layer_2.append(codes[b + 1] - 4096)
    layer_3.append(codes[b + 2] - 2 * 4096)
    layer_3.append(codes[b + 3] - 3 * 4096)
    layer_2.append(codes[b + 4] - 4 * 4096)
    layer_3.append(codes[b + 5] - 5 * 4096)
    layer_3.append(codes[b + 6] - 6 * 4096)

dev_snac = snac_model.quantizer.quantizers[0].codebook.weight.device
layers = [
    torch.tensor(layer_1).unsqueeze(0).to(dev_snac),
    torch.tensor(layer_2).unsqueeze(0).to(dev_snac),
    torch.tensor(layer_3).unsqueeze(0).to(dev_snac),
]

with torch.no_grad():
    audio = snac_model.decode(layers).squeeze().cpu().numpy()

# 'audio' is the 24kHz waveform.
# Optional:
# from scipy.io.wavfile import write as write_wav
# write_wav("output.wav", 24000, audio)

🗣️ Available Voices

You can generate speech using the following voices (source):

alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse

🌧️ Available Emotions for each voice

alloy

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_astonishment_surprise_amazement_and_shock
intense_confusion_bewilderment_disorientation_and_perplexity
intense_pride_dignity_self_confidence_and_honor
intense_sourness_tartness_and_acidity
intense_sympathy_compassion_warmth_trust_and_tenderness

ash

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_astonishment_surprise_amazement_and_shock
intense_sympathy_compassion_warmth_trust_and_tenderness

ballad

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_astonishment_surprise_amazement_and_shock
intense_confusion_bewilderment_disorientation_and_perplexity
intense_helplessness_powerlessness_desperation_and_submission
intense_pride_dignity_self_confidence_and_honor
intense_sourness_tartness_and_acidity

coral

intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_confusion_bewilderment_disorientation_and_perplexity
intense_helplessness_powerlessness_desperation_and_submission
intense_pride_dignity_self_confidence_and_honor
intense_sourness_tartness_and_acidity
intense_sympathy_compassion_warmth_trust_and_tenderness

echo

intense_interest_fascination_curiosity_and_intrigue
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_astonishment_surprise_amazement_and_shock
intense_helplessness_powerlessness_desperation_and_submission
intense_pride_dignity_self_confidence_and_honor
intense_sympathy_compassion_warmth_trust_and_tenderness

fable

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_helplessness_powerlessness_desperation_and_submission
intense_sourness_tartness_and_acidity

nova

intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_contempt_disdain_loathing_and_detestation
intense_astonishment_surprise_amazement_and_shock
intense_confusion_bewilderment_disorientation_and_perplexity
intense_helplessness_powerlessness_desperation_and_submission
intense_pride_dignity_self_confidence_and_honor
intense_sourness_tartness_and_acidity
intense_sympathy_compassion_warmth_trust_and_tenderness

onyx

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_numbness_detachment_insensitivity_and_apathy
intense_confusion_bewilderment_disorientation_and_perplexity
intense_helplessness_powerlessness_desperation_and_submission
intense_pride_dignity_self_confidence_and_honor
intense_sympathy_compassion_warmth_trust_and_tenderness

sage

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_astonishment_surprise_amazement_and_shock
intense_confusion_bewilderment_disorientation_and_perplexity
intense_pride_dignity_self_confidence_and_honor
intense_sourness_tartness_and_acidity
intense_sympathy_compassion_warmth_trust_and_tenderness

shimmer

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_astonishment_surprise_amazement_and_shock
intense_confusion_bewilderment_disorientation_and_perplexity
intense_helplessness_powerlessness_desperation_and_submission
intense_pride_dignity_self_confidence_and_honor
intense_sourness_tartness_and_acidity

verse

intense_interest_fascination_curiosity_and_intrigue
intense_fear_dread_apprehension_and_horror
intense_ecstasy_pleasure_bliss_rapture_and_beatitude
intense_numbness_detachment_insensitivity_and_apathy
intense_contempt_disdain_loathing_and_detestation
intense_astonishment_surprise_amazement_and_shock
intense_helplessness_powerlessness_desperation_and_submission
intense_sourness_tartness_and_acidity

📖 Citation

@misc{sirekist2025spanishTTS,
  author = {sirekist98},
  title = {Spanish TTS Model with Emotions and Multiple Voices},
  year = {2025},
  howpublished = {\url{https://huggingface.co/sirekist98/spanish_model}}
}

✨ Acknowledgements

❓ Questions or Contributions?

Open an issue or contact @sirekist98 on Hugging Face.

Thanks for checking out this model! 🚀

sirekist98
/

orpheustts_spanish_finetuned