Spaces:
Running
Running
File size: 2,628 Bytes
0f6c755 7b2aced 0f6c755 512cb02 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 |
---
title: VoxFactory
emoji: 🌬️
colorFrom: gray
colorTo: red
sdk: gradio
app_file: interface.py
pinned: false
license: mit
short_description: FinetuneASR Voxtral
---
# Finetune Voxtral for ASR with Transformers 🤗
This repository fine-tunes the [Voxtral](https://huggingface.co/Deep-unlearning/Voxtral) speech model on conversational speech datasets using the Hugging Face `transformers` and `datasets` libraries.
## Installation
### Step 1: Clone the repository
```bash
git clone https://github.com/Deep-unlearning/Finetune-Voxtral-ASR.git
cd Finetune-Voxtral-ASR
```
### Step 2: Set up environment
Choose your preferred package manager:
<details>
<summary>📦 Using UV (recommended)</summary>
[Install `uv`](https://docs.astral.sh/uv/getting-started/installation/)
```bash
uv venv .venv --python 3.10 && source .venv/bin/activate
uv pip install -r requirements.txt
```
</details>
<details>
<summary>🐍 Using pip</summary>
```bash
python -m venv .venv --python 3.10 && source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
```
</details>
## Dataset Preparation
Perfect — here’s a **drop-in replacement** for your README’s “Dataset Preparation” that matches your script (uses **`hf-audio/esb-datasets-test-only-sorted`** with the **`voxpopuli`** config, 16 kHz casting, and a small train/eval slice), and explains the Voxtral/LLaMA-style prompt+label masking your collator implements.
---
## Dataset Preparation
For ASR fine-tuning, inputs look like:
* **Inputs**: `[AUDIO] … [AUDIO] <transcribe> <reference transcription>`
* **Labels**: same sequence, but the prefix `[AUDIO] … [AUDIO] <transcribe>` is **masked with `-100`** so loss is computed **only** on the transcription tokens.
The `VoxtralDataCollator` already builds this sequence (prompt expansion via the processor and label masking).
The dataset only needs two fields:
```python
{
"audio": {"array": <float32 numpy array>, "sampling_rate": 16000, ...},
"text": "<reference transcription>"
}
```
If you want to swap to a different dataset, ensure after loading you still have:
* an **`audio`** column (cast to `Audio(sampling_rate=16000)`), and
* a **`text`** column (the reference transcription).
If your dataset uses different column names, map them to `audio` and `text` before returning.
## Training
Run the training script:
```bash
uv run train.py
```
Logs and checkpoints will be saved under the `outputs/` directory by default.
## Training with LoRA
You can also run the training script with LoRA:
```bash
uv run train_lora.py
```
**Happy fine-tuning Voxtral!** 🚀 |