README.md

metadata

title: SingingSDS
emoji: 🎶
colorFrom: pink
colorTo: yellow
sdk: gradio
sdk_version: 5.4.0
app_file: app.py
pinned: false
python_version: 3.11

SingingSDS: Role-Playing Singing Spoken Dialogue System

A role-playing singing dialogue system that converts speech input into character-based singing output.

Installation

Requirements

• Python 3.11+
• CUDA (optional, for GPU acceleration)

Install Dependencies

Option 1: Using Conda (Recommended)

conda create -n singingsds python=3.11

conda activate singingsds
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt

Option 2: Using pip only

pip install -r requirements.txt

Option 3: Using pip with virtual environment

python -m venv singingsds_env

# On Windows:
singingsds_env\Scripts\activate
# On macOS/Linux:
source singingsds_env/bin/activate

pip install -r requirements.txt

Usage

Command Line Interface (CLI)

Example Usage

python cli.py --query_audio tests/audio/hello.wav --config_path config/cli/yaoyin_default.yaml --output_audio outputs/yaoyin_hello.wav

Parameter Description

• --query_audio: Input audio file path (required)
• --config_path: Configuration file path (default: config/cli/yaoyin_default.yaml)
• --output_audio: Output audio file path (required)

Web Interface (Gradio)

Start the web interface:

python app.py

Then visit the displayed address in your browser to use the graphical interface.

Configuration

Character Configuration

The system supports multiple preset characters:

• Yaoyin (遥音): Default timbre is timbre2
• Limei (丽梅): Default timbre is timbre1

Model Configuration

ASR Models

• openai/whisper-large-v3-turbo
• openai/whisper-large-v3
• openai/whisper-medium
• sanchit-gandhi/whisper-small-dv
• facebook/wav2vec2-base-960h

LLM Models

• google/gemma-2-2b
• MiniMaxAI/MiniMax-M1-80k
• meta-llama/Llama-3.2-3B-Instruct

SVS Models

• espnet/mixdata_svs_visinger2_spkemb_lang_pretrained_avg (Bilingual)
• espnet/aceopencpop_svs_visinger2_40singer_pretrain (Chinese)

Project Structure

SingingSDS/
├── cli.py                 # Command line interface
├── interface.py           # Gradio interface
├── pipeline.py            # Core processing pipeline
├── app.py                 # Web application entry
├── requirements.txt       # Python dependencies
├── config/                # Configuration files
│   ├── cli/               # CLI-specific configuration
│   └── interface/         # Interface-specific configuration
├── modules/               # Core modules
│   ├── asr.py            # Speech recognition module
│   ├── llm.py            # Large language model module
│   ├── melody.py         # Melody control module
│   ├── svs/              # Singing voice synthesis modules
│   │   ├── base.py       # Base SVS class
│   │   ├── espnet.py     # ESPnet SVS implementation
│   │   ├── registry.py   # SVS model registry
│   │   └── __init__.py   # SVS module initialization
│   └── utils/            # Utility modules
│       ├── g2p.py        # Grapheme-to-phoneme conversion
│       ├── text_normalize.py # Text normalization
│       └── resources/    # Utility resources
├── characters/            # Character definitions
│   ├── base.py           # Base character class
│   ├── Limei.py          # Limei character definition
│   ├── Yaoyin.py         # Yaoyin character definition
│   └── __init__.py       # Character module initialization
├── evaluation/            # Evaluation modules
│   └── svs_eval.py       # SVS evaluation metrics
├── data/                  # Data directory
│   ├── kising/           # Kising dataset
│   └── touhou/           # Touhou dataset
├── resources/             # Project resources
├── data_handlers/         # Data handling utilities
├── assets/                # Static assets
└── tests/                 # Test files

Contributing

Issues and Pull Requests are welcome!

License