|
|
--- |
|
|
title: Semantic Deduplication |
|
|
emoji: 🧹 |
|
|
colorFrom: green |
|
|
colorTo: green |
|
|
sdk: gradio |
|
|
sdk_version: 5.32.1 |
|
|
app_file: app.py |
|
|
pinned: false |
|
|
license: mit |
|
|
short_description: Deduplicate HuggingFace datasets in seconds |
|
|
hf_oauth: true |
|
|
hf_oauth_scopes: |
|
|
- write-repos |
|
|
- manage-repos |
|
|
--- |
|
|
|
|
|
# Semantic Text Deduplication Using SemHash |
|
|
|
|
|
This Gradio application performs **semantic deduplication** on HuggingFace datasets using [SemHash](https://github.com/MinishLab/semhash) with [Model2Vec](https://github.com/MinishLab/model2vec) embeddings. |
|
|
|
|
|
## Features |
|
|
|
|
|
- **Two deduplication modes**: |
|
|
- **Single dataset**: Find and remove duplicates within one dataset |
|
|
- **Cross-dataset**: Remove entries from Dataset 2 that are similar to entries in Dataset 1 |
|
|
|
|
|
- **Customizable similarity threshold**: Control how strict the deduplication should be (0.0 = very loose, 1.0 = exact matches only) |
|
|
|
|
|
- **Detailed results**: View statistics and examples of found duplicates with word-level differences highlighted |
|
|
|
|
|
- **Hub Integration**: 🆕 **Push deduplicated datasets directly to the Hugging Face Hub** after logging in |
|
|
|
|
|
## How to Use |
|
|
|
|
|
### 1. Choose Deduplication Type |
|
|
- **Cross-dataset**: Useful for removing training data contamination from test sets |
|
|
- **Single dataset**: Clean up duplicate entries within a single dataset |
|
|
|
|
|
### 2. Configure Datasets |
|
|
- Enter the HuggingFace dataset names (e.g., `SetFit/amazon_massive_scenario_en-US`) |
|
|
- Specify the dataset splits (e.g., `train`, `test`, `validation`) |
|
|
- Set the text column name (usually `text`, `sentence`, or `content`) |
|
|
|
|
|
### 3. Set Similarity Threshold |
|
|
- **0.9** (default): Good balance between precision and recall |
|
|
- **Higher values** (0.95-0.99): More conservative, only removes very similar texts |
|
|
- **Lower values** (0.7-0.85): More aggressive, may remove semantically similar but different texts |
|
|
|
|
|
### 4. Run Deduplication |
|
|
Click **"Deduplicate"** to start the process. You'll see: |
|
|
- Loading progress for datasets |
|
|
- Deduplication progress |
|
|
- Results with statistics and example duplicates |
|
|
|
|
|
### 5. Push to Hub (New!) |
|
|
After deduplication completes: |
|
|
1. **Log in** with your Hugging Face account using the login button |
|
|
2. Enter a **dataset name** for your cleaned dataset |
|
|
3. Click **"Push to Hub"** to upload the deduplicated dataset |
|
|
|
|
|
The dataset will be saved as `your-username/dataset-name` and be publicly available. |
|
|
|
|
|
|
|
|
## Notes |
|
|
|
|
|
- The app preserves all original columns from the datasets |
|
|
- Only the text similarity is used for deduplication decisions |
|
|
- Deduplicated datasets maintain the same structure as the original |
|
|
- OAuth login is required only for pushing to the Hub, not for deduplication |
|
|
|
