README.md

---

title: Napolab Leaderboard
emoji: 🌎
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.38.2
app_file: app.py
pinned: true
python_version: "3.10"
tags:
  - nlp
  - portuguese
  - benchmarking
  - language-models
  - gradio
datasets:
  - ruanchaves/napolab
  - assin
  - assin2
  - ruanchaves/hatebr
  - ruanchaves/faquad-nli
short_description: "The Natural Portuguese Language Benchmark"
---


# Napolab Leaderboard - Gradio App

A comprehensive Gradio web application for exploring and benchmarking Portuguese language models using the Napolab dataset collection.

## Features

- **🏆 Benchmark Results**: Single comprehensive table with one column per dataset and clickable model links
- **📈 Model Analysis**: Radar chart showing model performance across all datasets
- **ℹ️ About**: Information about Napolab and citation details

## Installation

1. Navigate to the leaderboard directory:
```bash

cd dev/napolab/leaderboard

```

2. Install the required dependencies:
```bash

pip install -r requirements.txt

```

3. Extract data from external sources (optional but recommended):
```bash

# Extract data from Portuguese LLM Leaderboard

python extract_portuguese_leaderboard.py



# Download external models data

python download_external_models.py

```

4. Run the Gradio app:
```bash

python app.py

```

The app will be available at `http://localhost:7860`

## Data Management

The app uses a YAML configuration file (`data.yaml`) for adding new data, making it easy to edit and maintain.

### Data Extraction Scripts

The leaderboard includes scripts to automatically extract and update data from external sources:

#### `extract_portuguese_leaderboard.py`
This script extracts benchmark results from the Open Portuguese LLM Leaderboard:
- Fetches data from the Hugging Face Spaces leaderboard
- Updates the `portuguese_leaderboard.csv` file
- Includes both open-source and proprietary models
- Automatically handles data formatting and validation

#### `download_external_models.py`
This script downloads additional model data:
- Fetches model metadata from various sources
- Updates the `external_models.csv` file
- Includes model links and performance metrics
- Ensures data consistency with the main leaderboard

**Note**: These scripts require internet connection and may take a few minutes to complete. Run them periodically to keep the leaderboard data up to date.

## Usage

### Benchmark Results Tab
- **Single Comprehensive Table**: Shows all models with one column per dataset
- **Dataset Columns**: Each dataset has its own column showing model performance scores
- **Average Column**: Shows the average performance across all datasets for each model
- **Model Column**: Clickable links to Hugging Face model pages
- **Sorted Results**: Models are sorted by overall average performance (descending)

### Model Analysis Tab
- Radar chart showing each model's performance across all datasets
- **Default view**: Shows only bertimbau-large and mdeberta-v3-base models
- **Interactive legend**: Click to show/hide models, double-click to isolate
- Each line represents one model, each point represents one dataset
- Color-coded by model architecture
- Interactive hover information with detailed performance metrics

### Model Hub Tab
- Access links to pre-trained models on Hugging Face
- Models are organized by dataset and architecture type
- Direct links to model repositories

## Supported Datasets

The app includes all Napolab datasets:

- **ASSIN**: Semantic Similarity and Textual Entailment
- **ASSIN 2**: Semantic Similarity and Textual Entailment (v2)
- **Rerelem**: Relational Reasoning
- **HateBR**: Hate Speech Detection
- **Reli-SA**: Religious Sentiment Analysis
- **FaQUaD-NLI**: Factual Question Answering and NLI
- **PorSimplesSent**: Simple Sentences Sentiment Analysis

## Model Architectures

The benchmark includes models based on:
- **mDeBERTa v3**: Multilingual DeBERTa v3
- **BERT Large**: Large Portuguese BERT
- **BERT Base**: Base Portuguese BERT

## Data Management

The app now uses a YAML configuration file (`data.yaml`) for all data, making it easy to edit and maintain.

### Editing Data

Simply edit the `data.yaml` file to:
- Add new datasets
- Update benchmark results
- Add new models
- Modify model metadata

### Data Structure

The YAML file contains four main sections:

1. **datasets**: Information about each dataset
2. **benchmark_results**: Performance metrics for models on datasets

3. **model_metadata**: Model information (parameters, architecture, etc.)
4. **additional_models**: Additional models for the Model Hub



### Data Management Tools



Use the `manage_data.py` script for data operations:



```bash

# Validate the data structure

python manage_data.py validate



# Add a new dataset

python manage_data.py add-dataset \

  --dataset-name "new_dataset" \

  --dataset-display-name "New Dataset" \

  --dataset-description "Description of the dataset" \

  --dataset-tasks "Classification" "Sentiment Analysis" \

  --dataset-url "https://huggingface.co/datasets/new_dataset"



# Add benchmark results

python manage_data.py add-benchmark \

  --dataset-name "assin" \

  --model-name "new-model" \

  --metrics "accuracy=0.92" "f1=0.91"



# Add model metadata

python manage_data.py add-model \

  --model-name "new-model" \

  --parameters 110000000 \

  --architecture "BERT Base" \

  --base-model "bert-base-uncased" \

  --task "Classification" \

  --huggingface-url "https://huggingface.co/new-model"

```



### Customization



To add new datasets or benchmark results:



1. Edit the `data.yaml` file directly, or

2. Use the `manage_data.py` script for structured additions

3. The app will automatically reload the data when restarted



## Troubleshooting



- **Dataset loading errors**: Ensure you have internet connection to access Hugging Face datasets

- **Memory issues**: Reduce the number of samples in the Dataset Explorer

- **Port conflicts**: Change the port in the `app.launch()` call



## Contributing



Feel free to contribute by:

- Adding new datasets

- Improving visualizations

- Adding new features

- Reporting bugs



## License



This project follows the same license as the main Napolab repository.