File size: 5,974 Bytes
32a6246
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
ABOUT_TEXT = ABOUT_TEXT = """
# πŸ“ About AfroBench Leaderboard

The **AfroBench Leaderboard** is a platform for evaluating multilingual language models across **64 African languages** and over **15 diverse NLP tasks**. These tasks span **classification**, **reasoning**, **question answering**, **summarization**, and **machine translation**, and are grounded in over **22 benchmark datasets** focused on low-resource and underrepresented languages.

The goal of this leaderboard is to:
- 🌍 Highlight the performance of LLMs on African languages.
- πŸ§ͺ Support diagnostic and task-level evaluation across different LLMs.
- βš–οΈ Enable fair comparisons between open-source and closed models using both full and lite subsets of the benchmark.

This leaderboard supports two main views:
- **AfroBench**: The full evaluation benchmark organized by task, subtask, and dataset.
- **AfroBench-Lite**: A lightweight subset of the benchmark with a consistent set of languages across tasks, designed for efficient evaluation.

Each score is computed as the average across all selected columns and views, allowing flexible filtering and analysis.

---

## πŸ”— More Information

To learn more about the benchmark, datasets, task definitions, and evaluation procedures, please visit the official project site:  
πŸ‘‰ [AfroBench Website](https://mcgill-nlp.github.io/AfroBench/index.html)

You can also explore:
- πŸ“„ [AfroBench Paper on arXiv](https://arxiv.org/abs/2311.07978)  
- πŸ§‘πŸ½β€πŸ’» [AfroBench GitHub Repository](https://github.com/McGill-NLP/AfroBench)

"""


SUBMISSION_TEXT = """
<h1 align="center">
How to submit models/results to the leaderboard?
</h1>
We welcome the community to submit evaluation results of new models. We also provide an experiental feature for submitting models that our team will evaluate on the πŸ€— cluster.

## Submitting Models (experimental feature)
Inspired from the Open LLM Leaderboard, we welcome code models submission from the community that will be automatically evaluated. Please note that this is still an experimental feature.
Below are some guidlines to follow before submitting your model:

#### 1) Make sure you can load your model and tokenizer using AutoClasses:
```python
from transformers import AutoConfig, AutoModel, AutoTokenizer
config = AutoConfig.from_pretrained("your model name", revision=revision)
model = AutoModel.from_pretrained("your model name", revision=revision)
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
```
If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.
Note: make sure your model is public!
Note: if your model needs `use_remote_code=True`, we do not support this option yet.
#### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`!
#### 3) Make sure your model has an open license!
This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model πŸ€—
#### 4) Fill up your model card
When we add extra information about models to the leaderboard, it will be automatically taken from the model card.
"""

SUBMISSION_TEXT_2 = """
## Sumbitting Results
You also have the option for running evaluation yourself and submitting results. These results will be added as non-verified, the authors are however required to upload their generations in case other members want to check.

### 1 - Running Evaluation

We wrote a detailed guide for running the evaluation on your model. You can find the it in [bigcode-evaluation-harness/leaderboard](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/leaderboard). This will generate a json file summarizing the results, in addition to the raw generations and metric files.

### 2- Submitting Results πŸš€

To submit your results create a **Pull Request** in the community tab to add them under the [folder](https://huggingface.co/spaces/bigcode/multilingual-code-evals/tree/main/community_results) `community_results` in this repository:
- Create a folder called `ORG_MODELNAME_USERNAME` for example `bigcode_starcoder_loubnabnl`
- Put your json file with grouped scores from the guide, in addition generations folder and metrics folder in it.

The title of the PR should be `[Community Submission] Model: org/model, Username: your_username`, replace org and model with those corresponding to the model you evaluated.
"""
SUBMISSION_TEXT_3 = """
<h1 align="center">
How to submit models/results to the leaderboard?
</h1>
We welcome the community to submit evaluation results of new models. These results will be added as non-verified, the authors are however required to upload their generations in case other members want to check.

### 1 - Running Evaluation

We wrote a detailed guide for running the evaluation on your model. You can find the it in [bigcode-evaluation-harness/leaderboard](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/leaderboard). This will generate a json file summarizing the results, in addition to the raw generations and metric files.

### 2- Submitting Results πŸš€

To submit your results create a **Pull Request** in the community tab to add them under the [folder](https://huggingface.co/spaces/bigcode/multilingual-code-evals/tree/main/community_results) `community_results` in this repository:
- Create a folder called `ORG_MODELNAME_USERNAME` for example `bigcode_starcoder_loubnabnl`
- Put your json file with grouped scores from the guide, in addition generations folder and metrics folder in it.

The title of the PR should be `[Community Submission] Model: org/model, Username: your_username`, replace org and model with those corresponding to the model you evaluated.
"""