|
|
--- |
|
|
library_name: transformers |
|
|
datasets: |
|
|
- WebOrganizer/TopicAnnotations-Llama-3.1-8B |
|
|
- WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8 |
|
|
base_model: |
|
|
- answerdotai/ModernBERT-base |
|
|
--- |
|
|
# wissamantoun/WebOrganizer-TopicClassifier-ModernBERT |
|
|
|
|
|
[[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)] |
|
|
|
|
|
*All credit goes to the original authors of the model and dataset. This is a retraining of the original model with a different base model* |
|
|
|
|
|
The TopicClassifier organizes web content into 17 categories based on the URL and text contents of web pages. |
|
|
The model is a [ModernBERT-base](answerdotai/ModernBERT-base) with 140M parameters fine-tuned on the following training data: |
|
|
|
|
|
1. [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B): 1M documents annotated by Llama-3.1-8B (first-stage training) |
|
|
2. [WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8): 100K documents annotated by Llama-3.1-405B-FP8 (second-stage training) |
|
|
|
|
|
#### All Domain Classifiers |
|
|
- [wissamantoun/WebOrganizer-FormatClassifier-ModernBERT](https://huggingface.co/wissamantoun/WebOrganizer-FormatClassifier-ModernBERT) |
|
|
- [wissamantoun/WebOrganizer-TopicClassifier-ModernBERT](https://huggingface.co/wissamantoun/WebOrganizer-TopicClassifier-ModernBERT) *← you are here!* |
|
|
|
|
|
## Usage |
|
|
|
|
|
This classifier expects input in the following input format: |
|
|
``` |
|
|
{url} |
|
|
|
|
|
{text} |
|
|
``` |
|
|
|
|
|
Example: |
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("wissamantoun/wissamantoun/WebOrganizer-TopicClassifier-ModernBERT") |
|
|
model = AutoModelForSequenceClassification.from_pretrained( |
|
|
"wissamantoun/wissamantoun/WebOrganizer-TopicClassifier-ModernBERT", |
|
|
trust_remote_code=True, |
|
|
use_memory_efficient_attention=False) |
|
|
|
|
|
web_page = """http://www.example.com |
|
|
|
|
|
How to build a computer from scratch? Here are the components you need...""" |
|
|
|
|
|
inputs = tokenizer([web_page], return_tensors="pt") |
|
|
outputs = model(**inputs) |
|
|
|
|
|
probs = outputs.logits.softmax(dim=-1) |
|
|
print(probs.argmax(dim=-1)) |
|
|
# -> 5 ("Hardware" topic) |
|
|
``` |
|
|
|
|
|
You can convert the `logits` of the model with a softmax to obtain a probability distribution over the following 24 categories (in order of labels, also see `id2label` and `label2id` in the model config): |
|
|
1. Adult |
|
|
2. Art & Design |
|
|
3. Software Dev. |
|
|
4. Crime & Law |
|
|
5. Education & Jobs |
|
|
6. Hardware |
|
|
7. Entertainment |
|
|
8. Social Life |
|
|
9. Fashion & Beauty |
|
|
10. Finance & Business |
|
|
11. Food & Dining |
|
|
12. Games |
|
|
13. Health |
|
|
14. History |
|
|
15. Home & Hobbies |
|
|
16. Industrial |
|
|
17. Literature |
|
|
18. Politics |
|
|
19. Religion |
|
|
20. Science & Tech. |
|
|
21. Software |
|
|
22. Sports & Fitness |
|
|
23. Transportation |
|
|
24. Travel |
|
|
|
|
|
The full definitions of the categories can be found in the [taxonomy config](https://github.com/CodeCreator/WebOrganizer/blob/main/define_domains/taxonomies/topics.yaml). |
|
|
|
|
|
# Scores |
|
|
``` |
|
|
***** pred metrics ***** |
|
|
test_accuracy = 0.8585 |
|
|
test_accuracy__0 = 0.9346 |
|
|
test_accuracy__1 = 0.7317 |
|
|
test_accuracy__10 = 0.9148 |
|
|
test_accuracy__11 = 0.8927 |
|
|
test_accuracy__12 = 0.8687 |
|
|
test_accuracy__13 = 0.814 |
|
|
test_accuracy__14 = 0.8616 |
|
|
test_accuracy__15 = 0.7179 |
|
|
test_accuracy__16 = 0.855 |
|
|
test_accuracy__17 = 0.8246 |
|
|
test_accuracy__18 = 0.907 |
|
|
test_accuracy__19 = 0.8333 |
|
|
test_accuracy__2 = 0.866 |
|
|
test_accuracy__20 = 0.8294 |
|
|
test_accuracy__21 = 0.9441 |
|
|
test_accuracy__22 = 0.8788 |
|
|
test_accuracy__23 = 0.9 |
|
|
test_accuracy__3 = 0.847 |
|
|
test_accuracy__4 = 0.8442 |
|
|
test_accuracy__5 = 0.8189 |
|
|
test_accuracy__6 = 0.8997 |
|
|
test_accuracy__7 = 0.7295 |
|
|
test_accuracy__8 = 0.8937 |
|
|
test_accuracy__9 = 0.8665 |
|
|
test_accuracy_conf50 = 0.8674 |
|
|
test_accuracy_conf50__0 = 0.9434 |
|
|
test_accuracy_conf50__1 = 0.7453 |
|
|
test_accuracy_conf50__10 = 0.93 |
|
|
test_accuracy_conf50__11 = 0.8958 |
|
|
test_accuracy_conf50__12 = 0.8768 |
|
|
test_accuracy_conf50__13 = 0.8193 |
|
|
test_accuracy_conf50__14 = 0.8691 |
|
|
test_accuracy_conf50__15 = 0.7237 |
|
|
test_accuracy_conf50__16 = 0.864 |
|
|
test_accuracy_conf50__17 = 0.8358 |
|
|
test_accuracy_conf50__18 = 0.91 |
|
|
test_accuracy_conf50__19 = 0.8481 |
|
|
test_accuracy_conf50__2 = 0.8768 |
|
|
test_accuracy_conf50__20 = 0.8434 |
|
|
test_accuracy_conf50__21 = 0.9505 |
|
|
test_accuracy_conf50__22 = 0.8844 |
|
|
test_accuracy_conf50__23 = 0.9028 |
|
|
test_accuracy_conf50__3 = 0.8571 |
|
|
test_accuracy_conf50__4 = 0.851 |
|
|
test_accuracy_conf50__5 = 0.8206 |
|
|
test_accuracy_conf50__6 = 0.9071 |
|
|
test_accuracy_conf50__7 = 0.7442 |
|
|
test_accuracy_conf50__8 = 0.9006 |
|
|
test_accuracy_conf50__9 = 0.8761 |
|
|
test_accuracy_conf75 = 0.9178 <--- Metric from the paper |
|
|
test_accuracy_conf75__0 = 0.95 |
|
|
test_accuracy_conf75__1 = 0.8413 |
|
|
test_accuracy_conf75__10 = 0.9556 |
|
|
test_accuracy_conf75__11 = 0.9298 |
|
|
test_accuracy_conf75__12 = 0.9299 |
|
|
test_accuracy_conf75__13 = 0.8788 |
|
|
test_accuracy_conf75__14 = 0.9126 |
|
|
test_accuracy_conf75__15 = 0.8253 |
|
|
test_accuracy_conf75__16 = 0.8885 |
|
|
test_accuracy_conf75__17 = 0.8968 |
|
|
test_accuracy_conf75__18 = 0.938 |
|
|
test_accuracy_conf75__19 = 0.9113 |
|
|
test_accuracy_conf75__2 = 0.9029 |
|
|
test_accuracy_conf75__20 = 0.8966 |
|
|
test_accuracy_conf75__21 = 0.968 |
|
|
test_accuracy_conf75__22 = 0.9225 |
|
|
test_accuracy_conf75__23 = 0.9444 |
|
|
test_accuracy_conf75__3 = 0.9319 |
|
|
test_accuracy_conf75__4 = 0.8976 |
|
|
test_accuracy_conf75__5 = 0.9167 |
|
|
test_accuracy_conf75__6 = 0.9483 |
|
|
test_accuracy_conf75__7 = 0.804 |
|
|
test_accuracy_conf75__8 = 0.9448 |
|
|
test_accuracy_conf75__9 = 0.932 |
|
|
test_accuracy_label_average = 0.8531 |
|
|
test_accuracy_label_average_conf50 = 0.8615 |
|
|
test_accuracy_label_average_conf75 = 0.9111 |
|
|
test_accuracy_label_min = 0.7179 |
|
|
test_accuracy_label_min_conf50 = 0.7237 |
|
|
test_accuracy_label_min_conf75 = 0.804 <--- Metric from the paper |
|
|
test_loss = 0.4694 |
|
|
test_proportion_conf50 = 0.9811 |
|
|
test_proportion_conf75 = 0.8535 |
|
|
test_runtime = 0:00:08.39 |
|
|
test_samples_per_second = 1191.144 |
|
|
test_steps_per_second = 37.283 |
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
## Citation |
|
|
```bibtex |
|
|
@article{wettig2025organize, |
|
|
title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation}, |
|
|
author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini}, |
|
|
journal={arXiv preprint arXiv:2502.10341}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
