🦊 JQL: Judging Quality across Languages

🧩 Main Pipeline Steps

1. 📋 Ground Truth Creation: Human annotators label monolingual documents based on a structured instruction prompt. These documents are translated into all target languages to create a multilingual gold-standard dataset. (See Figure 1)
2. 🤖 LLM-as-a-Judge Selection & Data Annotation: Strong multilingual LLMs (e.g., Gemma, Mistral, LLaMA) are evaluated against the ground truth, and top-performing models are used to produce synthetic annotations. (See Figure 1)
3. 🪶 Lightweight Annotator Training: Train compact regression heads on frozen multilingual embeddings to create efficient, high-throughput annotators. (See Figure 1)
4. 🚀 Scalable Data Filtering: Use trained annotators to filter large-scale pretraining corpora using quantile thresholds. (See Figure 1)

📊 Results

• Accuracy: Spearman’s ρ > 0.87 with human ground truth
• Downstream LLM Training:
  • +7.2% benchmark performance improvement
  • +4.8% token retention vs. FineWeb2 heuristic filter
  • Effective threshold strategies: 0.6 and 0.7 quantile
• Annotation Speed: ~11,000 docs/min (A100 GPU, avg. 690 tokens)

📁 Available Artifacts

• ✅ Ground truth annotations in 35 languages
• ✅ Synthetic LLM-annotated dataset (14M+ documents)
• ✅ Lightweight annotation models:
  • JQL-Gemma
  • JQL-Mistral
  • JQL-Llama
• ✅ Training & inference scripts (coming soon)

📜 Citation

If you use JQL, the annotations, or the pretrained annotators, please cite the paper:

@article{your2024jql,
  title={JQL: Judging Quality across Languages},
  author={Your, Name and Collaborators, Here},
  journal={Conference or preprint archive},
  year={2024}
}