|
|
--- |
|
|
title: README |
|
|
emoji: 😻 |
|
|
colorFrom: indigo |
|
|
colorTo: indigo |
|
|
sdk: static |
|
|
pinned: false |
|
|
--- |
|
|
|
|
|
# EvoEval: Evolving Coding Benchmarks via LLM |
|
|
|
|
|
**EvoEval**<sup>1</sup> is a holistic benchmark suite created by _evolving_ **HumanEval** problems: |
|
|
- 🔥 Contains **828** new problems across **5** 🌠 semantic-altering and **2** ⭐ semantic-preserving benchmarks |
|
|
- 🔮 Allows evaluation/comparison across different **dimensions** and problem **types** (i.e., _Difficult_, _Creative_ or _Tool Use_ problems). See our [**visualization tool**](https://evo-eval.github.io/visualization.html) for ready-to-use comparison |
|
|
- 🏆 Complete with [**leaderboard**](https://evo-eval.github.io/leaderboard.html), **groundtruth solutions**, **robust testcases** and **evaluation scripts** to easily fit into your evaluation pipeline |
|
|
- 🤖 Generated LLM code samples from **>50** different models to save you time in running experiments |
|
|
|
|
|
<sup>1</sup> coincidentally similar pronunciation with 😈 EvilEval |
|
|
|
|
|
- GitHub: [evo-eval/evoeval](https://github.com/evo-eval/evoeval) |
|
|
- Webpage: [evo-eval.github.io](https://evo-eval.github.io/) |
|
|
- Leaderboard: [evo-eval.github.io/leaderboard.html](https://evo-eval.github.io/leaderboard.html) |
|
|
- Visualization: [evo-eval.github.io/visualization.html](https://evo-eval.github.io/visualization.html) |
|
|
- Paper: [arXiv](https://arxiv.org/abs/2403.19114) |
|
|
- PyPI: [evoeval](https://pypi.org/project/evoeval/) |
|
|
|
