|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
<h1 align="center"> |
|
|
MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System |
|
|
</h1> |
|
|
<p align="center"> |
|
|
<a href="https://arxiv.org/abs/2503.09600"> |
|
|
<img alt="arXiv Paper" src="https://img.shields.io/badge/arXiv-Paper-b31b1b.svg?logo=arxiv"> |
|
|
</a> |
|
|
<a href="https://huggingface.co/papers/2503.09600"> |
|
|
<img src="https://img.shields.io/badge/Huggingface-Paper-yellow?style=flat-square&logo=huggingface"> |
|
|
</a> |
|
|
<a href="https://huggingface.co/datasets/Robot2050/Meta-chunker"> |
|
|
<img src="https://img.shields.io/badge/Huggingface-Dataset-FF6F00?style=flat-square&logo=huggingface"> |
|
|
</a> |
|
|
<a href="https://huggingface.co/Robot2050/Meta-chunker-1.5B"> |
|
|
<img src="https://img.shields.io/badge/Model-1.5B 20K-FF6F00?style=flat-square&logo=huggingface"> |
|
|
</a> |
|
|
<a href="https://huggingface.co/Robot2050/Meta-chunker-1.5B-60K"> |
|
|
<img src="https://img.shields.io/badge/Model-1.5B 60K-FF6F00?style=flat-square&logo=huggingface"> |
|
|
</a> |
|
|
<a href="https://opensource.org/license/apache-2-0"> |
|
|
<img alt="Apache 2.0 License" src="https://img.shields.io/badge/License-Apache_2.0-green.svg?logo=apache"> |
|
|
</a> |
|
|
</p> |
|
|
The MoC was fully fine-tuned on the Qwen2.5-1.5B-Instruct utilizing 20K data entries from the CRUD benchmark, which was prepared with GPT-4o. Leveraging the segmented data generated by GPT-4o, we assigned granularity labels ranging from 0 to 3 to the text, corresponding to average chunk length intervals such as (0, 120], (120, 150], (150, 180], and (180, +∞). |
|
|
|
