|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- tiiuae/falcon-refinedweb |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- perplexity |
|
|
- accuracy |
|
|
--- |
|
|
## Description |
|
|
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
Models trained on 300B tokens, including dense FFN ones and low-rank FFN ones. |
|
|
|
|
|
|
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find it useful, please consider citing the paper: |
|
|
|
|
|
``` |
|
|
@article{wei2024building, |
|
|
title={Building on efficient foundations: Effective training of LLMs with structured feedforward layers}, |
|
|
author={Wei, Xiuying and Moalla, Skander and Pascanu, Razvan and Gulcehre, Caglar}, |
|
|
journal={Advances in Neural Information Processing Systems}, |
|
|
volume={37}, |
|
|
pages={4689--4717}, |
|
|
year={2024} |
|
|
} |
|
|
|
|
|
@article{wei2024investigating, |
|
|
title={Investigating low-rank training in transformer language models: Efficiency and scaling analysis}, |
|
|
author={Wei, Xiuying and Moalla, Skander and Pascanu, Razvan and Gulcehre, Caglar}, |
|
|
journal={arXiv preprint arXiv:2407.09835}, |
|
|
year={2024} |
|
|
} |
|
|
``` |
