File size: 832 Bytes
9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 9189966 887c9c5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
---
datasets:
- HuggingFaceFW/fineweb
language:
- en
---
# Encoder-Decoder model with DeBERTa encoder
## pre-trained models
- `deliciouscat/deberta-v3-base-encoder-decoder-v0.2`
-> 297511524(298M) params
## Data used
- `HuggingFaceFW/fineweb`
- AiHub ko-en translation corpus (English part)
- Some papers that I kept
## Training hparams
- optimizer: AdamW, lr=3e-5, betas=(0.875, 0.997)
- batch size: 12
-> training on denoising objective (BART), 29523 step
## How to use
```
from transformers import AutoTokenizer, EncoderDecoderModel
model = EncoderDecoderModel.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")
tokenizer = AutoTokenizer.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")
```
## Future work!
- train more scientific data
- fine-tune on keyword extraction task |