Sci-BETO-large

Sci-BETO is a domain-specific RoBERTa encoder pretrained entirely on Spanish scientific texts.

---

Model Description

Sci-BETO-large is a transformer-based encoder following the RoBERTa architecture (355M parameters).
It was pretrained from scratch using byte-level BPE tokenization on a large corpus of Spanish open-access scientific publications, including theses, dissertations, and peer-reviewed papers from Colombian universities and international repositories.

The model was designed to capture scientific discourse, terminology, and abstract reasoning patterns typical of research documents in economics, engineering, medicine, and the social sciences.

Property	Value
Architecture	RoBERTa-large
Parameters	355M
Vocabulary size	50,262
Tokenizer	Byte-Level BPE (trained from scratch)
Pretraining objective	Masked Language Modeling (MLM)
Pretraining steps	92K
Max sequence length	512 tokens
Framework	Transformers

---

Pretraining Data

The pretraining corpus includes over 11 billion tokens from Spanish academic and scientific sources:

• Open-access repositories of Colombian universities (Universidad de los Andes, Universidad Nacional, Universidad Javeriana, Universidad del Rosario).
• CORE API and institutional repositories (theses, dissertations, working papers).
• Tax Statutes in Colombia

The final dataset covers multiple disciplines (economics, medicine, engineering, humanities), ensuring representation across scientific domains.

Source	# Documents	# Words (deduplicated)	Percentage (%)
Universidad de los Andes	33,858	365,752,780	3.23
Universidad Nacional	44,686	537,022,975	4.75
CORE API	2,181,689	9,624,189,002	85.10
Universidad del Rosario	22,404	183,356,109	1.62
Universidad Javeriana	25,624	323,918,445	2.86
Tax Statutes in Colombia	392	13,924,060	0.12
Extra	2	261,131,453	2.31
Total	2,308,655	11,309,295,824	100.00

---

Benchmarks

Sci-BETO was fine-tuned and benchmarked across multiple downstream tasks, both general-domain and scientific:

Dataset	Metric	Sci-BETO Large	Sci-BETO Base	BETO	BERTIN
WikiCAT	F1 (macro)	0.7738	0.7583	0.7624	0.7598
PAWS-X (es)	F1 (macro)	0.9148	0.8794	0.8985	0.8961
PharmaCoNER	F1 (micro)	0.8959	0.8733	0.8845	0.8802
CANTEMIST	F1 (micro)	0.8809	0.8784	0.8954	0.8956
NLI (ESNLI-R)	F1 (micro)	—	—	—	—
BanRep (JEL)	Exact Match	0.6116	0.6043	0.5933	0.5807
Rosario	F1 (macro)	0.9203	0.9194	0.9079	0.9121
Econ-IE	F1 (micro)	0.5256	0.5158	0.5199	0.4992

On average, Sci-BETO achieves comparable or superior results to general-domain Spanish models in specialized contexts (scientific, biomedical, economic), while maintaining strong performance in general text understanding.

---

Intended Use

• Research and experimentation in Spanish scientific NLP.
• Downstream fine-tuning for:
  • Text classification (scientific or academic domains),
  • Named Entity Recognition (NER),
  • Semantic similarity and paraphrase detection,
  • Knowledge extraction from academic documents.

---

Limitations

• The model may underperform on highly informal or non-academic Spanish (e.g., social media).
• It is not designed for generative tasks (e.g., text completion, chat).
• Domain bias toward academic register and Latin American Spanish variants.
• Pretraining corpus excludes English or bilingual data.

---

Example Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Flaglab/Sci-BETO-large")
model = AutoModelForMaskedLM.from_pretrained("Flaglab/Sci-BETO-large")

text = "El Banco de la República va a subir las [mask] de interes."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
masked_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_token = tokenizer.decode(logits[0, masked_index].argmax(dim=-1))
print("Predicted token:", predicted_token)