| | --- |
| | language: |
| | - tr |
| | license: apache-2.0 |
| | tags: |
| | - turkish |
| | - diffusion |
| | - masked-diffusion |
| | - non-autoregressive |
| | - foundation-model |
| | - dllm |
| | datasets: |
| | - turkish-nlp-suite/Havadis |
| | - turkish-nlp-suite/temiz-OSCAR |
| | - wikimedia/wikipedia |
| | metrics: |
| | - perplexity |
| | --- |
| | |
| | # DiffutronLM-0.3B-Base |
| |
|
| | **DiffutronLM-0.3B-Base** is the foundational Masked Diffusion Language Model (MDLM) of the Diffutron series, tailored specifically for the Turkish language. |
| |
|
| | This model represents the completion of the **Continual Pre-training (CPT)** phase. It has successfully adapted the multilingual representations of its backbone to the agglutinative complexity and morphological nuances of Turkish. |
| |
|
| | ⚠️ **Note:** This is a base foundation model. It has **not** been instruction-tuned or aligned for chat capabilities. If you are looking for a model that follows prompts and answers questions, please use `DiffutronLM-0.3B-Instruct`. |
| |
|
| | ## 📌 Model Details |
| |
|
| | * **Model Type:** Masked Diffusion Language Model (MDLM) Base |
| | * **Base Architecture:** `jhu-clsp/mmBERT-base` (Multilingual Encoder) |
| | * **Language:** Turkish |
| | * **Parameter Count:** 307M (0.3B) |
| | * **Context Length:** 512 tokens |
| | * **Training Libraries:** `dllm`, PyTorch |
| | * **Status:** Foundation / Base Model (Post-CPT) |
| |
|
| | ## 🚀 Architecture & Continual Pre-training (CPT) |
| |
|
| | Unlike standard autoregressive models, Diffutron models text generation as a discrete diffusion process. To align the base encoder's latent space with the Turkish target distribution while preserving cross-lingual reasoning, this model underwent a specialized CPT pipeline: |
| |
|
| | * **Data Curation:** Trained on a composite dataset of approximately 2 million sequences (max length 512) sourced from: |
| | * **Havadis:** Comprehensive Turkish news articles. |
| | * **Temiz-OSCAR:** A cleaned, filtered subset of the Common Crawl-based Turkish OSCAR corpus. |
| | * **Turkish Wikipedia:** High-quality encyclopedic sequences. |
| | * **Efficient Adaptation via LoRA:** Instead of full-parameter fine-tuning which risks catastrophic forgetting, we applied Low-Rank Adaptation (LoRA) with a high rank ($r=256$, $\alpha=256$) targeting all linear modules (Attention Q, K, V, O and MLP Input, Output). This resulted in ~14.94% trainable parameters. |
| | * **Objective:** Masked Language Modeling (MLM). |
| |
|
| | ## 📊 Intrinsic Evaluation |
| |
|
| | To quantify the improvements gained from the CPT phase, we conducted an intrinsic evaluation using perplexity on the **Bilkent Turkish Writings Dataset** (evaluated with a masked language modeling probability of 0.15). |
| |
|
| | The CPT process resulted in a significant reduction in perplexity, indicating a strong alignment with Turkish linguistic structures: |
| |
|
| | * **jhu-clsp/mmBERT-base (Pre-CPT):** 3.42 |
| | * **DiffutronLM-0.3B-Base (Post-CPT):** **2.75** |
| |
|
| | *(Note: Downstream task evaluations on the CETVEL benchmark were conducted on the Instruct-tuned versions of this model.)* |
| |
|
| | ## 💻 Usage |
| |
|
| | As a base masked diffusion model, this checkpoint is ideal for: |
| | 1. **Further Fine-tuning:** Acting as a starting point for domain-specific continued pre-training or custom instruction tuning. |
| | 2. **Masked Token Prediction:** Filling in blanks or reconstructing corrupted text. |
| | 3. **Unconditional/Conditional Generation:** Generating text using a discrete diffusion sampling loop (e.g., via the `dllm` library). |
| |
|
| | Because it uses a non-autoregressive paradigm, standard `AutoModelForCausalLM.generate()` pipelines will not work. Please utilize discrete diffusion generation strategies. |
| |
|
| | ## ⚠️ Limitations |
| |
|
| | * **No Instruction Tuning:** Will not respond to QA prompts or instructions naturally. |
| | * **Multilingual Backbone:** While heavily adapted to Turkish, it is built upon a multilingual encoder. |
| | * **Context Window:** Restricted to a 512-token context window during the base phase. |
| |
|
| | ## 📝 Citation |
| |
|
| | ```bibtex |
| | @misc{diffutron2026, |
| | author = {Kocabay, Şuayp Talha and Akkuş, Talha Rüzgar}, |
| | title = {Diffutron: A Masked Diffusion Language Model for Turkish Language}, |
| | year = {2026}, |
| | publisher = {Hugging Face}, |
| | howpublished = {\url{[https://huggingface.co/collections/diffutron/diffutronlm](https://huggingface.co/collections/diffutron/diffutronlm)}} |
| | } |
