Update to epoch 2 checkpoint with 3.36 validation perplexity

README.md

@@ -11,22 +11,37 @@ tags:
 datasets:
 - humbleworth/registered-domains
 base_model: google/canine-c
+model-index:
+- name: domain-mlm-epoch-2
+  results:
+  - task:
+      type: fill-mask
+      name: Masked Language Modeling
+    dataset:
+      name: humbleworth/registered-domains
+      type: humbleworth/registered-domains
+      split: validation
+    metrics:
+    - type: perplexity
+      value: 3.36
+      name: Validation Perplexity
 ---
 
-# Domain MLM - CANINE Character-Level Model for Domain Names
+# Domain MLM - CANINE Character-Level Model for Domain Names (Epoch 2)
 
 This model is a CANINE-based character-level language model that has been further pre-trained on domain names using masked language modeling (MLM). It's designed to understand and predict patterns in domain names at the character level.
 
 ## Model Description
 
-This is a checkpoint from epoch 1 of training CANINE-c on domain name data. The model continues pretraining from Google's CANINE-c base model, adapting it specifically to domain name patterns through masked character prediction.
+This is a checkpoint from epoch 2 of training CANINE-c on domain name data. The model continues pretraining from Google's CANINE-c base model, adapting it specifically to domain name patterns through masked character prediction.
 
 ### Key Features
 
 - **Character-level processing**: Works directly with Unicode code points, no tokenization required
 - **Domain-specific**: Pre-trained on 255M registered domain names
-- **Masked Language Modeling**: Trained to predict masked characters in domain names
+- **Masked Language Modeling**: Trained to predict masked characters in domain names (25% masking probability)
 - **Efficient**: 132M parameters, suitable for downstream fine-tuning
+- **Strong Performance**: Achieved 3.36 validation perplexity
 
 ### Architecture
 
@@ -43,11 +58,27 @@ This is a checkpoint from epoch 1 of training CANINE-c on domain name data. The
 - **Training Data**: humbleworth/registered-domains dataset (255M domains)
 - **Training Objective**: Masked Language Modeling (MLM) with 25% masking probability
 - **Masking Strategy**: Mix of contiguous spans (80%) and random characters (20%)
-- **Optimizer**: AdamW with learning rate 1e-5
-- **Batch Size**: 256 per device with gradient accumulation (effective batch size: 512)
-- **Hardware**: Optimized for NVIDIA A100 40GB
+- **Optimizer**: AdamW with learning rate 3e-5, weight decay 0.01
+- **Batch Size**: 512 per device with gradient accumulation steps of 3 (effective batch size: 1,536)
+- **Hardware**: NVIDIA A100 40GB
 - **Mixed Precision**: BF16 automatic mixed precision
 - **Training Framework**: PyTorch with custom training loop
+- **Warmup Steps**: 2,000
+- **Total Steps**: ~830,000 (2 epochs completed at 332,200 steps)
+- **Training Time**: ~36 hours for 2 epochs
+
+### Performance Metrics
+
+**Epoch 2 Results**:
+- **Training Loss**: 1.29
+- **Training Perplexity**: 3.62
+- **Validation Loss**: 1.21
+- **Validation Perplexity**: 3.36
+- **Best Training Perplexity**: 3.49 (achieved during epoch 2)
+- **Processing Speed**: 4,037 samples/second
+- **GPU Memory Usage**: 2.85 GB (highly optimized)
+
+The model shows excellent convergence, improving from an initial perplexity of 10.08 to 3.36 on validation data. The validation perplexity of 3.36 indicates the model effectively narrows down character predictions to approximately 3-4 likely candidates on average.
 
 ## Intended Uses & Limitations
 
@@ -58,14 +89,15 @@ This is a checkpoint from epoch 1 of training CANINE-c on domain name data. The
 - Feature extraction for domain-related tasks
 - Fine-tuning for domain classification tasks
 - Domain name generation (with additional fine-tuning)
+- Character-level anomaly detection in domains
 
 ### Limitations
 
-- This is an early checkpoint (epoch 1) - later checkpoints may perform better
 - Primarily trained on ASCII domain names
-- Limited to domains up to 128 characters
+- Limited to domains up to 64 characters (training max_length)
 - Not suitable for general text understanding tasks
 - Performance on internationalized domain names (IDN) may be limited
+- The model has learned strong biases toward common TLDs (.com, .net, .org)
 
 ## How to Use
 
@@ -76,11 +108,11 @@ import torch
 from transformers import CanineTokenizer, CanineModel, CanineConfig
 
 # Load tokenizer
-tokenizer = CanineTokenizer.from_pretrained('humbleworth/domain-mlm')
+tokenizer = CanineTokenizer.from_pretrained('humbleworth/domain-mlm-epoch-2')
 
 # Load base CANINE model
-config = CanineConfig.from_pretrained('humbleworth/domain-mlm')
-model = CanineModel.from_pretrained('humbleworth/domain-mlm')
+config = CanineConfig.from_pretrained('humbleworth/domain-mlm-epoch-2')
+model = CanineModel.from_pretrained('humbleworth/domain-mlm-epoch-2')
 
 # Encode a domain
 domain = "example.com"
@@ -106,7 +138,7 @@ from train_mlm import CanineForMaskedLM
 
 # Load model with MLM head
 model = CanineForMaskedLM(config)
-model.canine = CanineModel.from_pretrained('humbleworth/domain-mlm')
+model.canine = CanineModel.from_pretrained('humbleworth/domain-mlm-epoch-2')
 
 # Load MLM head weights
 state_dict = torch.load('training_state.bin', map_location='cpu')
@@ -127,6 +159,7 @@ The model was trained on the [humbleworth/registered-domains](https://huggingfac
 - **Source**: [Domains Project](https://domainsproject.org/)
 - **Character Set**: 100% ASCII (no internationalized domains)
 - **Average Length**: 15.9 characters (range: 4-77 characters)
+- **Training/Validation Split**: 99.9% / 0.1%
 
 ### TLD Distribution
 - **Total Unique TLDs**: 1,274
@@ -146,7 +179,23 @@ This comprehensive dataset provides excellent coverage of real-world domain patt
 
 ## Evaluation
 
-This is an intermediate checkpoint. Full evaluation metrics will be available with the final model release. The model achieved reasonable perplexity on the validation set during training.
+### Perplexity Analysis
+
+The model achieved a validation perplexity of **3.36**, which means:
+- The model effectively chooses between ~3.36 possible characters on average at each position
+- This represents excellent performance for domain name modeling
+- The low perplexity indicates strong pattern learning, including:
+  - TLD patterns (high certainty after dots)
+  - Common domain prefixes and suffixes
+  - Valid character sequences in domain names
+
+### Training Progression
+- **Initial**: Loss=2.31, Perplexity=10.08
+- **Epoch 1**: ~4.5-5.0 perplexity (estimated)
+- **Epoch 2**: Loss=1.21, Perplexity=3.36
+- **Best achieved**: Perplexity=3.49 (training), 3.36 (validation)
+
+The model appears to be approaching an asymptotic performance around 3.2-3.5 perplexity, suggesting it has learned most learnable patterns in the domain dataset.
 
 ## Technical Specifications
 
@@ -163,18 +212,20 @@ This is an intermediate checkpoint. Full evaluation metrics will be available wi
 - PyTorch 2.0+
 - Mixed precision training (BF16)
 - Custom training loop implementation
+- Gradient clipping: 1.0
+- Training tracked with Weights & Biases
 
 ## Citation
 
 If you use this model, please cite:
 
 ```bibtex
-@misc{domain-mlm-2024,
+@misc{domain-mlm-2025,
   title={Domain MLM: Character-Level Language Modeling for Domain Names},
   author={humbleworth},
-  year={2024},
+  year={2025},
   publisher={Hugging Face},
-  howpublished={\url{https://huggingface.co/humbleworth/domain-mlm}}
+  howpublished={\url{https://huggingface.co/humbleworth/domain-mlm-epoch-2}}
 }
 ```
 
@@ -186,4 +237,5 @@ This model is released under the Apache 2.0 license.
 
 - Based on Google's CANINE-c model
 - Trained using the humbleworth/registered-domains dataset
-- Optimized training code for NVIDIA A100 GPUs
+- Optimized training code for NVIDIA A100 GPUs
+- Training infrastructure provided by Lambda Labs

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:12ff2a2f7c7820d20880edd8f7521069c57ba546a67a8269c23b8ec26fdd5225
+oid sha256:f475ef3e19f78b48fbbd7dec1c0b888f5bda4e1800f6ca5b1106f799c3acb6e9
 size 528359880