tiny_encoder

A pre-trained IterativeBert encoder from the TiNER (Tiny Recursive NER) project.

Model Description

IterativeBert is a novel encoder architecture that applies a single transformer layer iteratively for deep representation learning with minimal parameters.

Architecture Details

Parameter	Value
Hidden Size	312
Attention Heads	6
L Cycles (Refinement)	8
Residual Mode	add
Vocab Size	30522
Max Position Embeddings	2048

Usage

from tiner.model import IterativeBert

# Load the model
encoder = IterativeBert.from_pretrained("paul-english/tiny_encoder")

# Use for encoding
outputs = encoder(input_ids, attention_mask=attention_mask)
hidden_states = outputs.last_hidden_state

Training Details

Training details not provided.

Limitations

• This is a pre-trained encoder, not a complete NER model
• Best used as a backbone for fine-tuning on downstream tasks
• Sequence length limited to 2048 tokens

Citation

If you use this model, please cite:

@software{tiner,
  title = {TiNER: Tiny Recursive Named Entity Recognition},
  url = {https://github.com/your-username/tiner}
}