Auron-1.1B (Archived — Scaling Wall)

Note: This model demonstrates a scaling limitation in Ouroboros weight sharing. Despite 4x more parameters than the 279M, it converges to nearly identical val_loss (3.180 vs 3.188). At dim=2048 with head_dim=64, the representation is wide enough for a single pass — shared loops become an echo chamber rather than iterative refinement.

For inference and testing, use Auron-510M (val_loss 3.035).

Model	Params	Final Val Loss	Scaling
Auron-279M	279M	3.188	Baseline
Auron-510M	510M	3.035	-0.153 (good)
Auron-1.1B	1.1B	3.180	+0.145 (regression)

Paper: Auron | Code: github.com/Fy-/Auron | Blog: HuggingFace

The Scaling Wall

• Root cause: Representation saturation at dim=2048 — loops add no new information
• Contributing: head_dim=64 produces 32 fragmented attention heads (Qwen 3.5 uses 256)
• Fix in progress: Chimera 1B v2 (head_dim=128) + Chimera-MoE (routed experts)

Architecture

• Type: Chimera (6 bottom + 6×3 top = 24 virtual)
• Dim: 2048, head_dim=64, expand_v=2
• Params: 1.1B (761M unique + 311M embed)
• Trained: 250K steps, 5B tokens, WSD schedule

from ouro import load_model, generate
model, tokenizer, device = load_model("nyxia/Auron-510M")  # Use 510M
generate(model, tokenizer, device, "The history of")

Built by Florian Gasquez (@nyxia). Part of Soulkyn.