Avery Yen's picture

1

Avery Yen

yen-av

·

AI & ML interests

None yet

Recent Activity

reacted to codelion's post with 🔥 about 3 hours ago

Introducing Dhara-70M: A diffusion language model that achieves 3.8x higher throughput than autoregressive models! Key findings from our research on optimal architectures for small language models: → Depth beats width: 32 layers outperforms 12 layers at the same parameter count → Best-in-class factuality: 47.5% on TruthfulQA → 10x training efficiency using WSD (Warmup-Stable-Decay) conversion → Canon layers add only 0.13% parameters but improve reasoning We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens. Blog: https://huggingface.co/blog/codelion/optimal-model-architecture Model: https://huggingface.co/codelion/dhara-70m

upvoted a paper 15 days ago

Eliciting Secret Knowledge from Language Models

updated a dataset about 1 month ago

yen-av/tunix-stem-sft

View all activity

Organizations

yen-av 's datasets 1

yen-av/tunix-stem-sft

Viewer • Updated Nov 26 • 1.11M • 76