Pro Creations's picture

4 5 10

Pro Creations PRO

ProCreations

·

https://pro-ai.glitch.me

ProCreations-Offical

AI & ML interests

AGI and small scale-high quality AI

Recent Activity

replied to Ruurd's post 1 day ago

The past year I have been trying to get diffusion models to work for language generation, without having to retrain a LLM from scratch. And recently, we finally succeeded: We introduce "LAD: LoRA-Adapted Denoiser", a method to convert a LLaMA model into a text diffusion model using LoRA finetuning and structured input corruption. 🎯 Try the demo and read the write-up here! https://ruurdkuiper.github.io/tini-lad/ Unlike autoregressive (word-for-word) models like ChatGPT, diffusion models iteratively refine a noised sequence. However, most current diffusion approaches rely on all-parameter retraining and repeatedly remasking tokens, which is costly and slow during both training and inference! 🧠 With LAD: - We can finetune an autoregressive model for diffusive generation in just 10 hours on a single GPU. - Test-time compute is fully adjustable: fewer steps means faster outputs while more steps improve output quality. - Due to our unique noising schedule, remasking is not always needed during inference. All tokens are attended to in each iteration! 🔍 LAD is built using: – A frozen LLaMA-8B backbone – Structured noising: token swaps, duplications, replacements, span shifts – Modified attention masks for bidirectional decoding 💡 We show that even small, fast-trained models can perform diffusive generation — with competitive benchmark performance, perplexity and more flexible test-time behavior than traditional transformers.

liked a dataset 1 day ago

ProCreations/Ultra-FineWeb-EDU

posted an update 2 days ago

NEW DATASETTTTT https://huggingface.co/datasets/ProCreations/Ultra-FineWeb-EDU This dataset is an extremely filtered version of https://huggingface.co/datasets/openbmb/Ultra-FineWeb for educational content only, inspired by https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu Right now dataset size is small (64k examples) and more examples are being processed as you read this!

View all activity

Organizations

ProCreations's activity

upvoted a collection 26 days ago

Qwen3

40 items • Updated 18 days ago • 740

upvoted a paper about 1 month ago

70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float

Paper • 2504.11651 • Published Apr 15 • 28

upvoted a collection about 2 months ago

Intellite

Intellectually lite • 3 items • Updated 12 days ago • 2

upvoted a collection about 1 year ago

Phi-3

Phi-3 family of small language and multi-modal models. Language models are available in short- and long-context lengths. • 26 items • Updated May 1 • 569