tencent/HY-Embodied-0.5
Image-Text-to-Text • 4B • Updated • 723 • 349
Kiều Sơn Tùng
codemaivanngu
AI & ML interests
None yet
Recent Activity
liked a model about 15 hours ago
tencent/HY-Embodied-0.5 liked a model about 15 hours ago
Qwen/Qwen3.5-9B upvoted a paper about 15 hours ago
Kronos: A Foundation Model for the Language of Financial MarketsOrganizations
None yet
liked 2
models about 15 hours ago
upvoted a paper about 15 hours ago
updated a
dataset about 20 hours ago
liked a
model 2 days ago
published a
dataset 12 days ago
published a
dataset 3 months ago
updated a
dataset 4 months ago
published a
dataset 4 months ago
updated a
dataset 4 months ago
published a
dataset 4 months ago
upvoted a paper 6 months ago
upvoted an article 7 months ago
Article
🪆 Introduction to Matryoshka Embedding Models
- +1
•
202
reactedto Jaward's post with 🔥 8 months ago
Post
3301
Towards batch sizes too small to meter🎉 beautiful work! And my personal favorite so far - I adore peak performance at small/nano scale. Everyone deserves to run/train AGI locally:) our data, our god model!
They showed that:
- you can train LLMs (upto 1B params) with as low as batch_size=1. This is unconventional given small batch sizes can lead to unstable/spiky training runs.
- you can have a stable train run with just vanilla SGD(stochastic gradient descent), no momentum required🤯
- small batch sizes are more robust to hyperparameters (i.e no worries with initialization)
- smaller batch sizes outperforms (“better per-Flops performance”) larger batch sizes.
“We recommend that practitioners training large models in memory-constrained settings exploit the benefits of small batch sizes rather than trying to emulate the large batch size setting (e.g., through gradient accumulation) typically used in industry.”
I’ve been doing this for ages - my mantra: all my experiments must scale on my 8gb ram m2 before moving to gpu. IOW I love being gpu poor. Checkout my nanoAI algo repo: https://github.com/Jaykef/ai-algorithms, all notebooks run on memory as low as 8gb ram
They showed that:
- you can train LLMs (upto 1B params) with as low as batch_size=1. This is unconventional given small batch sizes can lead to unstable/spiky training runs.
- you can have a stable train run with just vanilla SGD(stochastic gradient descent), no momentum required🤯
- small batch sizes are more robust to hyperparameters (i.e no worries with initialization)
- smaller batch sizes outperforms (“better per-Flops performance”) larger batch sizes.
“We recommend that practitioners training large models in memory-constrained settings exploit the benefits of small batch sizes rather than trying to emulate the large batch size setting (e.g., through gradient accumulation) typically used in industry.”
I’ve been doing this for ages - my mantra: all my experiments must scale on my 8gb ram m2 before moving to gpu. IOW I love being gpu poor. Checkout my nanoAI algo repo: https://github.com/Jaykef/ai-algorithms, all notebooks run on memory as low as 8gb ram
upvoted a collection 8 months ago
liked a
model 10 months ago
published a
Space 10 months ago
liked 2
datasets 11 months ago
updated a
model 11 months ago