Can I run this efficiently at scale on 8x h200?
#5
by
Movsai
- opened
It seems bigger than my 1,5tb vram?
Hi, Movsai, thanks so much for asking. AFAIK it is not going to fit as for bf16 it requires roughly 2T. For fp8 theoretically 1T would be ok, but in reality there'll be intermediate results generated during inference, so it is less likely to work flawlessly. 😥
So bf16 is pretty much 8x B300 only? Have you tried fp8 on 8x B200? That's a bit bigger than H200 (though I think 8x H200 is closer to 1.2tb vram than OP's 1.5tb vram).