RuntimeError: Cannot load `awq` weight, make sure the model is already quantized

#17
by markba - opened

I run this model in TGI with all defaults and receive this error:
RuntimeError: Cannot load awq weight, make sure the model is already quantized.
The machine is AWS g6.12xlarge - 4 nvidia GPU with 96gb of GPU memory.

This comment has been hidden (marked as Abuse)

Sign up or log in to comment