Model is not supporting by TGI
#1
by
maheshbabu9199
- opened
when trying to load this model using TGI, am getting the following which says
RuntimeError: [FT][ERROR] Invalid shape for quantized tensor. Number of rows of quantized matrix must be a multiple of 16 Assertion fail: /build/source/cutlass_kernels/cutlass_preprocessors.cc:164
text-generation-inference | 2025-05-27T08:01:55.437641Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output.
Anyone working on TGI, with this model.?
maheshbabu9199
changed discussion title from
model is supported by tgi
to Model is supporting by TGI
maheshbabu9199
changed discussion title from
Model is supporting by TGI
to Model is not supporting by TGI