mlx-community

/

ether0-8bit

like 1

Follow

MLX Community 6.48k

Model loads fine, but when running an inference

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        conversation=messages, add_generation_prompt=True
    )
response = generate(
    model=model,
    tokenizer=tokenizer,
    prompt=prompt,
)

I get the error:
[metal::device_info] Cannot get device info without metal backend