ValueError: GGUF magic invalid
Hello, thanks so much for sharing these scripts. I'm really looking forward to experimenting with control vectors.
I tried to run test_inference.py following your instructions. I have model folder Llama-3.1-8B-Instruct-exl2-5bpw and folder Llama-3.1-8B-Instruct-exl2-5bpw-vectors in my "models" folder. I have llama-3.1_8b-humility_vs_narcissism__narcissism.gguf in my vectors folder.
I tried running
$ python test_inference.py -m models/Llama-3.1-8B-Instruct-exl2-5bpw -p "what is your favourite activity?" --control_vectors narcissism:1
and I get this output:
-- Model: models/Llama-3.1-8B-Instruct-exl2-5bpw
-- Options: []
Loading: models/Llama-3.1-8B-Instruct-exl2-5bpw ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:04 0:00:00
-- Loaded model in 4.4543 seconds
-- Loading tokenizer...
Loaded control vector: humility_vs_narcissism, Direction: narcissism
Traceback (most recent call last):
File "/home/xxx/ExllamaV2-Control-Vectors/test_inference.py", line 113, in <module>
ExLlamaV2ModuleWrapper.wrap(model, args.control_vectors)
File "/home/xxx/ExllamaV2-Control-Vectors/exl2_wrapper.py", line 13, in wrap
reader = GGUFReader(file)
^^^^^^^^^^^^^^^^
File "/home/xxx/ExllamaV2-Control-Vectors/gguf/gguf_reader.py", line 96, in __init__
raise ValueError('GGUF magic invalid')
ValueError: GGUF magic invalid
Am I doing something wrong?
I got it working! The gguf_reader.py was, understandably, outdated, and couldn't read GGUF v3 files. I swapped out the gguf_reader.py to the latest version on https://github.com/ggml-org/llama.cpp/blob/master/gguf-py/gguf/gguf_reader.py, and it works wonderfully.
Thanks for sharing this script, I'm fascinated by the responses I'm seeing using control vectors!