Tokenizer JSON
#17
by
rageltman
- opened
Most of the Candle ecosystem inferencing engines want to parse a tokenizer.json file - Error: request error: https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/resolve/main/tokenizer.json: status code 404, is there any chance that one can be generated or do we need to pull in tiktoken libraries to communicate with this derivation?
I use this to convert json for llama.cpp
https://gist.github.com/cturan/20c332b6111cb37a72abb5f3742ff996