Legacy tokenizer.json and tokenizer_config.json

#10
by rageltman - opened

Runtimes like candle-vllm rely on the older tokernizer.json files to acquire and load the model. Working on adding a tekken parser but in the meantime, could someone please add the legacy json files alongside the tekken.json?

Sign up or log in to comment