Add-huggingface-tokenizer

#24
by michaelfeil - opened

Customers of baseten.co are eager to use this model. We however use the trt-llm bindings of tokenizers, where the tekken package is a major cause of friction.

Note: I may recommend to please also publish the artifacts in parallel to the hf style tokenizer library , even as second class citizen.

michaelfeil changed pull request status to open
Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment