Add-huggingface-tokenizer
#24
by
michaelfeil
- opened
Customers of baseten.co are eager to use this model. We however use the trt-llm bindings of tokenizers, where the tekken package is a major cause of friction.
Note: I may recommend to please also publish the artifacts in parallel to the hf style tokenizer library , even as second class citizen.
michaelfeil
changed pull request status to
open