How to increase context length to 256k?

#21

by JC1DA - opened Jun 29

JC1DA

Jun 29

It seems by default it's only 32k. How can I properly increase it to 256k in VLLM?
Thanks

Tencent org Jun 30

Thanks for your support,
we've update the HF's readme doc, one section was added about how to serve model in 256k in vLLM.

JC1DA

Jun 30

Thanks @asherszhang , appreciate it

JC1DA changed discussion status to closed Jun 30

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment