what’s the real latency like in streaming mode?
#5
by
abdullahfde
- opened
How long does it take to get the first audio chunk on a GPU ?
Also, does Kani actually stream audio as it generates text, or only after finishing the whole sentence?
On bare metal rtx 5090 it achieves realtime (~300ms). We are working on optimizations rn. Yes it generates stream as it generates text, thanks to super fast codec. No need to wait all speech tokens to be spitted out of llm