[Feature request] Dynamic splitfuse from Deepspeed (2x throughput)

[0xymoro]

Hi, putting this here:
https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen

The latency & throughput increase is significant though the comparisons are against vLLM. It seems like TRT does batching a bit differently so unsure if this can equally apply here.

[ncomly-nvidia]

Hey @0xymoro Thanks for sharing!!

Yes, splitfuse is an impressive advancement from Deepspeed, one we were also working as well! Our implementation will be a little different due to difference in batching strategies, but the idea of chunking the prefill is the same! Likely will land in the next few releases.

[ncomly-nvidia]

Hey @0xymoro, Chunked attention is now part of v0.7.1! We're still working on an example so I'll leave this open until that's done!

Edit: The kernels were added in v0.7.1, the full feature will be in v0.8!

[Shixiaowei02]

Hi @0xymoro ! Chunked context will be part of TensorRT-LLM v0.8. Thank you for your support!

[littletomatodonkey]

Hi @0xymoro ! Chunked context will be part of TensorRT-LLM v0.8. Thank you for your support!

Hi, @Shixiaowei02 Thanks for your great job! How can i use Chunked context in TensorRT-LLM? Is there any docs?