Optimized build 1x RTX Pro 6000: 177 to up to 340 tok/s, concurrency=4 @ 800 tok/s aggregate, 939K KV cache + RAM offloading
I have build a series of patches to improve serving this checkpoint on RTX Pro 6000 and ti should also work on DGX Spark.
https://github.com/mratsim/sglang-qwen38fn-sm120-turbo
Highlights:
- 939456 at 4 concurrency (can go over 1M if we reduce it as gated delta nets cache eats into KV cache)
- 856256 at 8 concurrency
- ~12500 tok/s prefill
- Up to 355 token/s decode on predictable tokens like code or compaction.
- 200 tok/s per stream for 4x concurrent requests
It was super fast in the benchmarks for me. I got about 10% less than you on average with a workstation edition version. As soon as I through the model live in hermes this was the first reply:
The user switched the!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! │
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I really with this thing would work as it is much faster than what I have been using.
I have not seen the looping above since the very first query. Has been solid since then.
Wow, looks promising. I will try this later on my DGX Spark and report back on how it goes.
I love it, works great!
Btw, my agent made a patch in my fork of your repo https://github.com/shussekaido/sglang-qwen38fn-sm120-turbo/pull/1