[Performance] Decode Offload improves the long texts performance 100% through dynamic block offload. (#17216)

Co-authored-by: zhangheng <hzh0425@apache.org>
2026-03-08 17:16:53 +08:00
parent 97a2a9be0f
commit 230fb55899
8 changed files with 290 additions and 33 deletions
@@ -276,6 +276,7 @@ class Envs:

    # Hi-Cache
    SGLANG_HICACHE_HF3FS_CONFIG_PATH = EnvStr(None)
+    SGLANG_HICACHE_DECODE_OFFLOAD_STRIDE = EnvInt(None)
    SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR = EnvStr(None)
    SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR = EnvStr(None)
    # Max fraction of cache (by token count) that can be pinned; 0 = disable pinning.