跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2609.39131 · v3

Characterizing High Bandwidth Flash for LLM Serving

首次提交 2026-09-30 15:02
版本更新 2026-11-19 18:42
雷达观测 2026-11-20 19:57
arXiv ↗PDF ↗

摘要

Large language model (LLM) serving requires substantial memory to store model weights and KV caches. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it important to retain KV cache for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for LLM serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 21.1-86.9% relative to HBM-only systems. Modeled energy savings reach 62.4%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 0.99 to 13.63 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling in serving LLM with HBF, positioning HBF as an actively utilized lower-level memory tier rather than passive storage.

作者备注:7 pages, 2 figures, 7 tables. v3: updated results and revised text. Accepted to the RAISE-Arch workshop at MICRO 2026

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v3,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v3 · 2026-11-19 18:42 · 最新观测版本

Characterizing High Bandwidth Flash for LLM Serving

Large language model (LLM) serving requires substantial memory to store model weights and KV caches. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it important to retain KV cache for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for LLM serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 21.1-86.9% relative to HBM-only systems. Modeled energy savings reach 62.4%, with benefits depending on the workload and weight placement. Buffered cache-aware scheduling extends estimated HBF write lifetime from 0.99 to 13.63 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling in serving LLM with HBF, positioning HBF as an actively utilized lower-level memory tier rather than passive storage.

查看此版本解读 →查看此版本原文 ↗