When Is High-Bandwidth Flash Cost-Effective for LLM Inference?
Zhiyu An, Yiping Li, Wan Du
原文摘要
Can high-bandwidth flash (HBF) make LLM inference cheaper than adding GPUs? By storing weights closer to the GPU than SSD, HBF may reduce the number of GPUs needed for model capacity. However, this integration is only justified if it meets service targets and its cost is recovered. We simulate this integration using public component data across nine dense and mixture-of-experts (MoE) models, five workloads, two GPU platforms, and 1--100 GPUs. At fixed demand, HBF reduces the minimum GPU count to meet a service target in 98 of 468 comparisons; majority of them are paused and short sessions, 63 remain cost-effective at half a GPU price per HBF stack. For single short-chat users, HBF enables large MoEs, e.g. DeepSeek V3 and Kimi K2, to meet a relaxed service target on one RTX~4090 GPU, where conventional serving fails. At fixed GPU count, HBF improves throughput in 48\% of matched comparisons below 16 GPUs and 13\% at larger counts. Even at 100 RTX~4090-scale GPUs, cascaded HBF raises offline throughput by 16.4--26.5$\times$ for three large MoEs. We found savings and throughput gains arise in different scenarios.
cs.AR