← 研踪 PaperTrailarXiv:2607.04031 · v2
TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout
首次提交 2026-07-05 05:19
版本更新 2026-11-04 13:44
雷达观测 2026-11-05 11:10
摘要
Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade LLM inference performance by up to an order of magnitude. The root cause is read amplification during tiled matrix multiplication, in which each memory request fetches far more data than the compute tile needs. Such waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, causing each request to spill well beyond tile boundaries. To eliminate read amplification in LGMS, we propose to use tile-major layout. Rather than storing data as a one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels can adopt tile-major by only changing the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for tile-major support in TMA-based kernels without code changes. We evaluate TileLens on a cycle-level GPU simulator using Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with our adaptive degree prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5 us NAND read latency, reducing the geomean slowdown from 1.62-6.69x with conventional layouts to within 10% of an HBM-only baseline.
作者备注：Accepted to HPCA 2027. Jae Hyung Ju and Euijun Chung contributed equally
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-04 13:44 · 最新观测版本
TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout
Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade LLM inference performance by up to an order of magnitude. The root cause is read amplification during tiled matrix multiplication, in which each memory request fetches far more data than the compute tile needs. Such waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, causing each request to spill well beyond tile boundaries. To eliminate read amplification in LGMS, we propose to use tile-major layout. Rather than storing data as a one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels can adopt tile-major by only changing the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for tile-major support in TMA-based kernels without code changes. We evaluate TileLens on a cycle-level GPU simulator using Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with our adaptive degree prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5 us NAND read latency, reducing the geomean slowdown from 1.62-6.69x with conventional layouts to within 10% of an HBM-only baseline.
查看此版本解读 →查看此版本原文 ↗