← 研踪 PaperTrailarXiv:2611.02401 · v1
Accelerating LLM Decoding with In-DRAM Parallelism
首次提交 2026-11-02 10:22
版本更新 2026-11-02 10:22
雷达观测 2026-11-04 23:42
摘要
Autoregressive decoding of large language models (LLMs) on edge devices is dominated by memory-bound general matrix--vector multiplication (GeMV) operations. Processing-using-DRAM (PUD) is a promising computing paradigm that can alleviate this bottleneck by reducing processor--memory data transfers and enabling massively parallel computation within DRAM (e.g., across 65,536 columns). However, weight matrices in individual LLM layers (e.g., a QKV matrix in Llama3 8B with 6,144 rows) are typically too small to fully utilize parallelism in PUD. To address this challenge, we propose an algorithm--system co-design for PUD-based LLM inference. At the algorithm level, we enable a subsequent projection to be packed with the current projection by reformulating it as a computation on the current residual state plus a correction for the residual update. Layer redundancy allows this correction to be approximated using fewer bit-serial steps. At the system level, we tailor data mapping and execution to improve utilization of either columns within DRAM subarrays (SIMD) or independently controlled banks (MIMD). Overall, we achieves up to $1.41\times$ end-to-end decoding speedup over the PUD baseline, corresponding to a $2.45\times$ speedup over processor-based execution.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-02 10:22 · 最新观测版本
Accelerating LLM Decoding with In-DRAM Parallelism
Autoregressive decoding of large language models (LLMs) on edge devices is dominated by memory-bound general matrix--vector multiplication (GeMV) operations. Processing-using-DRAM (PUD) is a promising computing paradigm that can alleviate this bottleneck by reducing processor--memory data transfers and enabling massively parallel computation within DRAM (e.g., across 65,536 columns). However, weight matrices in individual LLM layers (e.g., a QKV matrix in Llama3 8B with 6,144 rows) are typically too small to fully utilize parallelism in PUD. To address this challenge, we propose an algorithm--system co-design for PUD-based LLM inference. At the algorithm level, we enable a subsequent projection to be packed with the current projection by reformulating it as a computation on the current residual state plus a correction for the residual update. Layer redundancy allows this correction to be approximated using fewer bit-serial steps. At the system level, we tailor data mapping and execution to improve utilization of either columns within DRAM subarrays (SIMD) or independently controlled banks (MIMD). Overall, we achieves up to $1.41\times$ end-to-end decoding speedup over the PUD baseline, corresponding to a $2.45\times$ speedup over processor-based execution.
查看此版本解读 →查看此版本原文 ↗