arXiv:2606.07665 · v3

AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference

首次提交 2026-06-04 11:49

版本更新 2026-11-08 14:41

雷达观测 2026-11-10 12:16

摘要

Transformer inference increasingly relies on specialized compiler and runtime support, while recent LLMs can generate nontrivial CUDA kernels. However, unconstrained generation guarantees neither correctness nor performance. We present \textsc{AgentCompile}, an LLM-guided CUDA inference compiler that combines two complementary uses of LLMs. First, the LLM provides advisory metadata for compiler-derived region summaries and bounded candidate spaces. The compiler then instantiates template-based CUDA candidates, validates correctness, selects implementations by measured latency, and falls back when specialization is unsupported or unprofitable. Second, under compiler-defined contracts, the LLM directly generates five classes of decode-critical kernels to accelerate inference, prompted by distilled optimization principles. \textsc{AgentCompile} integrates these kernels into a serving runtime with paged KV cache, continuous batching, preemption, chunked prefill, and bucketed full-step CUDA Graph replay. Across six evaluated model families, \textsc{AgentCompile} achieves speedups of \textbf{2.23--6.98$\times$} over PyTorch eager for single-request generation, and \textbf{1.04--1.16$\times$} over vLLM for both single-request generation and multi-request serving. Our code is publicly available at https://github.com/veneno1213822/AgentCompile.

作者备注：12 pages, 4 figures