← 研踪 PaperTrailarXiv:2601.03708 · v3
MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
首次提交 2026-01-07 16:46
版本更新 2026-11-15 15:11
雷达观测 2026-11-17 19:15
摘要
Large language models (LLMs) have achieved strong performance on code completion in general-purpose programming languages, but existing repository-level benchmarks focus almost exclusively on software code and largely overlook hardware description languages (HDLs). We present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark for multilingual hardware code completion at the repository level with functional evaluation. MHRC-Bench covers three major hardware design coding styles and annotates each completion target with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree (CST) analysis. Post-training billion-scale code LLMs on MHRC-Bench-Train enables smaller open-weight models to outperform much larger general-purpose models under matched context windows. Beyond aggregate accuracy, our analysis surfaces four findings that distinguish hardware from software code completion: \emph{(i)}~performance varies sharply across HDLs, with high-level synthesis (HLS) easiest and VHDL hardest in most settings across functional pass rate, exact match (EM), and edit similarity (ES); \emph{(ii)}~hardware accuracy is non-monotonic in code-structure depth, in contrast with the monotonic depth--accuracy curve known for software; \emph{(iii)}~accuracy plateaus once the input context exceeds 2{,}048 tokens, while software completion continues to benefit from longer context; and \emph{(iv)}~certain repository-level retrievers tuned on software underperform a no-retrieval baseline on HDL repositories because their structural and dependency assumptions are violated by hardware code.
作者备注:Accepted at NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-15 15:11 · 最新观测版本
MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
Large language models (LLMs) have achieved strong performance on code completion in general-purpose programming languages, but existing repository-level benchmarks focus almost exclusively on software code and largely overlook hardware description languages (HDLs). We present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark for multilingual hardware code completion at the repository level with functional evaluation. MHRC-Bench covers three major hardware design coding styles and annotates each completion target with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree (CST) analysis. Post-training billion-scale code LLMs on MHRC-Bench-Train enables smaller open-weight models to outperform much larger general-purpose models under matched context windows. Beyond aggregate accuracy, our analysis surfaces four findings that distinguish hardware from software code completion: \emph{(i)}~performance varies sharply across HDLs, with high-level synthesis (HLS) easiest and VHDL hardest in most settings across functional pass rate, exact match (EM), and edit similarity (ES); \emph{(ii)}~hardware accuracy is non-monotonic in code-structure depth, in contrast with the monotonic depth--accuracy curve known for software; \emph{(iii)}~accuracy plateaus once the input context exceeds 2{,}048 tokens, while software completion continues to benefit from longer context; and \emph{(iv)}~certain repository-level retrievers tuned on software underperform a no-retrieval baseline on HDL repositories because their structural and dependency assumptions are violated by hardware code.
查看此版本解读 →查看此版本原文 ↗