← 研踪 PaperTrailarXiv:2611.05013 · v1
One Strategy Does Not Fit All: Rethinking Data Labeling for Just-in-Time Vulnerability Prediction
首次提交 2026-11-03 22:57
版本更新 2026-11-03 22:57
雷达观测 2026-11-04 23:39
摘要
Just-In-Time Vulnerability Prediction (JIT-VP) identifies security risks in code changes at commit time, enabling early feedback before vulnerabilities enter software systems. However, its practical reliability is challenged by the difficulty of constructing accurate labels. Multiple strategies with different trade-offs have been proposed to identify vulnerability-fixing commits (VFCs) and vulnerability-introducing commits (VICs) for dataset construction, leading to substantially different datasets. This makes it difficult to separate model performance from dataset construction. Moreover, real-world quality assurance requires not only classification but also prioritizing high-risk commits and localizing vulnerable causes. This paper presents an extensive empirical study of how labeling strategies affect these downstream tasks. We constructed 28 datasets using four VFC and four VIC labeling strategies across OpenSSL and Linux kernel, then evaluated six representative JIT-VP models for classification, prioritization, and localization. Our results show that labeling effectiveness is strongly task-dependent. Manual VFC labeling combined with vulnerability-specific VIC labeling provides the most reliable performance for classification and prioritization, significantly outperforming 11 of 15 alternatives. Meanwhile, precision-first VFC labeling combined with LLM-assisted VIC labeling performs best for localization, significantly outperforming 7 of 15 alternatives. These findings demonstrate that no single labeling strategy is universally optimal and highlight the importance of task-oriented dataset construction.
作者备注：29 pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-03 22:57 · 最新观测版本
One Strategy Does Not Fit All: Rethinking Data Labeling for Just-in-Time Vulnerability Prediction
Just-In-Time Vulnerability Prediction (JIT-VP) identifies security risks in code changes at commit time, enabling early feedback before vulnerabilities enter software systems. However, its practical reliability is challenged by the difficulty of constructing accurate labels. Multiple strategies with different trade-offs have been proposed to identify vulnerability-fixing commits (VFCs) and vulnerability-introducing commits (VICs) for dataset construction, leading to substantially different datasets. This makes it difficult to separate model performance from dataset construction. Moreover, real-world quality assurance requires not only classification but also prioritizing high-risk commits and localizing vulnerable causes. This paper presents an extensive empirical study of how labeling strategies affect these downstream tasks. We constructed 28 datasets using four VFC and four VIC labeling strategies across OpenSSL and Linux kernel, then evaluated six representative JIT-VP models for classification, prioritization, and localization. Our results show that labeling effectiveness is strongly task-dependent. Manual VFC labeling combined with vulnerability-specific VIC labeling provides the most reliable performance for classification and prioritization, significantly outperforming 11 of 15 alternatives. Meanwhile, precision-first VFC labeling combined with LLM-assisted VIC labeling performs best for localization, significantly outperforming 7 of 15 alternatives. These findings demonstrate that no single labeling strategy is universally optimal and highlight the importance of task-oriented dataset construction.
查看此版本解读 →查看此版本原文 ↗