跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.10425 · v1

Governed Self-Improvement for a Local-First Security Analyzer: Baseline Measurements and a Preregistered Evaluation Protocol for Tantular Cyber

首次提交 2026-11-07 11:06
版本更新 2026-11-07 11:06
雷达观测 2026-11-10 10:11
arXiv ↗PDF ↗

摘要

Local-first security tools let organizations analyze code without sending it to third-party services, but the small language models they can run locally are weaker than hosted ones and are hard to improve without breaching the privacy guarantee that motivates them. We present Tantular Cyber, an open-source, read-only repository security analyzer that keeps deterministic scanner findings authoritative and confines a locally served language model to advisory explanations in Bahasa Indonesia. We report its measured v0.3.0 baseline on a synthetic, ground-truth benchmark: combining Semgrep with an interprocedural call-graph scanner yields precision 0.843 and recall 0.915 (Semgrep alone: 0.818 and 0.766). The advisory layer is the weak point: 57% of explanations met an evidence-grounding check, against 95% for a different 8B model under identical prompts. A forensic re-scoring of retained outputs shows that every grounding failure cites a code identifier absent from the evidence shown, and that a prompt change which appeared to raise grounding from 0.56 to 0.72 did so partly by citing nothing. We document two evaluation-validity hazards we encountered: silent model-file migration during a run and non-reproducible published figures. Building on these lessons and on constraint-aware evaluation from security operations research, we propose a governed self-improvement loop (code-free feedback, curated data, a cheapest-first intervention ladder from prompts to LoRA and weight-space edits, hard gates, and human release approval) and preregister the protocol by which its first cycle will be judged. We report no results for the loop itself.

作者备注:7 pages, 2 figures, 5 tables. Code and benchmark: https://github.com/ghifiardi/tantular-cyber

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-07 11:06 · 最新观测版本

Governed Self-Improvement for a Local-First Security Analyzer: Baseline Measurements and a Preregistered Evaluation Protocol for Tantular Cyber

Local-first security tools let organizations analyze code without sending it to third-party services, but the small language models they can run locally are weaker than hosted ones and are hard to improve without breaching the privacy guarantee that motivates them. We present Tantular Cyber, an open-source, read-only repository security analyzer that keeps deterministic scanner findings authoritative and confines a locally served language model to advisory explanations in Bahasa Indonesia. We report its measured v0.3.0 baseline on a synthetic, ground-truth benchmark: combining Semgrep with an interprocedural call-graph scanner yields precision 0.843 and recall 0.915 (Semgrep alone: 0.818 and 0.766). The advisory layer is the weak point: 57% of explanations met an evidence-grounding check, against 95% for a different 8B model under identical prompts. A forensic re-scoring of retained outputs shows that every grounding failure cites a code identifier absent from the evidence shown, and that a prompt change which appeared to raise grounding from 0.56 to 0.72 did so partly by citing nothing. We document two evaluation-validity hazards we encountered: silent model-file migration during a run and non-reproducible published figures. Building on these lessons and on constraint-aware evaluation from security operations research, we propose a governed self-improvement loop (code-free feedback, curated data, a cheapest-first intervention ladder from prompts to LoRA and weight-space edits, hard gates, and human release approval) and preregister the protocol by which its first cycle will be judged. We report no results for the loop itself.

查看此版本解读 →查看此版本原文 ↗