arXiv:2606.20668 · v2
BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
摘要
LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails against repurposed generalist LLMs. We present BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems, Operational), an independent operational benchmark that evaluates 33 systems from 17 developers: specialized guardrails (e.g., Llama Guard 4, ShieldGemma-2B, Lakera Guard) and frontier generalists repurposed as supervisors (e.g., GPT-5.4, Claude Sonnet 4.6, Grok 4.1 Fast), jointly on detection rate, false-positive rate, latency, and cost. We cover input/output moderation across 11 harm categories, jailbreak detection across 13 attack techniques, and prompt-injection detection, using in-house datasets built from handcrafted prompts, curated external samples, and quality-controlled synthetic generation, with synthetic moderation samples paraphrased to suppress generator fingerprints. Content-moderation runs are repeated five times and all rates reported with 95% intervals. The Pareto frontier reveals use-case-dependent tradeoffs. On content moderation, specialized supervisors are operationally ahead: top systems match the best generalist LLMs on detection (93-96%, no significant difference) at low false-positive rates while running ~5x faster; cost depends more on serving setup than on class. On jailbreak detection, generalist LLMs achieve higher median detection on three of four datasets and lower false-positive rates, but at higher latency and, for closed frontier models, 10-100x higher cost. On prompt injection, the two best dedicated detectors lead. We release the framework, an interactive leaderboard, and four datasets as a vendor-neutral basis for selecting safeguards under deployment constraints.
作者备注：Accepted at NeurIPS 2026 (Track on Evaluations and Datasets). Previously presented at the ICML 2026 Workshop on Trustworthy AI for Good (AI4GOOD). 60 pages, 6 figures; 10-page main text plus appendices