← 研踪 PaperTrailarXiv:2603.15457 · v2
Evasive Intelligence: Lessons from Malware Analysis for Evaluating AI Agents
首次提交 2026-03-16 23:55
版本更新 2026-11-03 05:53
雷达观测 2026-11-04 23:40
摘要
Tool-using artificial intelligence (AI) agents are increasingly adopted as systems that plan, observe their environment, and take actions over extended time periods. This shift challenges the current evaluation practice, in which models are tested in restricted, fully observable settings. We argue that such evaluations are vulnerable to a failure mode long known in computer security: a system that behaves benignly when it detects that it is being analyzed. Malware has exploited this for decades, by probing sandboxes, virtual machines, and instrumentation, before revealing its payload. The shared challenge is that the subject under observation can detect the conditions of its evaluation, so the setup used to observe behavior ends up shaping the behavior observed, yielding overly optimistic safety and robustness assessments. Recent AI-safety studies demonstrate pieces of this risk, though only preliminarily and under controlled conditions. Drawing on decades of malware-analysis experience, we outline concrete principles for evaluating AI agents as potentially adversarial: explicitly threat-modeling evaluator-discrimination; testing across harness variants and reporting the worst case; using shadow executions to provide a realistic runtime without irreversible side effects; treating environment probing as a risk signal; and performing unpredictable reassessments after deployment.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-03 05:53 · 最新观测版本
Evasive Intelligence: Lessons from Malware Analysis for Evaluating AI Agents
Tool-using artificial intelligence (AI) agents are increasingly adopted as systems that plan, observe their environment, and take actions over extended time periods. This shift challenges the current evaluation practice, in which models are tested in restricted, fully observable settings. We argue that such evaluations are vulnerable to a failure mode long known in computer security: a system that behaves benignly when it detects that it is being analyzed. Malware has exploited this for decades, by probing sandboxes, virtual machines, and instrumentation, before revealing its payload. The shared challenge is that the subject under observation can detect the conditions of its evaluation, so the setup used to observe behavior ends up shaping the behavior observed, yielding overly optimistic safety and robustness assessments. Recent AI-safety studies demonstrate pieces of this risk, though only preliminarily and under controlled conditions. Drawing on decades of malware-analysis experience, we outline concrete principles for evaluating AI agents as potentially adversarial: explicitly threat-modeling evaluator-discrimination; testing across harness variants and reporting the worst case; using shadow executions to provide a realistic runtime without irreversible side effects; treating environment probing as a risk signal; and performing unpredictable reassessments after deployment.
查看此版本解读 →查看此版本原文 ↗