无知几何:通过激活碎片化检测大语言模型中被抑制的知识
The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation
Adar Peleg, Dvir Alsheich, Tomer Ashuach, Avi Mendelson
AI 速读 基于摘要
该论文提出用概念查询与匿名化改写之间隐藏状态差异(anon-vectors)的几何一致性,来区分模型真正掌握与被抑制的知识。
cs.AI
查看原文摘要
Evaluating whether a large language model (LLM) genuinely knows a concept remains a fundamental problem, since behavioral metrics can be fooled by suppression: a model that refuses to answer may still harbor recoverable knowledge. The framework measures the geometric coherence of \textit{anon-vectors}, the differences in hidden states between a concept-specific query and its anonymized paraphrase, and shows that genuine knowledge forms a tight, low-entropy cluster while ignorance forces geometric fragmentation. Across ten instruction-tuned LLMs from three model families, all fragmentation metrics reliably separate known from unknown concepts, and post-unlearning, fragmentation geometry correlates with adversarial recovery across four attack types and five unlearning algorithms, though the direction of that relationship is shown to depend on attack methodology. Furthermore, a layer-resolved analysis reveals that different adversarial mechanisms exploit this geometry at varying depths, with high attack-specific correlations that are invisible to behavioral evaluation. Together, these results constitute a single-phase, attack-free proxy for unlearning robustness that captures aspects of parametric knowledge inaccessible to output-level metrics. A reference implementation is available at https://github.com/AdarPeleg/geometry-of-ignorance