arXiv:2606.16897 · v2

Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures

首次提交 2026-06-16 00:07

版本更新 2026-11-09 15:44

雷达观测 2026-11-10 13:18

摘要

Do language models encode the same prompted concepts in structurally compatible ways, beyond generic similarity induced by shared questions? We study contrastive-difference centered kernel alignment ($\mathrm{CKA}_Δ$), which compares per-question activation differences between opposing concept instructions. A matched same-concept versus cross-concept comparison separates the question of concept-conditioned structure from the magnitude of representational similarity. Across four checkpoints and eight personality-related traits, a fixed-final-layer analysis gives mean same- and cross-trait $\mathrm{CKA}_Δ$ of 0.7259 and 0.6828. Trait-label randomization detects correspondence conditional on this model and prompt set ($p=1/10001$, one-sided), but a model-and-trait-weighted 95% interval for the difference is $[-0.0330,0.1017]$, limiting broader generalization. Strict question-group holdouts retain high same-trait linear recoverability. More stringently, maps and preprocessing fitted on seven traits transfer a source polarity classifier for the eighth trait to handwritten test questions: mean affine and orthogonal accuracies are 88.10% and 88.57%, versus 47.70% for shuffled pairing. Performance varies substantially across traits and model directions. Synthetic controls show why moderate CKA and high affine accuracy alone do not establish functional equivalence. The results support a scoped diagnostic of concept-specific structure and held-out linear recoverability; they do not establish universal invariance or a robust advantage over raw CKA across architectures.

作者备注：Accepted at ACML 2026 (Conference Track). 13-page main paper plus 18-page supplementary material. Revised theoretical statements and statistical analysis; added grouped holdouts and concept-held-out transfer evaluation; narrowed claims