Mitigating Memorization Where It Happens
Xuanqi Zhang, Haoyang Shang, Xiaoxiao Li
Large language models (LLMs) can memorize and reproduce training sequences verbatim, raising privacy and copyright concerns. Existing mitigation methods apply interventions uniformly, often degrading predictions at the many positions unrelated to verbatim recall. We study a token-level log-likelihood ratio between a target model and a domain-competent reference with lower exposure to the evaluated sequences. Exposure-controlled experiments support its use for localizing positions associated with sequence-level verbatim recall. Across evaluated settings, a minority of positions carries a disproportionate share of the positive signal, motivating selective activation correction. From a first-order surrogate of the memorization-reduction objective, we derive a probe-steer framework, which decomposes intervention into a probe that flags high-signal activations and a steer that applies a correction only when the probe exceeds a threshold. We propose Gated Subspace Steering (GSS) as its practical instantiation: under a background normalization constraint, the probe-steer pairs that are optimal for this surrogate follow in closed form from the leading singular vectors of a whitened gradient-weighted activation matrix. Across four benchmarks, GSS achieves memorization suppression comparable to strong baselines, with favorable utility trade-offs in several settings, while requiring 10-100$\times$ less compute than optimization-based alternatives. Code is available at https://github.com/ubc-tea/Mitigating-Memorization-Where-It-Happens.
尚无 AI 解读，展示原文摘要