Hidden not Deleted: How Networks Suppress Entangled Features
Akash Samanta, Manish Pratap Singh, Debasis Chaudhuri
原文摘要
When a neural network stops reporting a feature, has it forgotten the feature or merely hidden it? Shared representations complicate the answer: removing one feature's direction can also damage another feature. We study this problem in a small autoencoder that compresses and reconstructs inputs, allowing individual feature contributions to be traced directly. With the representation fixed, restoring a single bias partially recovers target detection by exposing a signal hidden behind the output's activation threshold. Tests on target-absent inputs distinguish this recovery from interventions that simply increase output. The same bias intervention becomes less effective when the representation is allowed to change. We find that training only the other outputs can weaken the target's contribution, even without a loss on its own output. A conditional theoretical analysis explains how interference with retained predictions creates this pressure. Yet more flexible predictors still detect residual target information. These findings show how recovery depends on feature geometry and on which parameters are trained, and why a silent output is insufficient evidence of forgetting.
cs.LG · cs.AI