TIDE: Asymmetric Neural Circuits for Stabilized Temporal Inhibitory-Excitatory Dynamics
Alexander Kyuroson, Denis Kleyko, Marcus Liwicki
原文摘要
Recent Continuous Thought Machine (CTM) architecture decouples internal computation from external inputs via neural dynamics, but relies on unconstrained multi-layer perceptrons for its recurrent computations. We propose to model neural dynamics using asymmetric Excitatory-Inhibitory (E-I) networks, which can be stabilized via principles from network theory, while obeying Dale's principle and can be expressed as energy-based systems optimized through a game-theoretic loss. Building on this perspective, we introduce Temporal Inhibitory-Excitatory Dynamic Engine (TIDE), a neuro-inspired architecture that computes internal representations through neural dynamics stabilized by incorporating the Wilson-Cowan dynamics and lateral inhibition. TIDE balances biological realism by, for instance, using Hierarchical Receptive Fields and enforcing Dale's principle to ensure a realistic $80:20$ E-I balance ratio with an end-to-end trainable architecture. The aim of this paper is to introduce a new architecture that brings neuro-inspired learning and architectural principles to the forefront. We present proofs of convergence, stability, and complexity bounds, along with empirical ablation studies. Overall, TIDE achieves higher $\texttt{top-1}$ accuracy than CTM on the majority of the investigated datasets while requiring a smaller training budget. Furthermore, across varying training budgets, TIDE improves $\texttt{top-1}$ accuracy on ImageNet-C by an average of $+1.65$% to $+6.42$% under different perturbation severity levels.
cs.LG