Replacing Attention with Cellular-Recurrent Architectures

TinyCeNN-LM addresses the high computational cost of standard Transformer attention mechanisms by proposing a conversion framework that maps pretrained attention weights into Cellular Neural Network (CeNN)-inspired layers. Unlike traditional attention, which scales quadratically with sequence length, these cellular-recurrent layers operate with linear complexity. The core innovation lies in the structural mapping of attention heads into local, recurrent interactions that mimic the spatial-temporal dynamics of CeNNs, allowing the model to process long sequences more efficiently without the memory bottlenecks inherent in standard self-attention.

The Quality-Gated Conversion Process

To ensure that the model retains its original capabilities after the architectural shift, the authors implement a "quality-gated" conversion strategy. This process involves a selective distillation phase where the recurrent layers are trained to approximate the output distribution of the original attention heads. The "gate" acts as a performance threshold: only those attention heads that can be mapped to the cellular-recurrent structure with minimal loss in perplexity or task accuracy are converted. This hybrid approach allows the model to retain standard attention in layers where complex, long-range dependencies are critical, while offloading simpler, repetitive patterns to the more efficient cellular-recurrent layers. This selective replacement minimizes the performance degradation typically associated with aggressive model pruning or architectural distillation.