Fine-tuning with local learning rules helps to compress and accelerate spiking neural networks without accuracy loss

Spiking neural networks (SNNs) are believed to be highly energy- and computationally efficient machine learning algorithms, especially when implemented on neuromorphic hardware. Some recent studies have revealed that lateral (intralayer) inhibitory connectivity is necessary for effective and stable learning of SNNs. However, for large-scale SNNs, lateral inhibitory connections require an additional large amount of calculations. This negatively affects both the SNN inference time and the size of required computing resources. In this study, we propose a fine-tuning procedure using original local learning rules, called FEELING, to be applied to the weights of interneuron sublayer, which is introduced for organizing more efficient competition between excitatory neurons. At the same time, the initialization of interneuron weight values is implemented by singular value decomposition of intralayer inhibitory weight matrix characterizing the original SNN architecture before optimization. The proposed procedure allows to compress and accelerate large-scale SNNs with lateral inhibition in the layers, alongside with maintenance of their recognition accuracy, as it is shown on MNIST dataset. We demonstrate that this new optimization technique is superior to simple pruning of inhibitory connections, even also followed by fine-tuning. Moreover, this method of fine-tuned decomposition suggests the association of excitatory and inhibitory functions to two different sublayers of neurons, as it is naturally observed in a biological neural system. We hope that findings of this study not only reveal some new aspects of the effective computation and bio-plausible architecture for SNNs but also assume a hypothetic reason for the evolutionary preference of inhibitory neurons over inhibitory connections.