Mutual Information-Based Neural Network Distillation for Improving Photonic Neural Network Training

Photonic neural networks are among the most promising recently proposed neuromorphic solutions for providing fast and energy efficient Deep Learning (DL) implementations. However, photonic networks need to be first appropriately trained to support the underlying hardware implementation (e.g., activation functions), since it is not possible to directly deploy an already trained DL model into a neuromorphic platform that uses different transfer functions compared to regular networks. The main hypothesis examined in this paper is whether we can employ a neural network distillation approach to transfer the knowledge from a regular DL model into a photonic neural network reducing in this way the accuracy gap between regular and photonic models. Furthermore, we propose a novel fully differentiable formulation of mutual information that can be used for efficiently transferring the knowledge between layers of a regular neural networks and photonic neural networks. In this way, we further increase the effectiveness of distillation by also taking into account the behavior of intermediate layers of DL models. Experiments conducted on two image datasets demonstrate the effectiveness of the proposed approach, further closing the accuracy gap between photonic and regular DL models.