Towards Out-of-core Neural Networks on Microcontrollers

To run neural networks (NNs) on microcontroller units (MCUs), memory size is the major constraint. While algorithm-level techniques exist to reduce NN memory footprints, the resultant losses in NN accuracy and generality disqualify MCUs for many important use cases. To address the constraint, we investigate out-of-core execution of NNs on MCUs: dynamically swapping NN data tiles between an MCU's small SRAM and its large, low-cost external flash. Accordingly, we present a scheduler design that automatically schedules compute tasks and swapping IO tasks in order to minimize the IO overhead in swapping. Out-of-core NNs on MCUs raise multiple concerns: execution slowdown, storage wear out, energy consumption, and data security. Our empirical study shows that none of these concerns is a showstopper; the key benefit - MCUs being able to run large NNs with full accuracy/generality - trumps the overheads. Our findings suggest that MCUs can play a much greater role in edge intelligence.