Efficient Neural Architecture Search for Long Short-Term Memory Networks

Automated machine learning (AutoML) is a technique which helps to determine the optimal or near-optimal model for a specific dataset and has been a focused research area during the last years. The automation of model design opens doors for non-machine learning experts to utilize machine learning models in several scenarios, which is both appealing for a wide range of researchers and for cloud services as well. Neural Architecture Search is a subfield of AutoML where the optimal artificial neural network model's architecture is generally searched with adaptive algorithms. This paper proposes a method to apply Efficient Neural Architecture Search (ENAS) to LSTM-like recurrent architecture, which uses a gating mechanism an inner memory. Using this method, the paper investigates if the handcrafted Long Short-Term Memory (LSTM) cell is an optimal or near-optimal solution of sequence modelling for a given dataset, or other, automatically defined recurrent structures outperform. The performance of vanilla LSTM, and advanced recurrent architectures designed by random search, and reinforcement learning-based ENAS are examined and compared. The proposed methods are evaluated in a text generation task on the Penn TreeBank dataset.