Bidirectional Multi-Stack RNNs with Attention for Machine Translation

This project developed a novel attention algorithm for multi-stack bidirectional encoder-decoder RNN (including GRU and LSTM) sequence-to-sequence models, particularly for language translation tasks. The attention mechanism utilizes matrix rearranging and multiplication to compute the significance of the vectors in the encoder output to the vectors in the current decoder hidden states when predicting each word. Our approach achieved 98% of the performance of fine-tuned pretrained T5-small, with 30% to 50% fewer parameters depending on vocabulary size, making our model an ideal choice in cases of single-processor training, low processor resource, limited memory, small dataset, or tasks not supported by pretrained transformers.