Informer: an efficient transformer architecture using convolutional layers

Abstract

The use of Transformer based architectures has been extended in recent years, reaching the level of State of the Art (SOTA) in numerous tasks in the field of Natural Language Processing (NLP). However, despite the advantages of this architecture, it has some negative factors, such as the high number of parameters it uses. That is why the use of this type of architecture can become expensive for research teams with limited resources. New variants have emerged with the purpose of improving the efficiency of the Transformer architecture, addressing different aspects of it. In this paper we will focus on the development of a new architecture that seeks to reduce the memory consumption of the Transformer, meanwhile is able to achieve a SOTA result in two different datasets [14] for the Neural Machine Translation (NMT) task.