Please use this identifier to cite or link to this item: http://hdl.handle.net/10553/106914
Title: Informer, an information organization transformer architecture
Authors: Estupiñán Ojeda, Cristian David 
Guerra Artal, Cayetano 
Hernández Tejera, Francisco Mario 
UNESCO Clasification: 120304 Inteligencia artificial
Keywords: Convolution
Deep learning
Informer
Linear transformer
Neural machine translation, et al
Issue Date: 2021
Publisher: SciTePress Digital Library
Journal: ICAART (Setúbal) 
Conference: 13th International Conference on Agents and Artificial Intelligence (ICAART 2021) 
Abstract: The use of architectures based on transformers presents a state of the art revolution in natural language processing (NLP). The employment of these architectures with high computational costs has increased in the last few months, despite the existing use of parallelization techniques. This is due to the high performance that is obtained by increasing the size of the learnable parameters for these kinds of architectures, while maintaining the models' predictability. This relates to the fact that it is difficult to do research with limited computational resources. A restrictive element is the memory usage, which seriously affects the replication of experiments. We are presenting a new architecture called Informer, which seeks to exploit the concept of information organization. For the sake of evaluation, we use a neural machine translation (NMT) dataset, the English-Vietnamese IWSLT15 dataset (Luong and Manning, 2015). In this paper, we also compare this proposal with architectures that reduce the computational cost to O(n · r), such as Linformer (Wang et al., 2020). In addition, we have managed to improve the SOTA of the BLEU score from 33.27 to 35.11.
URI: http://hdl.handle.net/10553/106914
ISBN: 978-989-758-484-8
ISSN: 2184-433X
DOI: 10.5220/0010372703810389
Source: ICAART 2021 - Proceedings of the 13th International Conference on Agents and Artificial Intelligence [ISSN 2184-433X] ,v. 2, p. 381-389, (Enero 2021)
Appears in Collections:Actas de congresos
Show full item record

Google ScholarTM

Check

Altmetric


Share



Export metadata



Items in accedaCRIS are protected by copyright, with all rights reserved, unless otherwise indicated.