Estudio acerca del entrenamiento con alineamiento de representaciones multimodales en el espacio latente

Abstract

In recent years, there has been significant interest in integrating into deep learning models the ability to process and interpret multiple data modalities, such as text and images, jointly. To achieve this, many modern architectures project these data into a shared latent space, where representations of equivalent concepts—although expressed in different forms—are expected to be semantically aligned. This cross-modal alignment is fundamental for tasks such as cross-generation and for obtaining a more comprehensive modeling of the world around us. This work specifically addresses the problem of aligning latent representations between the text and image modalities, proposing an approach based on variational autoencoders trained jointly. To this end, a reformulation of sequence-to-sequence models is proposed, allowing their integration within a variational framework by incorporating latent variables capable of modeling the inherent uncertainty of natural language. In particular, this will be done for three recurrent architectures widely used in text modeling: LSTM, GRU, and xLSTM. These architectures will be integrated into a joint model with image autoencoders, sharing a common latent space where representations from both modalities converge. Inspired by the theoretical framework of the MVAE, a similar notation and formulation is adopted to ensure conceptual compatibility, although an alternative architecture better suited for sequential tasks is proposed. As part of the work, an extensible experimentation framework has been developed, designed to facilitate scientific collaboration and a more practical, formal, and reproducible approach. This framework allows for defining and executing experiments in a structured manner, accelerating the incorporation of new architectures and multimodal configurations. Experiments are conducted on the MNIST, FashionMNIST, and CelebA datasets. The source code of the experiments, as well as the framework, is available at: https://github.com/ricardocardn/Re-MVAE.