Design and independent training of composable and reusable neural modules

Abstract

Monolithic neural networks and end-to-end training have become the dominating trend in the field of deep learning, but the steady increase in complexity and training costs has raised concerns about the effectiveness and efficiency of this approach. We propose modular training as an alternative strategy for building modular neural networks by composing neural modules that can be trained independently and then kept for future use. We analyse the requirements and challenges regarding modularity and compositionality and, with that information in hand, we provide a detailed design and implementation guideline. We show experimental results of applying this modular approach to a Visual Question Answering (VQA) task parting from a previously published modular network and we evaluate its impact on the final performance, with respect to a baseline trained end-to-end. We also perform compositionality tests on CLEVR.