Modularity as a means for complexity management in neural networks learning

Abstract

Training a Neural Network (NN) with lots of parameters orintricate architectures creates undesired phenomena that com-plicate the optimization process. To address this issue we pro-pose a first modular approach to NN design, wherein the NNis decomposed into a control module and several functionalmodules, implementing primitive operations. We illustratethe modular concept by comparing performances between amonolithic and a modular NN on a list sorting problem andshow the benefits in terms of training speed, training stabil-ity and maintainability. We also discuss some questions thatarise in modular NNs.