The Growing Wiki¶
Fig. 1 Illustration of neuron addition in a growing network.¶
Growing Neural Networks start small and grow over the course of training, by e.g. adding neurons or layers. By only growing as large as necessary, and no larger, such methods offer the potential of improved efficiency and remove the need to select architectures by hand.
The Growing Wiki is primarily maintained by the Inria Tau team, but outside contributors are always welcome on GitHub. We also manage the Gromo PyTorch package for growing neural networks.
Introduction to growing NNs
Start here for an introduction and the mathematical background behind growing methods.
Growing design questions
The problem of growth can be divided into a number of key questions.
Applications
A number of concrete applications of Growing Neural Networks.
Taxonomy of growing algorithms¶
Growing algorithms can be classified across multiple axes: the type of growth, whether we aim to preserve the function output during growth or improve it, and also the objective that they aim to optimise during growth. We provide a non-exhaustive classification of growth methods.
Growing width
Expanding existing layers by adding neurons, or sparse connections.
Growing depth
Inserting entire layers or blocks to a network.
Function-preserving morphisms
Growth operations that preserve the model’s current function output.
