NORTH

NORTH [MRLW22] takes a geometric perspective: new neurons should add new directions in activation space, orthogonal to existing activations. It seeks \(\boldsymbol{\Psi}\) such that new activations \(\boldsymbol{H_{\text{ext}}}\) are orthogonal to existing activations \(\boldsymbol{H}^{(l-1)}\). NORTH uses this criterion as a trigger, adding neurons where and when \(\boldsymbol{H}^{(l-1)}\) rank is above a predefined threshold.

References

[MRLW22]

Kaitlin Maile, Emmanuel Rachelson, Hervé Luga, and Dennis George Wilson. When, where, and how to add new neurons to ANNs. In Proceedings of the First International Conference on Automated Machine Learning, 18/1–12. PMLR, 2022. ISSN: 2640-3498. URL: https://proceedings.mlr.press/v188/maile22a.html.