AutoGrow¶
AutoGrow [WYCL20] considers the problem of increasing the number of blocks in ResNet [HZRS16] and VGG [SZ15] style architectures, by organising the network into several “stages”. The first block in each stage implements a downsampling of the spatial resolution, after which the spatial resolution is fixed for the remaining blocks in that stage. By increasing the number of blocks, one can grow the network to an arbitrary depth while respecting shape constraints. They contest the Net2Net notion that function-preserving morphisms are the best way to initialise new layer weights, and instead prefer random initialisation [WYCL20]. This has corroborated by later layer-growing studies [WWM+24].
References¶
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR. December 2016. arXiv:1512.03385. URL: http://arxiv.org/abs/1512.03385, doi:10.48550/arXiv.1512.03385.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. April 2015. arXiv:1409.1556. URL: http://arxiv.org/abs/1409.1556, doi:10.48550/arXiv.1409.1556.
Wei Wen, Feng Yan, Yiran Chen, and Hai Li. AutoGrow: Automatic Layer Growing in Deep Convolutional Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, 833–841. New York, NY, USA, August 2020. Association for Computing Machinery. URL: https://dl.acm.org/doi/10.1145/3394486.3403126, doi:10.1145/3394486.3403126.
Haihang Wu, Wei Wang, Tamasha Malepathirana, Damith Senanayake, Denny Oetomo, and Saman Halgamuge. When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks. In AAAI, volume 38, 5994–6002. 2024. Number: 6. URL: https://ojs.aaai.org/index.php/AAAI/article/view/28414, doi:10.1609/aaai.v38i6.28414.