Firefly

Firefly [WLSL20] splits existing layers and add new neurons. Then it optimizes those new parameters through classical gradient descent. Finally, only the most loss-improving architecture changes are kept.

References

[WLSL20]

Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks. In NeurIPS. 2020. URL: https://proceedings.neurips.cc/paper_files/paper/2020/hash/fdbe012e2e11314b96402b32c0df26b7-Abstract.html.