Evolving Normalization-Activation Layers

Explore the evolution of normalization-activation layers in deep neural networks through this informative video. Dive into a proposed evolutionary search method aimed at discovering the optimal combined normalization-activation layer for specific settings. Learn about EvoNorms, a set of newly discovered layers that surpass existing design patterns, with some being independent from batch statistics. Examine the effectiveness of these layers across various image classification models, including ResNets, MobileNets, and EfficientNets, as well as their transferability to instance segmentation and image synthesis tasks. Gain insights into the search algorithm, mutation process, tournament selection, Pareto frontier, and rejection criteria used in the evolutionary approach. Understand the implications of this research for improving deep neural network performance across different applications.

Intro
Image Neural Networks
Evolving Layers
Search Space
Method Search
Mutation
Tournament
Pareto Frontier
Rejection Step
Rejection Criteria
Results