In the hidden layer, the activation function will decide what is being determined by the neural network, is it possible for an AI to generate activation function for itself so it can improve upon itself?
In the hidden layer, the activation function will decide what is being determined by the neural network, is it possible for an AI to generate activation function for itself so it can improve upon itself?
I just looked it up, and apparently someone implemented dynamic activation functions in a CNN: https://www.nature.com/articles/s41598-022-19020-y . I’ve never seen something like this elsewhere. I have included various activation functions in hyperparameter searches before full training to find the “best” one on datasets. I haven’t really seen much of a difference in validation performance between activation functions.
Found another paper using dynamic activation functions with transformers: https://arxiv.org/pdf/2208.14111.pdf