Machine Learning | Artificial Intelligence

963 readers

5 users here now

Welcome to Machine Learning – a versatile digital hub where Artificial Intelligence enthusiasts unite. From news flashes and coding tutorials to ML-themed humor, our community covers the gamut of machine learning topics. Regardless of whether you're an AI expert, a budding programmer, or simply curious about the field, this is your space to share, learn, and connect over all things machine learning. Let's weave algorithms and spark innovation together.

founded 2 years ago

MODERATORS

Hopps

A newbie question on neural network (sopuli.xyz)

submitted 2 years ago by [email protected] to c/machinelearning

4 comments fedilink hide all child comments

In the hidden layer, the activation function will decide what is being determined by the neural network, is it possible for an AI to generate activation function for itself so it can improve upon itself?

you are viewing a single comment's thread
view the rest of the comments

[–] 80085 5 points 2 years ago* (last edited 2 years ago)

I just looked it up, and apparently someone implemented dynamic activation functions in a CNN: https://www.nature.com/articles/s41598-022-19020-y . I've never seen something like this elsewhere. I have included various activation functions in hyperparameter searches before full training to find the "best" one on datasets. I haven't really seen much of a difference in validation performance between activation functions.

Found another paper using dynamic activation functions with transformers: https://arxiv.org/pdf/2208.14111.pdf