Adam maintains moving averages of both gradients and squared gradients. This allows it to use different effective learning rates for different parameters while also benefiting from momentum.
It is widely used because it performs well across many tasks without extensive tuning. However, alternatives such as SGD with momentum may sometimes provide better final generalisation.
Also in Neural Networks
JOIN NOW
Begin the first module
It is free, it is the real curriculum, and if it is not for you, you have lost nothing but an evening.
Join any time · Build AI skills at your pace