Learning-rate annealing methods for deep neural networks
Research output: Contribution to journal › Journal article › Research › peer-review
Documents
- Learning-Rate Annealing Methods for Deep Neural Networks
Final published version, 3.38 MB, PDF document
Deep neural networks (DNNs) have achieved great success in the last decades. DNN is optimized using the stochastic gradient descent (SGD) with learning rate annealing that overtakes the adaptive methods in many tasks. However, there is no common choice regarding the scheduled-annealing for SGD. This paper aims to present empirical analysis of learning rate annealing based on the experimental results using the major data-sets on the image classification that is one of the key applications of the DNNs. Our experiment involves recent deep neural network models in combination with a variety of learning rate annealing methods. We also propose an annealing combining the sigmoid function with warmup that is shown to overtake both the adaptive methods and the other existing schedules in accuracy in most cases with DNNs.
Original language | English |
---|---|
Article number | 2029 |
Journal | Electronics (Switzerland) |
Volume | 10 |
Issue number | 16 |
Number of pages | 12 |
DOIs | |
Publication status | Published - 2021 |
Bibliographical note
Publisher Copyright:
© 2021 by the authors. Licensee MDPI, Basel, Switzerland.
- Image classification, Learning rate annealing, Stochastic gradient descent
Research areas
ID: 279140643