Learning rate range test for the vision transformer

Rinka Kiriyama; Akio Sashima; Ikuko Shimizu

doi:10.1117/12.2692013

21 August 2023 Learning rate range test for the vision transformer

Rinka Kiriyama, Akio Sashima, Ikuko Shimizu

Proceedings Volume 12783, International Conference on Images, Signals, and Computing (ICISC 2023); 1278303 (2023) https://doi.org/10.1117/12.2692013
Event: International Conference on Images, Signals, and Computing (ICISC2023), 2023, Chengdu, China

Abstract

The solutions obtained by training the deep neural network are highly dependent on the parameters including the learning rate. Therefore, finding the appropriate settings for training deep neural networks is very important. In particular, it is necessary to find the better settings for SOTA models of Vision Transformer(ViT), whose structure is different from ordinal models. In this paper, we focus on the learning rate to find a better value using the Learning Rate Range Test (LRRT). Through our experiments, we found that the appropriate LR is located where the decrease in loss value stops in the LRRT. In addition, we discuss about the effects of the number of epochs and the LR warm up.

Citation Download Citation

Rinka Kiriyama, Akio Sashima, and Ikuko Shimizu "Learning rate range test for the vision transformer", Proc. SPIE 12783, International Conference on Images, Signals, and Computing (ICISC 2023), 1278303 (21 August 2023); https://doi.org/10.1117/12.2692013

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available