If you have ever worked on a Computer Vision project, you might know that using a learning rate scheduler might significantly increase your model training performance. On this page, we will:
- Сover the Step Learning Rate (StepLR) scheduler;
- Check out its parameters;
- See a potential effect from StepLR on a learning curve;
- And check out how to work with StepLR using Python and the PyTorch framework.
Let’s jump in.
StepLR is a scheduling technique that decays the learning rate by gamma every N epochs (or every N evaluation periods, if iteration training is used).
Compared to
ExponentialLR, which divides the learning rate
every epoch, the StepLR keeps the learning rate the
same over
N epochs before reducing it.
- Step Size - the decay of the learning rate happens every N epochs. This "N" is the step size.
- Gamma - a multiplicative factor by which the learning rate is decayed. For instance, if the learning rate is 1000 and gamma is 0.5, the new learning rate will be 1000 x 0.5 = 500.
The gamma value should be less than 1 to reduce the learning rate.
import torch
model = [Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=0.01, amsgrad=False)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1, last_epoch=-1, verbose=False)
for epoch in range(20):
for input, target in dataset:
optimizer.zero_grad()
output = model(input)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()
scheduler.step()
Accelerated Annotation.
Maximize model performance quickly with AI-powered labeling and 100% QA.
Learn more