StepLR

If you have ever worked on a Computer Vision project, you might know that using a learning rate scheduler might significantly increase your model training performance. On this page, we will:

• Сover the Step Learning Rate (StepLR) scheduler;
• Check out its parameters;
• See a potential effect from StepLR on a learning curve;
• And check out how to work with StepLR using Python and the PyTorch framework.

Let’s jump in.

StepLR explained

StepLR is a scheduling technique that decays the learning rate by gamma every N epochs (or every N evaluation periods, if iteration training is used).

Parameters

• Step Size - the decay of the learning rate happens every N epochs. This "N" is the step size.
• Gamma - a multiplicative factor by which the learning rate is decayed. For instance, if the learning rate is 1000 and gamma is 0.5, the new learning rate will be 1000 x 0.5 = 500.

StepLR visualized

Code Implementation

    import torch
model = [Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=0.01, amsgrad=False)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1, last_epoch=-1, verbose=False)
for epoch in range(20):
    for input, target in dataset:
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
    scheduler.step()
    Copy
  

Learn more about other schedulers…

• ExponentialLR;
• CyclicLR;
• MultiStepLR;
• ReduceLROnPlateau;
• CosineAnnealingLR.