Getting started
Introduction
Overview of topics
How to contribute
General best practices
Key principles of Computer Vision
Convolution
Computer Vision tasks
Classification / Tagging
Object Detection
Semantic Segmentation
Instance Segmentation
Panoptic Segmentation
Attribute Prediction
Model architectures
ResNet
Faster R-CNN
Mask R-CNN
DeepLabv3+
U-Net
FBNetV3
U-Net++
Efficient Net
PAN
PSPNet
LinkNet
FPN
RetinaNet
Cascade RCNN
FBNetV3IS
FBNetV3OD
CascadeMask R-CNN
HybridTask Cascade
Metrics
Confusion Matrix
Intersection over Union (IoU)
Accuracy
Hamming score
Precision
Recall
Precision-Recall curve and AUC-PR
F-beta score
Average Precision
mean Average Precision (mAP)
Average Loss
Loss
Comprehensive overview of loss functions in Machine Learning
Cross-Entropy Loss
Binary Cross-Entropy Loss
Focal loss
Bounding Box Regression Loss
CrossEntropyIoULoss2D
Solver / Optimizer
Comprehensive overview of solvers/optimizers in Deep Learning
Adam
- AMSgrad Variant (Adam)
SGD
Adadelta
Adagrad
AdaMax
Adamw
ASGD
Rprop
RMSprop
Lion
Weight Decay
Base Learning Rate
Momentum (SGD)
- Dampening (SGD)
Epsilon Coefficient
Training Parameters
Patience
Min delta
Seed
Batch Size
Iterations
Epoch
Scheduler
Comprehensive overview of learning rate schedulers in Machine Learning
ExponentialLR
CyclicLR
StepLR
MultiStepLR
ReduceLROnPlateau
CosineAnnealingLR
Augmentations
Comprehensive overview of augmentations in Machine Learning
Horizontal Flip
Vertical Flip
Random Crop
Random Sized Crop
Rotate
Resize
Blur
Smallest max size
Center Crop
Color Jitter
Gaussian Noise
Shift Scale Rotate
Longest max size
Equalize
To gray
Shear
Mosaic
Copy Paste
Extrapolation methods
Interpolation methods
Deployment
Primitive deployment using web frameworks
Commonly used web frameworks
Containerized Deployment
Orchestrated Deployment
Challenges of Deployment
Splits
General Information
Random
Stratification

ASGD

Average Stochastic Gradient Descent, abbreviated as ASGD, averages the weights that are calculated in every iteration.

$$$w_{t+1}=w_t-\eta \nabla Q(w_t)$$$

where $$w_t$$ being the weight tensor , $$\eta$$ being the base learning rate and $$\nabla Q(w_t)$$ being the gradient of the objective function evaluated at $$w_t$$ .

With the given update rule SGD assigns calculated weight to the model. But with ASGD assigns the following averaged weight $$\overline{w}$$ ,

$$$\overline{w}=\frac{1}{N} \sum_{t=1}^Nw_t$$$

where $$w_t$$ is the weight tensor calculated in iteration 't'.

Such averaging is used when the data is noisy.

Major Parameters

Lambda

It is the decay term for the past weights used in the average.

Alpha

It is the power value that is used to update the learning rate.

TO

It is the optimization step at which the averaging is started. If the required number of iteration is lower than the TO value, then the averaging will not happen.

Code Implementation

    # importing the library
import torch
import torch.nn as nn

x = torch.randn(10, 3)
y = torch.randn(10, 2)

# Build a fully connected layer.
linear = nn.Linear(3, 2)

# Build MSE loss function and optimizer.
criterion = nn.MSELoss()

# Optimization method using ASGD
optimizer = torch.optim.ASGD(linear.parameters(), lr=0.01, lambd=0.0001, 
alpha=0.75, t0=1000000.0, weight_decay=0)

# Forward pass.
pred = linear(x)

# Compute loss.
loss = criterion(pred, y)
print('loss:', loss.item())

optimizer.step()

Last updated on May 17, 2023

Adamw

Rprop