Getting started
Introduction
Overview of topics
How to contribute
General best practices
Key principles of Computer Vision
Convolution
Computer Vision tasks
Classification / Tagging
Object Detection
Semantic Segmentation
Instance Segmentation
Panoptic Segmentation
Attribute Prediction
Model architectures
ResNet
Faster R-CNN
Mask R-CNN
DeepLabv3+
U-Net
FBNetV3
U-Net++
Efficient Net
PAN
PSPNet
LinkNet
FPN
RetinaNet
Cascade RCNN
FBNetV3IS
FBNetV3OD
CascadeMask R-CNN
HybridTask Cascade
Metrics
Confusion Matrix
Intersection over Union (IoU)
Accuracy
Hamming score
Precision
Recall
Precision-Recall curve and AUC-PR
F-beta score
Average Precision
mean Average Precision (mAP)
Average Loss
Loss
Comprehensive overview of loss functions in Machine Learning
Cross-Entropy Loss
Binary Cross-Entropy Loss
Focal loss
Bounding Box Regression Loss
CrossEntropyIoULoss2D
Solver / Optimizer
Comprehensive overview of solvers/optimizers in Deep Learning
Adam
- AMSgrad Variant (Adam)
SGD
Adadelta
Adagrad
AdaMax
Adamw
ASGD
Rprop
RMSprop
Lion
Weight Decay
Base Learning Rate
Momentum (SGD)
- Dampening (SGD)
Epsilon Coefficient
Training Parameters
Patience
Min delta
Seed
Batch Size
Iterations
Epoch
Scheduler
Comprehensive overview of learning rate schedulers in Machine Learning
ExponentialLR
CyclicLR
StepLR
MultiStepLR
ReduceLROnPlateau
CosineAnnealingLR
Augmentations
Comprehensive overview of augmentations in Machine Learning
Horizontal Flip
Vertical Flip
Random Crop
Random Sized Crop
Rotate
Resize
Blur
Smallest max size
Center Crop
Color Jitter
Gaussian Noise
Shift Scale Rotate
Longest max size
Equalize
To gray
Shear
Mosaic
Copy Paste
Extrapolation methods
Interpolation methods
Deployment
Primitive deployment using web frameworks
Commonly used web frameworks
Containerized Deployment
Orchestrated Deployment
Challenges of Deployment
Splits
General Information
Random
Stratification

Adadelta

Adadelta was proposed with the aim to solve the diminishing learning rate problem that was seen in the Adagrad. Adagrad uses the knowledge of all the past gradients in its update, whereas Adadelta uses only a certain "window" of past gradients to update the parameters.

The mathematical formulation of Adadelta is similar to that of RMSprop.

Both Adadelta and RMSprop were developed independently to eliminate the problem of Adagrad. They are suitable for optimizing non-stationary and non-convex problems.

Major Parameters

Rho

Rho is same as the $beta$ of RMSprop. It is the smoothing constant whose value ranges from 0 to 1. Higher value of Rho suggests that more number of previously calculated squares of gradient are taken into account, making the curve relatively "smooth".

Code Implementation

    # importing the library
import torch
import torch.nn as nn

x = torch.randn(10, 3)
y = torch.randn(10, 2)

# Build a fully connected layer.
linear = nn.Linear(3, 2)

# Build MSE loss function and optimizer.
criterion = nn.MSELoss()

# Optimization method using RMSprop
optimizer = torch.optim.RMSProp(linear.parameters(), lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)

# Forward pass.
pred = linear(x)

# Compute loss.
loss = criterion(pred, y)
print('loss:', loss.item())

optimizer.step()

Last updated on May 17, 2023

SGD

Adagrad