Entropy

Entropy is a heuristic that measures the model’s uncertainty about the class of the image. The higher is entropy – the more “confused” the model is. As a result, data samples with higher entropy are ranked higher and offered for annotation. 

Below is a mathematical formula for entropy calculation:

Imagine that our model predicts whether an object is a dog or a muffin. Let’s say it gave the following probabilities for instances A and B:

• Instance A: “dog” – 0.5, “muffin” – 0.5.
  Entropy = H([0.5,0.5]) = 1.0
• Instance B: “dog” – 1.0,  “muffin” – 0.0.
  Entropy = H([1.0,0.0]) = 0.0

As you see, the model was completely confused when the class probabilities were equal. In contrast, when one of the classes had a 100% probability, the entropy was equal to 0 since there was no uncertainty.

Entropy might be useful in datasets with very diverse images and classes. 

Learn more about the other heuristics:

• Variance
• Margin