MathLive

Kullback–Leibler divergence

Measures how much one probability distribution P differs from a reference distribution Q, as the expected log-ratio of their probabilities under P. It is used in information theory, statistics and machine learning to compare models with data or with a target distribution.

D_{\mathrm{KL}}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right)

What the symbols mean

the true or observed probability distribution over the outcomes
the reference or approximating probability distribution
the set of possible outcomes (the sample space)
an outcome in the sample space
probability of outcome x under P
probability of outcome x under Q

About this formula

  • FieldStatistics
  • TopicInformation theory
  • LevelUniversity
  • Also calledKL divergence, relative entropy, information divergence, information gain, KLIC
  • SourceWikidata Q255166

Write it in Obsidian

MathLive is a free visual math editor for Obsidian. Run Find a formula, type “kullback–leibler divergence”, and it goes into your note as editable LaTeX. You can also scan formulas from screenshots, PDFs and handwriting.

Get MathLive for ObsidianSee how it works

Related formulas