We give a new characterization of relative entropy, also known as the Kullback--Leibler divergence. We use a number of interesting categories related to probability theory. In particular, we consider a category FinStat where an object is a finite set equipped with a probability distribution, while a morphism is a measure-preserving function $f \maps X \to Y$ together with a stochastic right inverse $s \maps Y \to X$. The function $f$ can be thought of as a measurement process, while $s$ provides a hypothesis about the state of the measured system given the result of a measurement. Given this data we can define the entropy of the probability distribution on $X$ relative to the `prior' given by pushing the probability distribution on $Y$ forwards along $s$. We say that $s$ is `optimal' if these distributions agree. We show that any convex linear, lower semicontinuous functor from FinStat to the additive monoid $[0,\infty]$ which vanishes when $s$ is optimal must be a scalar multiple of this relative entropy. Our proof is independent of all earlier characterizations, but inspired by the work of Petz.
Keywords: relative entropy, Kullback-Leibler divergence, measures of information, categorical probability theory
2010 MSC: Primary 94A17, Secondary 62F15, 18B99
Theory and Applications of Categories, Vol. 29, 2014, No. 16, pp 421-456.
Published 2014-08-12.
TAC Home