Summary: Draft of 30 Jan 2005 4:38 p.m.
Constrained Entropy, Free Energy, and the Legendre Transform
S. M. Aji, S. L. Fogal, R. J. McEliece, and B. Wang
California Institute of Technology
Abstract. The Legendre transform is well known to physicists but perhaps not so well
known to information theorists. To remedy this, in this paper we carefully describe the
Legendre transform and illustrate its usefulness to information theorists by showing that it
transforms a broad class of maximum entropy functions (x) into "free energy" functions
F(s) which are easier to compute. Then if s
is the unique solution to the equation
F(s) = x, then it is advantageous to compute (x) by the formula
(x) = x · s
Alternatively, we can use F(s) to give a parametric description of (x):
x = F(s)
= F(s) · s - F(s)
for s L, the domain of f(s).
1. The Legendre Transform for Information Theorists.
In this section we give our version of the Legendre transform, specially tailored for infor-