Adaptive Systems Theory Section

Defence Research Agency

St Andrews Rd

Malvern

Worcestershire

WR14 3PS

United Kingdom

A vector space **S** is partitioned into a number of
intersecting subspaces **(S_1, S_2, ... , S_n)**, where
**S_i \cap S_j \neq \emptyset** in general. The
corresponding vectors are denoted as **X** and
**(X_1, X_2, ... ,X_n)**, respectively, and the
**S_i** will be referred to as ``cliques''. Now suppose
that **n** marginal PDFs **Q_1(X_1), Q_2(X_2),
... , Q_n(X_n)** are supplied. These PDFs can be inconsistent
because the cliques overlap each other (via **S_i \cap S_j \neq
\emptyset)**; the PDFs are overcomplete. The problem is to
convert these PDFs into a consistent set of PDFs.

The full joint PDF of **X** is denoted as
**P(X)**, and its marginal PDFs as
**P_i(X_i)**. An average entropy is defined as **H
= \sum H_i**, where **H_i** is the entropy
**-\int dX_i P_i (X_i)\log(P_i (X_i)/Q_i (X_i))**. The
optimisation problem is to find the **P(X)** that
maximises **H**. This is
readily solved to yield

This equation can be solved in closed form for the marginal PDFs
**P_i(X_i)**, but, in general, it is not possible to
solve in closed form for the full joint PDF **P(X)**. The
solutions may be constructed by a recursive algorithm based on clique
intersections **S_i \cap S_j, (S_i \cap S_j)\cap(S_k \cap
S_l)**, etc.

For instance, suppose that **X = (x_1, x_2, x_3)**, which
is symmetrically partitioned as **(X_1, X_2, X_3) = ((x_1,
x_2), (x_2, x_3), (x_3, x_1))**. The maximum entropy marginal
PDFs are obtained by iterating the replacement rule

where **Z** is a normalisation factor. This converges to
the maximum entropy solution **P_1((x_1, x_2))**. The other
2 marginal PDFs **P_2((x_2, x_3))** and **P_3((x_3,
x_1))** are symmetrically related to **P_1((x_1,
x_2))**. Whilst this replacement rule may be obtained manually
with little difficulty, in cases where the cliques overlap in
complicated ways it is better to obtain machine-generated replacement
rules. A Mathematica package has been written to convert a set of
cliques written as a set of sets, {{1,2}, {2,3}, {3,1}} in
this example, into a set of replacement rules. The exponents that
appear in the solutions may conveniently be tabulated. All
inequivalent low-dimensional scenarios are considered, and a set of
tables of replacement rules is derived using the Mathematica
package.

This method of constructing a set of maximum entropy marginal PDFs may
very easily be implemented if the original marginal PDFs
**Q_1(X_1), Q_2(X_2), ... , Q_n(X_n)** are supplied as
multidimensional histograms. As the replacement rules are applied
iteratively they progressively massage the histograms into the
required maximum entropy form.

These methods can readily be generalised to the case of a weighted
average entropy **H = \sum w_i H_i** to produce
closed-form solutions. These solutions smoothly interpolate between
the solutions that are obtained in the unweighted case.

MaxEnt 94 Abstracts / mas@mrao.cam.ac.uk