A Maximum Entropy Method of Reconciling Inconsistent Marginal PDFs

Stephen P Luttrell
Adaptive Systems Theory Section
Defence Research Agency
St Andrews Rd
WR14 3PS
United Kingdom


In this paper maximum entropy methods are used to derive a scheme for converting an inconsistent set of marginal PDFs into a consistent set. This technique could find application in Bayesian data fusion where one cannot perform a full Bayesian analysis based on the raw data. For instance, raw data might be collected by a number of distributed sensors, and there might be only a limited bandwidth communication link for collating the data at a ``data fusion centre''; this is a very common scenario. One possible approach would then be to perform a limited Bayesian analysis separately at each sensor, and to transmit only posterior PDFs back to the data fusion centre, where the various PDFs must be combined. There are many variations on this approach.

A vector space S is partitioned into a number of intersecting subspaces (S_1, S_2, ... , S_n), where S_i \cap S_j \neq \emptyset in general. The corresponding vectors are denoted as X and (X_1, X_2, ... ,X_n), respectively, and the S_i will be referred to as ``cliques''. Now suppose that n marginal PDFs Q_1(X_1), Q_2(X_2), ... , Q_n(X_n) are supplied. These PDFs can be inconsistent because the cliques overlap each other (via S_i \cap S_j \neq \emptyset); the PDFs are overcomplete. The problem is to convert these PDFs into a consistent set of PDFs.

The full joint PDF of X is denoted as P(X), and its marginal PDFs as P_i(X_i). An average entropy is defined as H = \sum H_i, where H_i is the entropy -\int dX_i P_i (X_i)\log(P_i (X_i)/Q_i (X_i)). The optimisation problem is to find the P(X) that maximises H. This is readily solved to yield

P_1(X_1) P_2(X_2) ... P_n(X_n) \propto Q_1(X_1) Q_2(X_2) ... Q_n(X_n)

This equation can be solved in closed form for the marginal PDFs P_i(X_i), but, in general, it is not possible to solve in closed form for the full joint PDF P(X). The solutions may be constructed by a recursive algorithm based on clique intersections S_i \cap S_j, (S_i \cap S_j)\cap(S_k \cap S_l), etc.

For instance, suppose that X = (x_1, x_2, x_3), which is symmetrically partitioned as (X_1, X_2, X_3) = ((x_1, x_2), (x_2, x_3), (x_3, x_1)). The maximum entropy marginal PDFs are obtained by iterating the replacement rule

Q_1((x_1, x_2)) \rightarrow Z^{-1} Q_1((x_1, x_2))
Q_2^{\frac{1}{2}}((x_2)) Q_3^{\frac{1}{2}}((x_1)) /
Q_1^{\frac{1}{2}}(x_1) Q_1^{\frac{1}{2}}(x_2)
where Z is a normalisation factor. This converges to the maximum entropy solution P_1((x_1, x_2)). The other 2 marginal PDFs P_2((x_2, x_3)) and P_3((x_3, x_1)) are symmetrically related to P_1((x_1, x_2)). Whilst this replacement rule may be obtained manually with little difficulty, in cases where the cliques overlap in complicated ways it is better to obtain machine-generated replacement rules. A Mathematica package has been written to convert a set of cliques written as a set of sets, {{1,2}, {2,3}, {3,1}} in this example, into a set of replacement rules. The exponents that appear in the solutions may conveniently be tabulated. All inequivalent low-dimensional scenarios are considered, and a set of tables of replacement rules is derived using the Mathematica package.

This method of constructing a set of maximum entropy marginal PDFs may very easily be implemented if the original marginal PDFs Q_1(X_1), Q_2(X_2), ... , Q_n(X_n) are supplied as multidimensional histograms. As the replacement rules are applied iteratively they progressively massage the histograms into the required maximum entropy form.

These methods can readily be generalised to the case of a weighted average entropy H = \sum w_i H_i to produce closed-form solutions. These solutions smoothly interpolate between the solutions that are obtained in the unweighted case.

MaxEnt 94 Abstracts / mas@mrao.cam.ac.uk