bham.leakiest.infotheory

Class ShannonEntropy

java.lang.Object

bham.leakiest.infotheory.ShannonEntropy

public class ShannonEntropy
extends java.lang.Object

This is a library of useful information theory definitions related to Shanon-entropy-based information measure.
Probability Mass Functions are represented by an array of doubles p(element i) = pmf[i] or by ProbDist class.

Version:

1.4

Author:

Tom Chothia, Yusuke Kawamoto

Field Summary

Fields

Modifier and Type	Field and Description
static int	base_log Base when computing the logarithms.

Constructor Summary

Constructors

Constructor and Description
ShannonEntropy()

Method Summary

Modifier and Type	Method and Description
static double	conditionalEntropy(double[] pmf, double[][] matrix_W) Calculates the conditional entropy of a channel matrix given an input PMF.
static double	conditionalEntropy(ProbDist pd, Channel channel) Calculates the conditional entropy of a channel given an input probability distribution.
static double	D(double[] p, double[] q) Calculates the relative entropy of two PMFs p and q.
static double	entropy(double[] pmf) Calculates the entropy of a PMF array.
static double	entropy(ProbDist pd) Calculates the entropy of a probability distribution.
static double	H(double[] pmf) Calculates the entropy of a PMF array.
static double	H(double[] pmf, double[][] W) Calculates the conditional entropy of a channel matrix given an input PMF.
static double	H(ProbDist pd) Calculates the entropy of a probability distribution.
static double	H(ProbDist pd, Channel channel) Calculates the conditional entropy of a channel given an input probability distribution.
static double	I(double[] Q, double[][] W) Calculates mutual information between an input PMF and a channel matrix.
static double	I(ProbDist pd, Channel channel) Calculates mutual information between an input PMF and a channel matrix.
static double	jointEntropy(double[][] p) Calculates the joint entropy of X and Y where p(x,y) = p[x_index][y_index].
static double	KullbackLeibler(double[] p, double[] q) Calculates the relative entropy of two PMFs p and q.
static double	MIuniformInput(double[][] W) Calculates the mutual information given an uniform input distribution.
static double	mutualInformation(double[] Q, double[][] W) Calculates mutual information between an input PMF and a channel matrix.
static double	mutualInformation(ProbDist pd, Channel channel) Calculates mutual information between an input PMF and a channel matrix.
static double	relativeEntropy(double[] p, double[] q) Calculates the relative entropy of two PMFs p and q.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

base_log

public static final int base_log

Base when computing the logarithms.

See Also:

Constant Field Values

Constructor Detail

ShannonEntropy

public ShannonEntropy()

Method Detail

conditionalEntropy

public static double conditionalEntropy(double[] pmf,
                                        double[][] matrix_W)

Calculates the conditional entropy of a channel matrix given an input PMF.
N.B. the inputs to this function are a PMF and a Channel Matrix that links that PMF with the other PMF i.e., X and the channel to Y The inputs are NOT two pmfs i.e., H(Q,W) not H(X|Y) matrix_W[input][output] = W(output|input).

Parameters:

pmf - PMF array

matrix_W - channel matrix array

Returns:

the conditional entropy of matrix_W given pmf

conditionalEntropy

public static double conditionalEntropy(ProbDist pd,
                                        Channel channel)

Calculates the conditional entropy of a channel given an input probability distribution.

Parameters:

pd - input probability distribution

channel - channel

Returns:

the conditional entropy of channel given pd

D

public static double D(double[] p,
                       double[] q)
                throws java.lang.ArithmeticException

Calculates the relative entropy of two PMFs p and q. Produces the same result as relativeEntropy(double[] p, double[] q).

Parameters:

p - PMF array

q - PMF array

Returns:

the relative entropy of two PMFs p and q

Throws:

java.lang.ArithmeticException - Exception occured by division by zero

entropy

public static double entropy(double[] pmf)

Calculates the entropy of a PMF array.

Parameters:

pmf - PMF array

Returns:

the entropy of pmf

entropy

public static double entropy(ProbDist pd)

Calculates the entropy of a probability distribution.

Parameters:

pd - probability distribution

Returns:

the entropy of pd

H

public static double H(double[] pmf)

Calculates the entropy of a PMF array.

Parameters:

pmf - PMF array

Returns:

the entropy of pmf

H

public static double H(double[] pmf,
                       double[][] W)

Calculates the conditional entropy of a channel matrix given an input PMF. Produces the same result as conditionalEntropy(double[] pmf, double[][] matrix_W).

Parameters:

pmf - PMF array

W - channel matrix array

Returns:

the conditional entropy of W given pmf

H

public static double H(ProbDist pd)

Calculates the entropy of a probability distribution.

Parameters:

pd - probability distribution

Returns:

the entropy of pd

H

public static double H(ProbDist pd,
                       Channel channel)

Calculates the conditional entropy of a channel given an input probability distribution. Produces the same result as conditionalEntropy(ProbDist pd, Channel channel).

Parameters:

pd - input probability distribution

channel - channel

Returns:

the conditional entropy of channel given pd

I

public static double I(double[] Q,
                       double[][] W)

Calculates mutual information between an input PMF and a channel matrix. Produces the same result as mutualInformation(double[] Q, double[][] W).

Parameters:

Q - input PMF array

W - channel matrix

Returns:

the mutual information between Q and W

I

public static double I(ProbDist pd,
                       Channel channel)

Calculates mutual information between an input PMF and a channel matrix. Produces the same result as mutualInformation(ProbDist pd, Channel channel).

Parameters:

pd - input probability distribution

channel - channel

Returns:

the mutual information between pd and channel

jointEntropy

public static double jointEntropy(double[][] p)

Calculates the joint entropy of X and Y where p(x,y) = p[x_index][y_index].
H( (X,Y) ) = - Σ_x Σ_y p(x,y).log(p(x,y))

Parameters:

p - the joint probability of X and Y

Returns:

the joint entropy of X a d Y

KullbackLeibler

public static double KullbackLeibler(double[] p,
                                     double[] q)
                              throws java.lang.ArithmeticException

Calculates the relative entropy of two PMFs p and q. Produces the same result as relativeEntropy(double[] p, double[] q).

Parameters:

p - PMF array

q - PMF array

Returns:

the relative entropy of two PMFs p and q

Throws:

java.lang.ArithmeticException - Exception occured by division by zero

MIuniformInput

public static double MIuniformInput(double[][] W)

Calculates the mutual information given an uniform input distribution.

Parameters:

W - channel matrix

Returns:

The mutual information

mutualInformation

public static double mutualInformation(double[] Q,
                                       double[][] W)

Calculates mutual information between an input PMF and a channel matrix.
N.B. the inputs to this function are a PMF and a Channel Matrix that links that PMF with the other PMF i.e., X and the channel to Y. The inputs are NOT two pmfs i.e., I(Q,W) not I(X;Y).
This method returns I(Q,W) = I(X;Y) = H(X) - H(X|Y).

Parameters:

Q - input PMF array

W - channel matrix

Returns:

mutual information between Q and W

mutualInformation

public static double mutualInformation(ProbDist pd,
                                       Channel channel)

Calculates mutual information between an input PMF and a channel matrix.

Parameters:

pd - input probability distribution

channel - channel

Returns:

the mutual information between pd and channel

relativeEntropy

public static double relativeEntropy(double[] p,
                                     double[] q)
                              throws java.lang.ArithmeticException

Calculates the relative entropy of two PMFs p and q.
D(p||q) = Σ_x p(x) log(p(x)/q(x))
0.log(0/q) = 0 and p.log(p/0) = inf, (we throw an exception rather that return inf)

Parameters:

p - PMF array

q - PMF array

Returns:

the relative entropy of two PMFs p and q

Throws:

java.lang.ArithmeticException - Exception occured by division by zero