Mutual Information

Explanation

1. Purpose

This program calculates the Mutual Information $I(X;Y)$ between two discrete random variables, $X$ and $Y$, from a given joint probability distribution matrix $P(X,Y)$. It measures how much information variable $Y$ reveals about variable $X$.

2. Key Concepts

• Matrix Reductions (sum): Summing along specific dimensions (rows vs. columns) to find marginal distributions.
• Logical Indexing (Px > 0): Filtering out zero probabilities to avoid log2(0) mathematical errors ($-\infty$).
• Nested Loops: Traversing a 2D matrix element-by-element to compute conditional probabilities and entropy.
• Formatted Output (fprintf): Displaying floating-point numbers rounded to 4 decimal places.

3. Code Walkthrough

1. Normalization & Marginals:

   • P / sum(P(:)) ensures the joint probabilities sum up to $1$.
   • sum(P, 2) sums across columns to give the marginal $P(X)$.
   • sum(P, 1) sums across rows to give the marginal $P(Y)$.

2. Entropy $H(X)$:

   • Uses the formula $H(X) = -\sum P(x) \log_2 P(x)$.
   • Calculated in a single vectorized step using logical masking Px(Px > 0) to ignore zeros.

3. Conditional Entropy $H(X|Y)$:

   • Uses nested for loops to iterate through each cell $(x, y)$.
   • Computes conditional probability: $P(x|y) = \frac{P(x,y)}{P(y)}$.
   • Accumulates the sum: $H(X|Y) = -\sum_{x,y} P(x,y) \log_2 P(x|y)$.

4. Mutual Information $I(X;Y)$:

   • Applies the information-theoretic identity: $I(X;Y) = H(X) - H(X|Y)$.

4. Output

Entropy H(X): 1.5813 bits
Conditional Entropy H(X|Y): 1.4631 bits
Mutual Information I(X;Y): 0.1182 bits