Joint Entropy, Marginal & Conditional Entropy

Explanation

1. Purpose

This program calculates the marginal probabilities, marginal entropies ($H(X), H(Y)$), joint entropy ($H(X,Y)$), and conditional entropies ($H(X|Y), H(Y|X)$) for two discrete random variables given their joint probability distribution matrix.

2. Key Concepts

• Nested Loops: Used to traverse 2D matrices row-by-row and column-by-column.
• Accumulators: Patterns like sum = sum + value used to compute sums and Shannon entropy: $-\sum p \log_2(p)$.
• Matrix Reduction: Summing across rows or columns to convert joint probabilities into marginal probabilities.
• Formatted Output: Using fprintf with format specifiers (e.g., %.4f, \t, \n) to display clean numerical results.

3. Code Walkthrough

1. Initialization:

   • Pxy defines the $2 \times 2$ joint probability matrix where rows represent variable $X$ and columns represent variable $Y$.
   • [m, n] = size(Pxy) extracts the dimensions ($2 \times 2$).

2. Marginal Probabilities ($P_x$ and $P_y$):

   • Computes $P(X)$ by summing across each row: $P(x_j) = \sum_k P(x_j, y_k)$.
   • Computes $P(Y)$ by summing across each column: $P(y_k) = \sum_j P(x_j, y_k)$.

3. Entropy Calculations:

   • Marginal Entropies (Hx, Hy): Uses standard Shannon entropy formula: $-\sum P \log_2(P)$ for each variable.
   • Joint Entropy (Hxy): Computes uncertainty of both variables combined: $-\sum \sum P_{xy} \log_2(P_{xy})$.

4. Conditional Entropy:

   • Uses the chain rule identities to solve for remaining uncertainty:
     • $H(X|Y) = H(X,Y) - H(Y)$
     • $H(Y|X) = H(X,Y) - H(X)$

4. Output

0.1000	 0.3000
0.2000	 0.4000

Px = [0.30 0.70]
Py = [0.40 0.60]
Hx = 0.8813
Hy = 0.9710
Hxy = 1.8464
H(x|y) = 0.8755
H(y|x) = 0.9652

(Note: In MATLAB, matrices are printed column-wise in fprintf, which is why the first printout appears transposed).