Simulation The following problem requires some programming using R, or any comparable programming language 10. Kullback-Leibler divergence (KL) of two distributions px and py is defined as D(px, PY) = EPx (x) log PX (2) (PY (3) KL divergence is a measure of discrepancy between two distributions with D = 0 for px = py and D > 0 for px + py. Using KL divergence, we then define mutual information (MI) for two jointly distributed r.v.'s as I( X, Y ) = D(PX,Y , PXPY), that is KL divergence between the joint distribution px,y and the product of the marginals px and py. The latter is the hypothetical joint distribution under independence. In this example we use MI to judge whether two variables X, Y are independent or not. We record n pairs (Xi, Yi) ~Px,Y, i = 1, ..., n, independently (data). Here Xi E {1, 2, 3, 4} and Yi E {1, 2, 3}. The following (4 x 3) table summarizes the data by reporting nx,y (x, y) = #{(Xi, Yi) = (x, y)}, the count of observations with Xi = x and Yi = y. Table 1. counts nx,y (x, y) (center block), nx (x) (right column) and ny (y) (bottom row). y 1 2 3 10 9 2 21 10 7 19 17 11 5 33 4 22 18 7 47 59 45 16 The row totals are nx(x) = #{Xi = x} and similarly the column totals report ny (y) = #{Yi = y}. Let fx,y(x, y) = nx,y(x, y) denote the ( relative) frequencies, and simiarly for fx (x) and fy(y). We use fx ~ px as an estimate for px, and fy ~ py as an estimate for fy. Then i = Efx,y (x, y) log ( fx, Y (x,y) fx (x) fy (y) x, yserves as estimate for 1 (A, Y ). ' 1n the tollowrng questlons we implement a poss1hle approach to decrde whether to report that X I Y, or X ,K Y. The logic is (a) If X I Y were true, then [(X, Y) : 0. (b) Instead of I(X,Y) we can only evaluate I m [(X, Y). It is okay for f > 0, but it should not be \"too large'7 if X I Y were true; (c) To judge how much is \"too large\\f