please solve part b(ii)

for your help i attach part b(i) solution below

Q3. Softmax Regression (a) Suppose X = {x1, ..., In} is a finite event space with the probability pi for event xi. Then recall its entropy is given by n H(P1, . . . , Pn) - - _ pi log pi i=1 Show that the entropy is maximized when pi - -, i.e. uniformly distributed. Hint: Maximize H using Lagrange multiplier with the constraint _t-1 Pi = 1. (b) The following exercise is to show that the Kullback-Leibler Divergence is a nonnegative function and is zero if and only if two probability distribution are equal. Suppose again X = {x1,. .., n} is a finite event space with nonzero probability pi and qi for event ci. Then recall the Kullback-Leibler Divergence of p relative to q is n DKL(Pllq) = - _pilos ($2 i-1 (i) Show that log z 0, with equality if and only if z = 1. (ii) Using part (i), show that n Epi log 120 , I i= 1 and conclude the Kullback-Leibler Divergence is a nonnegative function. (iii) Using parts (i) and (ii), show that DKL(pl|q) = 0 if and only if Pi = qibli) To show that log z s Z- 1 for all zZO, with equality if and only if 2 = 1 For this purpose . Let us consider a function fiz) = z - 1 - log (2) for 270 . then f ( z ) = 4 - 1 = 2 -1 f ( 2 ) 7 0 when 271, and f ' ( 2 ) Lo when OL Z L 1 and f' ( 2 ) = 0 when 2 = 1 . Z =1 is the critical point. Also F LODZ f"( 2) = 12 * f( 1 ) = 1 7 0 f (z ) has a minima at 2 = 1. f (+ ) E f (Z ) V 270 1 - 1 - log ( 4 ) 2 2 - 1 - 10gz , V270 0 5 Z - 1 - logz , V270 fi log ( +) 2 0 log Z E Z-1 V Z70 since minimum occurs only at 2=1 . hence log z = Z-i only for Z=1. Hence logz s Z -1 for all 270 , with equality if and only if 2= 1