Policy Gradient Theorem $[20$ points$]$

Given an MDP with a state space $S,$ Discrete action space $A=[a_1,a_2,a_3],$ Reward function $R,$

discount factor $,$ and a policy with the follwing functional representation:

$(a_1|s)=\frac{\text{e}\text{x}\text{p}(z(s,a_1))}{{\displaystyle \underset{\text{a}\text{i}\text{n}\text{A}}{\overset{?}{}}}\text{e}\text{x}\text{p}(z(s,a))}.$

Use the policy gradient theorem to show the follwing:

$gra{d}_{z}J()=d^{}(s)(a|s)A^{}(s,a),$

where $d^{}$ is the steady state distribution of the Markov chain induced by $$ and $A^{}(s,a)=$

$Q^{}(s,a)-V^{}(s)$