Suppose we run value iteration in an MDP with only non-negative rewards (that is, R(s, a, s') ≥ 0 for any (s, a, s')). Let the values on the kth iteration be V_k(s) and the optimal values be V^∗ (s). Initially, the values are 0 (that is, V₀(s) = 0 for any s).

a. Mark all of the options that are guaranteed to be true. 

(i) For any s, a, s' , V₁(s) = R(s, a, s') 

(ii) For any s, a, s' , V₁(s) ≤ R(s, a, s') 

(iii) For any s, a, s' , V₁(s) ≥ R(s, a, s') 

(iv) None of the above are guaranteed to be true. 

b. Mark all of the options that are guaranteed to be true. 

(i) For any k, s, V_k(s) = V^∗ (s) 

(ii) For any k, s, V_k(s) ≤ V^∗ (s) 

(iii) For any k, s, V_k(s) ≥ V^∗ (s) 

(iv) None of the above are guaranteed to be true.