At state x , with probability 1 the state transits to y1 , i.e.,

P(y1|x)=1.

Then at state y1 , we have

P(y1|y1)=p,P(y2|y1)=1p,

which says there is probability p we stay in y1 and probability 1p the state transits to y2 . Finally, state y2 is the absorbing state so that

P(y2|y2)=1.

The instant reward is set as 1 for starting in state y1 and 0 elsewhere:

R(y1,a,y1)=1,R(y1,a,y2)=1,,R(s,a,s)=0 otherwise.

The discount factor is denoted by ( 0<<1 ).

My problem is defining this with p and 1-p . It confuses me. I know how to do Bellman equations when they involve the usual T, R and V* .

This is the question:

Define V(y1) as the optimal value function of the state y1 . Compute V(y1) via Bellman's Equation. (The answer is a formula in terms of ,p ).

V(y1)=

Find Q(x,a) .

Q(x,a)=