Consider a queue$-$system where packets arrive sequentially. The queue is generally used to model packet arrivals in many applications such as routers, server,

and cloud. Queue$-$system is also used to model vehicles arrival at the transportation nodes $($e$.$g$.,$ intersection$).$ At a given time the number of packets arrive is a

random variable $\backslash$Lambda with probability distribution p$(\backslash$Lambda $=\backslash$lambda $)=$ p$\backslash$lambda independent of

other time$-$steps. At a given time t$,$ the number of packets waiting to be served

is given by q$($t$)($also known as the queue$-$state$).$ The packets which are served

is given by the random variable D with probability p$($D $=$ d$)=$ pd $($independent

of other time steps$).$ The packets those are served leave the queue. The random

variable $\backslash$Lambda and D take integer values between $[0,10].$

$1.$ Express the state as s$($t$)=$ q$($t$).$ Write the state$-$evolution process, i$.$e$.,$

write the probability of reaching s$($t $+1)$ given the state is s$($t$).$ Is it a

Markov chain?

$2.$ Now assume that you have N such queues with queue$-$states q$($t$)=\{$q$1($t$),...,$ qN $($t$)\}.$

You can only schedule only one of the queues to serve. Number of packets arrive at the i$-$th queue is given by $\backslash$lambda i$($t$).$ If you schedule i$-$th queue

the number of packets served is given by di$($t$).$ Now express the stateevolution given s$($t$)=$ q$($t$)$ and the action a$($t$),$ here a$($t$)$ is $1$ for the queue

you scheduled, and $0$ for other queues