How to study the Stationary Distribution for a reducible Markov chain along with the assumptions
one is aperiodic and the finite state.
Here I am making one more, here I am giving the stationary distribution.
So I am giving the result for a reducible finite Markov chain, Markov chain is as in
the finite state space and it is reducible one.
With the closed communicating class has aperiodic states.
There is a mistake the closed communicating class of state has aperiodic states, aperiodic
the closed communicating class of states as aperiodic then the stationary distribution
exists that is going to be unique also and that is given by the vector V which consist
of two sub vectors V1, 0 vector that you can find out.
And this is nothing but the Ergodic Theorem at the reducible Markov chain with the assumption
finite state space and the states of closed communicating class has aperiodic states.
In that case you get the unique stationary distribution and that unique stationary distribution
has a two sub one that vectors are V1 and vectors of 0.
Before you can get the stationary distribution you can find out what is the n-step transition
probability for the same reducible Markov chain model.
So the P1 is going to be, you have a sub matrix Stochastic sub matrix P1 therefore that is
going to be P1 power n whereas for every n this is going to be a function of n.
R is the sub matrix which is the one step going from the transient state to the closed
communicating class.
Now the R power n is nothing but a function of n that element that sub matrix is corresponding
to the transient state to the communicating class.
Whereas the transient to transient that is going to be a power n.
It is a Q matrix Q matrix is the sub matrix for one step T to T whereas the Q power n
is the element corresponding to the n step transition probability matrix.
So as n tends to infinite the Stochastic sub matrix that power n that will the vector of
V1, V1 is the sub few elements that is corresponding to the stationary state probability for the
state corresponding to the closed communicating class of states.
So e is the vector of the entries one, one, one and so on multiplied by the V1.
And the transient to transient n-Step transition probability as n tends to infinity this will
tends to 0 this is obvious.
Because the states are transient states for a finite n you have a probability Q power
n whereas as n tends to infinite the system will not be in the transient state.
Therefore, the long run proportion of the time the system being in the transient states
that is 0 as n tends to infinite.
Therefore, Q power n tends to 0 or this will tens the to the stationary state probabilities.
Therefore, this stationary distribution vector V consist of few elements of zeros that is
corresponding to a transient states, transient state probabilities in a long run.
And V1 is the steady state probabilities in a longer run it is not steady state its stationary
distribution probability in a longer run for the closed communicating class of states.
So this one can solve by using equation pi P is equal to pi you can get this pi’s that
pi’s is in the notation and here it is Vi, V1.
So now I am making a further assumption the states are going to be the positive recurrent.
So already I made a aperiodic states now I am making one more assumption it is a positive
recurrent.
Once it is a positive recurrent then the limiting probability is limit n tends to infinite that
probability is going to be Vj for the positive recurrent states and for all the transient
states the probabilities are going to be zero.
And since we have a reducible Markov chain with one closed communicating class.
And all other states are transient states this stationary distribution stationary state
probabilities these probabilities are independent of the initial state i.
That means either the system you can start a time zero in the one of the states in closed
communicating class or transient states.
In a longer run, ultimately the system will be in one of the states in the closed communicating
class whether it started initially from the closed communicating class or transient state.
Therefore, this stationary distribution is independent of initial state i and for transient
state you can conclude immediately these probabilities are zeros and for a positive recurrent states
you can make it Vj and you can compute this Vj.
Now I am going to give one simple example in which we have a infinite state.
This is going to be a reducible Markov chain because the states till five not till five
including four and two the system come to the state three there is no arc from three
to four or three to two therefore the states 2,4,5,6 and so on all those states are transient
states whereas the state one and three are going to form a one closed communicating class.
Therefore, this is the reducible Markov chain with the one closed communicating class one
and three and all other states are going to be transient states therefore as n tends to
infinite this probability is are going to be zero for these states 2,4,5 and so on and
these probabilities are independent of the initial state i.
So wherever the i whether i is belonging to the one of the event one of the states in
the closed communicating class of states or the transient state immaterial of that this
is stationary distribution are zeros for the transient state for the closed communicating
class of states you can find out this probability by separately making the Markov chain.
The states one and three you can make it separately and there is a arc from one to three with
the probability half there is self-loop with the probability one by two and there is a
self-loop in the state three with the probability one by three and the arc from three to one
is two by three.
So what do you want to find out this stationary distribution for this two states.
Therefore, you make as stochastic sub matrix with the states one and three.
That is one by two, one by three, two by three and one by three this is also stochastic matrix
you can verify.
Now if you want to find out the stationary distribution for these two states you solve
by pi P1 is equal to pi.
That means pi 1, pi 3 times P1 that is one by two, one by three oh sorry, one by two,
this is one by two, one by two, one by two and this is two by three; one by three that
is equal to pi 1, pi 3.
You take the first equation that is pi 1, half times pi 1one plus two third pi 3 that
is equal to pi one.
So from here you will get pi three is equal to three by four pi 1.
Now we use a pi 1 plus pi 3 is equal to one.
So using this we will get pi 1 is equal to four by seven.
Once you know the pi 1 the pi 3 is going to be three by seven.
So you do not want to find out stationary distribution for the whole model instead of
that you can find out what is the closed communicating class.
And you can solve only a closed communicating class that sub matrix pi P1 is equal to pi
and you will get pi 1 and pi 3 and that is going to be in a longer run that is equal
to four by seven and three by seven and all other states are going to be zero.