Now I am going to move the Stationary Distribution, the stationary distribution also a very important
concept in the Markov chain and as such first I am going to give the definition of a stationary
distribution.
The vector pi is called a stationary distribution of a time homogeneous discrete time Markov
chain, if that vector satisfies the first condition, all these values pi j's are greater
than or equal to 0 for all j and the summation over pi j's that is going to be 1, and the
third condition pi is going to be same as the pi times P, where P is the one step transition
probability matrix. So any vector pi satisfies these three conditions.
Then, that vector is going to be call it as a stationary distribution, this is a nothing
to do with the limiting distribution the one I have discussed earlier, but for an irreducible
aperiodic Markov chain the limiting distribution is same as the stationary distribution that
is also going to be same as the equilibrium or a steady state distribution. All these
three distributions are going to be same for a irreducible aperiodic Markov chain.
But, for in general, all these three things are going to be different, so here I am giving
the definition of a stationary distribution by satisfying these three properties.
Now I am going to give some important results for that, the first result is for an irreducible
aperiodic, positive recurrent Markov chain the stationary distribution exists and it
is unique, the one definition I have given earlier, I have discussed aperiodic irreducible
I have to include the positive recurrent also. Because these three things are important for
an irreducible aperiodic positive recurrent Markov chain all these three distributions
limiting distribution, stationary distribution, steady state or equilibrium distribution,
all three are same.
I have to include the positive recurrent also, so what I am giving in this result then pi
is uniquely determined by solving this equation pi equal to pi P with the summation of a pi's
are going to be 1. So if I solve pi is equal to pi P along with the summation of pi is
equal to 1 that will give a unique pi and that pi is going to be a stationary distribution
for a irreducible, aperiodic, positive recurrent Markov chain, irreducible means all the states
are communicating with all other states.
Aperiodic means the periodicity for a state is 1, the greatest common divisor of a system
coming back to the same state all the possible steps that greatest common divisor is 1. The
positive recurrent means it’s a recurrent state that means with the probability 1, the
system start from one state and coming back to the same state that probability is 1, the
positive recurrent means, the mean recurrence time that is going to be a finite value.
If these three conditions are going to be satisfied by any time homogeneous discrete
time Markov chain. Then the stationary distribution can be computed using pi is equal to pi P
and the summation is equal to 1 that is going to be a unique value.
I am giving the same example - I am giving the same example that is the two state model
with states 0 and 1 with probability is a self-loop 1 - a and self-loop 1 - b and the
system going from the state is 0 to 1 in one step that is a and the system is going from
the state 1 to 0 that probability is b, so I am giving a very simple two state model
and you can solve pi is equal to pi P and summation is equal to 1 and you will get the
probabilities.
And these probabilities are same as the probability you got it in the limiting state probability,
if you solve.
If you solve the two state model with the pi is equal to pi P, you will get the probability
is that pi 0 is going to be b divided by a + b and pi 1 is going to be a divided by a
+ b, and it satisfies the summation of pi is equal to 1 and it also satisfied pi is
equal to pi P, that means in this model it is irreducible aperiodic positive recurrent
model, therefore the limiting distribution is same as the stationary distributions also.
The second example, that is with the infinite state
so here the number of states are going to be a countably infinite, I
can start with the to find out the stationary distributions before that I have to cross
check whether it is going to be irreducible aperiodic positive recurrent Markov chain,
it is irreducible because the way I have given the probabilities. I make the assumption the
probabilities are lies between 0 to 1.
And the probability of lies - the q is also lies between 0 to1, therefore each state is
communicating with each other state, therefore it is going to be a irreducible, the second
one it is it has to be aperiodic, aperiodic means the periodicity for each states, because
the greatest common divisor is going to be 1, because the coming back to the state is
via self-loop are going to the some other state and coming back and their also has a
self-loop.
Therefore, it is going to be all the states are going to be aperiodic, therefore the Markov
chain is aperiodic. The third one positive recurrent, since it is a infinite state model
you cannot get the you cannot come to the conclusion whether these Mu 00 is going to
be a finite quantity unless otherwise substituting the value of p and q.
So what I will do I will make the assumption assume that all states are positive recurrent
then later I will find out what is the condition to be a positive recurrent, so I make the
assumption even I do not want to make the - I do not want to make the assumption for
all the states are going to be positive recurrent I can make the assumption for only 1 state
is going to be positive recurrent and since it is a irreducible Markov Chain and all the
states are going to be of the same type.
Therefore, it will come to the conclusion all the states are going to be a positive
recurrent so I make the assumption one state is going to be positive recurrent therefore
it lands up all the states are going to be a positive recurrent, now once I made a assumption
of all the states are positive recurrent therefore it satisfies all the results of a the first
result that is a irreducible aperiodic positive recurrent Markov chain with the infinite state
space.
Therefore, I can find out the I can come to the conclusion the limiting distribution sorry,
the stationary distribution exists and it is going to be unique and that can be computed
by solving the equation pi is equal to pi P with the summation of pi, over i is equal
to 1, where pi is the vector and P is the one step transition probability matrix, that
one step transition probability matrix can be created using the state transition diagram
which I have given.
So if I take the - if I find out what is the first equation from this vector pi is equal
to pi 0, pi 1, pi 2 and so on, here also this and P is the matrix.
Therefore, I will get the first equation as pi 0 is equal to pi 0 times (1 – p) + pi
1 times q, so this is the first equation of a, from the matrix pi is equal to in the matrix
form pi is equal to pi P, so the first equation is pi 0 is equal to pi naught times 1 minus
p + pi 1 times q, so from this equation I can get pi 1, because I can take this pi 0
this side and I can cancel, so I will get a pi 1 is equal to p divided by q times pi
0.
From the first equation we get the relation pi 1 in terms of pi 0, now I will take a second
equation from pi is equal to pi P, so that will give pi 1 is equal to pi 0 times p +
pi 1 times 1 minus p minus q + pi 2 times q, so this equation have pi 0, pi 1 and pi
2, so what I can do I can write pi 1 in terms of pi 0 and I can simplify this equation if
I simplify, I will get pi 2 is same as p square by q square times pi 0.
Because I am substituting pi 1 in terms of pi 0, in this equation therefore I get pi
2, in terms of pi 0, that is pi 2 is equal to p square by q square times pi 0. Similarly,
if I take the third equation and do the same thing finally, I get pi 3 is equal to p cube
by q cube pi 0, the same way I can go further therefore I am get pi n in terms of pi 0,
for n is equal to 1 2 3 and so on.
So this is the way I can solve this equation pi is equal to pi P that is a homogeneous
equation we have to be very careful with the homogeneous equation, so that trivial solutions
are going to be 0, but we are try to find out the non-trivial solutions therefore we
are using the normalization that is the summation of pi is equal to 1, till now I have not used,
so I have just simplified that pi is equal to pi P and getting pi n in terms of pi 0.
Now I have to use summation of a pi i is equal to 1 starting from i is equal to 0 to infinity
therefore the pi 0 will be out 1 + p by q + p square by q square and so on, that is
equal to 1, therefore the pi 0 is going to be 1 divided by 1 + p by q +p square by q
square and so on, that is pi 0, since it is a infinite terms in the denominator as long
as this is converges you will get a non-zero value for a pi0.
In turn you will get a pi n is equal to p by q power n times pi 0 provided this denominator
is going to be converges, when the denominator is going to be converges in this situation
as long as the p by q is going to be less than 1, if p by q is less than 1 earlier condition
is p is lies between 0 to 1 and q is lies between 0 to 1, now I am making the additional
condition p by q is less than 1 that will ensure the denominator converges therefore
the pi 0 is going to be a non-zero value.
Therefore, the pi n's are going to be p divided by q power n times pi 0, where pi 0 is written
1 divided by 1 + p by q + p by q whole square and so on, so provided p by q is less than
1, if you recall we made the assumption the states are going to be a positive recurrent,
if this p by q is less than 1, then you can conclude the mean recurrence time is going
to be a finite value.
If you make the assumption p by q is less than 1 that will ensure the mean recurrence
time for any state is going to be a finite value therefore all the states are going to
be positive recurrent and then the stationary distribution exists therefore this is the
condition for a positive recurrent state for this model and the stationary distributions
that is going to be pi n is equal to p by q power n times pi 0.
This is nothing but in a longer run what is the probability that the system will be in
the state n that probability is a p by q power n times this pi 0 and pi 0 is given in this
form, and in this example we have taken each state for the p by q is same for all the states
we can go for a in general situation the system going from 0 to 1 could be p 0, system going
from the state 1 to 2 maybe p 1 and so on.
Therefore, it need not all the p i p's need not be same and the q is also not be same
so you can generalized this model and this model is nothing but a one dimensional random
walk and here the 0 is, it's a barrier the system is not going away from the 0 in the
left side therefore 0 is the barrier and this is a one dimensional random walk in which
the system is a keep moving into the different states in subsequent steps.
And there is a possibility the system may be in the same state with the positive probability
of 1 minus (p + q) in this model, in general you can go for the p, p 0, p 1, p 2 and so
on, and similarly, q 1, q 2, q 3 and so on also.