Now we are moving into the second type, in the second type this is a reducible Markov
Chain, but here each close communicating class consists of only one element that is nothing
but the absorbing states but more than one communicating class are possible, therefore
this type is called with one or more absorbing states, here also my interest is to find out
the stationary distribution. The stationary distribution here the interests are of the
different way.
One is the probability of the absorption, the other one is the what is the mean time
before absorption, so for that I'm making further assumption the state space is going
to be finite, so with the time making Canonical form. The Canonical form consists of all the
absorbing states that I label it as A and all the transient states are T. Therefore,
the state space S is a A union capital T.
Therefore, the Canonical form I collect all the absorbing states in the first few rows
and then remaining will be the all the transient states, since the absorbing states P ii is
equal to 1, therefore you will have a identity matrix for the sub matrix of the matrix P
corresponding to A to A. Whereas A to T absorbing states to the transient states that elements
are going to be zero.
So that is the submatrix with the entities zero, whereas T to A will be some matrix capital
R and T to T will be sub matrix Q.
So if you go for what is the n step transition probability since it is identity matrix again
also you will have identity matrix, whereas T to A that is going to be a function of n
whereas T to T, will be a power n that is Q raise to power n, as n tends to infinity
the system won't be in the transient states therefore Q power n will tend to zero submatrix
as n tends to infinity, and these probabilities are going to be zero for all i, j belonging
to T, T is nothing but the set of transient states.
Our interest is here what is the probability of absorption? Because we have a few one or
more absorbing states so if the system starts from some transient state what is the probability
that the system will be absorbed into these absorption states, so for that I am going
to start with the Chapman Kolmogorov equation. That's Chapman Kolmogorov equation for the
(n + 1) th step the system going from the state i to k.
That probability same as what are all the possible the system can go make a one step
from i to j and then j to k in n steps all the possibilities j belonging to S, where
S is the state space. I know either I have a one sorry, either I have a transient states
or all other states are absorbing states. Therefore, if k is going to be the absorbing
state then P kk is equal to 1, that means a one-step transition probability of system
moving from state k to k that is 1.
Therefore, i to k in (n + 1) steps that probability I can split I can make i to k in one step
then forever I will be in the state k plus I would have move to the state i to j, where
j is another transient state it could be same also then j to k in n steps. Now I am defining
what is the meaning of probability of absorption? That I am denoting with the letter a suffix
i, k that is nothing.
But the probability that the system starts in state i it starts in state i eventually
get absorbed in a absorbing state k, so the first letter is the starting state and the
second k is the absorbing state, so this is a probability of absorption starting from
the state i to the absorption state k. Now, I am taking the equation 1 as I make n tends
to infinity in both side the, left hand side will be a of i, k because as n tends to infinity
so this will be a i k, similarly P jk of n that is also a suffix jk.
Therefore, I will have a ik, this side and a jk, so this is sort of recursive equation
so this is in the element form I can go for in the matrix form.
So I can write a ik as a matrix capital A, therefore in the matrix form the previous
this equation for all values of i this equation as in the matrix form A is equal to R matrix,
because this P i to k where i is the transient state and k is the absorbing state so transient
state to the absorption state - transient state to the absorption state that submatrix
is R.
Therefore, in the matrix form capital A is equal to R matrix + Q matrix that is a the
one step transition of system is moving from transient to transient multiplied by A matrix
so I can do the simplification so I get A matrix equals to I minus Q inverse R matrix,
and here this I minus Q inverse that is nothing but the fundamental matrix. So once you are
able to calculate find out the fundamental matrix multiplied by the R matrix that will
give the probability of absorption starting from the transient state and reaching absorption
state.
And these probabilities not independent of initial state that is very important whereas
the previous type of reducible Markov chain that is a independent of initial state whereas
here the probability of absorption is not independent of the initial state i. So this
you can visualize through one example that I am going to present later.
The next result interested in the reducible Markov chain with one or more absorbing states
that is a what is the time to absorption? Basically our interest to get the meantime
to absorption starting from the transient state to a absorption state, that means how
much time on average the system is spending in the transient states before absorption
that is very important.
Because many application has reducible Markov chain in which more than one absorption states
are there with the transient state therefore what is the mean time up to absorption that
means how much time spending in the transient states before the absorption so for that I
am going to define the random variable Ti. The Ti denotes the number of steps including
the starting state i in which the Markov chain reminds in a transient state before entering
a absorption state.
So there is a possibility the system would have been spending at least one step before
absorption or two steps or three steps and so on, therefore that is going to be a random
variable it is a discrete random variable with the possible values are 1, 2, 3 and so
on. Our interest is not only finding out the distribution of Ti, our interest is to find
out what is the mean time up to absorption, from the transient state to a absorbing state.
So this probability can be computed by find out what is the probability of Ti is equal
to n for some n, n can take the value 1, 2, 3 and so on. So that discrete random variable
probability, probability mass function can be computed in this way you find out what
is the probability of Ti is greater than or equal to n minus 1 minus what is the probability
that Ti is greater than or equal to n.
If you find the difference that is same as the probability mass at n, but this is same
as the Ti greater than or equal to n minus 1, that is same as the (n -1 )th step the
system is in the transient state, if Ti is going to be greater than or equal to n minus
1, that means the system spends at least (n – 1) steps in the transient states once
it goes to the absorptions state then it need cannot come back to the transient states.
Therefore, the meaning of Ti greater than or equal to n minus1 that is same as the (n-1)
th step the system in the transient states, so both the events are equivalent, therefore
the probabilities are equal. Similarly, you can argue Ti greater than or equal to n means
at least n steps the system in the transient state before absorption, that is same as in
the n th step the system is in the transient state.
The probability of (n- 1) th step the system is in the transient state that is same as
what is the what are all the possibilities the system would have move from state i to
j in (n – 1) steps, you add all the possibilities j belonging to T, you add all the possibilities
of the transient states that summation will give this probability similarly, for the Xn
belonging to T, this is in the for fixed i, where i is belonging to the transient state.
now I will go for I know that for i, j belonging to T, the n step transition probability is
nothing but the submatrix that is Q power n, if you recall the way we made Canonical
form of a P matrix the T to T, that is a Q matrix, therefore for any nth step that is
going to be Q power n, so this is what I am using for i, j belonging to T the sub matrix
of P power n that is Q power n.
Therefore, for i, j belonging to T, the n step transition of system moving from i to
j that is Q power n, therefore I can substitute here in the above equation so the probability
mass at n that is same as Q power n minus 1 into I minus Q into e vector, once I know
the probability mass function for the discrete random variable Ti then I can find out the
mean, mean is nothing but summation n times the probability mass at n Ti is equal to n.
If I add submission over n, that is going to be the meantime up to absorption that is
going to be do the simple calculation you will get a I minus Q inverse into e vector,
this I minus Q inverse is nothing but the fundamental matrix that means if you find
out the fundamental matrix multiplied by the R sub matrix you will get the probability
of absorption if you multiplied by the e vector you will get the mean time up to the absorption.