We are discussing the forward Kolmogorov equation for a special case of Continuous-time Markov
Chain that is the birth death process. For a birth death process, the Q matrix is a tridiagonal
matrix. Therefore, you will have the equations from the forward Kolmogorov equation, you
will have a only two terms in the right hand side for the first equation and you will have
only three terms, the diagonal element and two off-diagonal elements.
Therefore, the first equation one can discuss first, the P dash of i,0 that is nothing,
but the system is not moved from the state zero, moving from the state zero that rate
is lambda 0, therefore not moving minus lambda 0 times the probability and the system can
come from the state one with the rate mu 1. Therefore, mu 1 times P i,1 of t.
For all other equations either the system comes from the previous state with the rate
lambda j minus 1 or it comes from the forward one state with the rate mu j plus 1 or not
moving anywhere. So these are all the all possibilities therefore with these three possibilities
you have a three terms in the right hand side and that is the net rate for any strategy.
So if you solve this equation with this initial condition, Kronecker delta i comma j, you
will have the solution of a P i,j.
Here I am discussing the steady state distribution, the way I have discussed the limiting distribution
that is a limit t tends to infinity, probability of i,j of t exist, then it is called the limiting
distribution and the stationary distribution is nothing but for the DTMC is pi is equal
to pi P, summation of pi i is equal to 1. For the CTMC pi Q is equal to zero and the
summation of pi i is equal to 1. That is going to be the steady state distribution, stationary
distribution.
Now I am discussing the steady state distribution, that is nothing but when t tends to infinity
the birth death process may reach steady state or equilibrium condition. That means the state
probability’s does not depend on time. That is a meaning of a steady state distribution.
As t tends to infinity, whenever we say the birth death process reaches a steady state
or at equilibrium, that state probability does not depend on time.
That means if a steady state solution exist, since the state probability does not depend
on time t, the derivative of the time dependent state probability at time t, that derivative
at t tends to infinity becomes zero. If the steady state solution exists. Since the state
probability’s does not depend on time t as t tends to infinity, I can write as a pi
i is a limit t tends to infinity of pi i of t.
So this is different from the way we discussed earlier that conditional probability P i j
of t. But using P i j of t, one can find out what is pi j of t, pi i of t.
That is nothing but the pi of t that I have given in the first lecture for the CTMC. The
pi i of t that is nothing but what is the probability that the system will be in the
state i at times. That is same as what is the probability that the system will be in
the state i given that it was in the state some k at times zero multiplied by what is
the probability that it was in the state k at times.
That is nothing but summation of k and this is nothing but the transition probability
and this is nothing but the initial probability vector element. So using P ki of t or P ij
of t that is a conditional probability, one can get the unconditional probability. This
is nothing but the distribution of X of t.
So this is the probability mass function, probability mass at state i. So now what I
am defining, whenever the steady state distribution exists, that means it is independent of time
t. Therefore, as t tends to infinity the pi i of t can be written as the pi i and whenever
the steady state solution exists, I can use limit t tends to infinity, the derivative
of a pi i of t that is going to be zero.
Therefore, I am going to use these two to get the steady state probabilities for the
birth death process. Since, as a t tends to infinity, the derivative of pi i of t is equal
to zero, therefore, all the left hand side in the forward Kolmogorov equation that is
going to be zero, the right hand side you will have a as a t tends to infinity, the
pi i of t, that can be written as the pi zero and pi one.
So the way we write the conditional probability for P i j with the Kolmogorov forward equation,
you can write the similar equation for the unconditional probability pi i also. So now
I am putting the left hand side zeros because of this condition, limit t tends to infinity,
the derivatives equal to zero and the right hand side I am using as a t tends to infinity,
this probability is nothing but the pi i.
Therefore, it is going to be minus lambda0 times pi 0 plus mu 1 times pi and all other
equation as a three terms. In this homogeneous equation and you need a one normalising condition.
So from this homogeneous equation, I can get regressively pi i in terms of pi 0. So from
the first equation, I can get a pi 1 in terms of pi 0 and the second equation.
I can get a pi 2 in terms of first pi 1 then I can get a pi 1 in terms of pi 0. Therefore,
regressively I can get pi i in terms of pi 0 for all i greater than or equal to 1.
Now I can use a normalising condition, summation of pi i is equal to 1, therefore, I will get
a pi 0 is equal to 1 divided by summation of this many terms in the product form. Since
we need a steady state probabilities and all the pi i are in terms of pi 0. As long as
the denominator is converges, you will have a pi 0 is greater than zero. So ones the pi
0 is greater than zero, then you will get all the pi i with the summation of pi i is
equal to 1.
So whenever these series converges, then I will have a steady state distribution with
the positive probability and a summation of probability is going to be 1. So this is the
condition for a steady state distribution for a birth death process because we started
with a birth death process forward Kolmogorov equation using these two conditions we have
simplified into this form and use a normalising condition and get the pi 0.
As long as the summation is or the series is converges, then we will have the steady
state. If the series diverges, that means by substituting the values for the lambda
i's and mu i’s and if the series denominator series diverges, then the pi 0 is going to
be zero in turn all the pi i’s are equal to zero therefore the steady state distribution
will not exist if the denominator series diverges.
I am going to give a one simple result, for a irreducible positive recurrent time homogeneous
CTMC, we know that a limiting distribution exist, a stationary distribution exist. Now
I am including the steady state distribution also exist, I have given for a steady state
distribution for the birth rate process, not for the CTMC but here I am giving the result
for the CTMC. All the three distribution exist and all are going to be same.
Whenever the CTMC is a time homogeneous irreducible positive recurrent, all these three distributions
are same and one can evaluate, one can solve this two equation by pi Q is equal to zero
and with the summation of pi is equal to 1, you can get the limiting distribution, stationary
distribution or steady state or equilibrium distribution.
As a special case of birth death process, I am going to discuss these two process in
this lecture.
Whenever, we say the birth death process is a pure birth process, that means all the death
rates are going to be zero, we started with a birth death process with the only lambda
i’s are greater than zero and the mu i’s are going to be zero, then it is going to
be call it as a pure birth process. There is a one special case of pure birth process
with the lambda i’s are going to be constant, that is lambda, that is a Poisson process.
I am going to discuss in the next lecture and in these pure birth process, these lambda
i’s are the function of i. Here all the states are transient states.
Here I am discussing the pure death process. A birth death process is said to be a pure
death process if the birth rates are zero and the death rates are non zero. In particular,
we shall obtain the time dependent probabilities of a pure death process in which the death
rates mu i’s are equal to i times mu.
As I given the example, as a fourth example in the birth death process, this state zero
is a absorbing barrier. Therefore, the state zero is absorbing state and all other states
are going to be transient state. And here the limiting distribution exists and one can
also find time dependent probabilities for this model.
Suppose you start with the assumption, the system at time zero, in the system is in the
state n, at times zero the system in the state n at times zero. With that assumption, I can
frame the equation that is the pi n dash of t is equal to minus n times mu of pi n of
t.
That means the rate in which the system is in the state n that is nothing but not moving
to the state n minus 1 with the rate n minus n times mu. Therefore, the equation for the
state n that is a pi n dash of t that is equal to not moving from the state n therefore minus
that outgoing rate that is n times mu being the state is n therefore pi n of t. I can
use the initial condition pi n of zero is equal to 1, so I will get pi n of t.
For the second equation, I have to go for what is the equation for the state n minus
1. So the pi n minus 1 dash t, that is nothing but either the system come from the state
n or not moving from the state n minus 1. Therefore, system coming from the state n
that is a n mu times the system being the state n minus n minus 1 times mu pi n minus
1 of t. So we will have a two terms in the right hand side coming from the one forward
state or not moving from the same state.
So you will have a two terms for j is equal to 1 to n minus 1. For the last state, that
is the state zero, the system comes from the state 1. Since the state zero is absorbing
states, there is no second term. So it is going to be mu times pi n of t. So you know
pi n of t, use the pi n of t in the equation for n minus 1 and get the pi n minus 1, like
that you find out till pi 1. Use the pi 1 to get the pi 0 of t. Use the recursive way.
So using the recursive way, you will get the pi j of t is equal to n C j combination n
C j and e power minus mu times t power j, this is survival probability of system being
in the state and 1 minus e power minus mu of t n minus j. Suppose the system being in
the state j, that means from the state n this many combinations would have come and the
survival probability is e power minus mu times t and that power.
So this is nothing but the probability p power j and one minus p power n minus j. Therefore,
this pi j follows the binominal distribution with the survival probability e power minus
mu t being in the state j.
So for the pure death process, I have explained the time dependent probabilities of being
in the state j, that is a unconditioned probability. So with this the summary of this lecture is,
I have discussed the limiting, stationary and a steady state distribution, I have introduced
a birth death process. Some important results also discussed.
And at the end, I have discussed the pure birth and pure death process also. In the
next lecture, I am going to explain the important pure birth process that is the Poisson process.
And these are all the reference books. Thanks.