I am going to give one more example this has 3 states and this is a state transition diagram
and the values are nothing, but the rates in which the system is moving from one state
to other states.
So that is the difference between the state transition diagram of a DTMC and the CTMC.
So this is a rate in which the system is moving from one state to another state and some arcs
are not there that means there is a no way the system is moving from the state 2 to 3
in a small interval of time.
Whereas all the other possibilities I have given.
So the corresponding Q matrix it is a three cross three matrix and you can make out all
the row sum are going to be 0 and the diagonal elements are minus of some of other values
the same rows and other than the diagonal elements the values are greater than or equal
to 0.
My interest is to find out the time dependent solution for this example also.
I can make a forward Kolmogorov equation P dash of t is equal to P of t times Q it is
a 3 cross 3 matrix therefore I will have a 3 equations and I have one equation I can
have summation of probabilities equal to 1 and I can start with the initial condition
the system being in the state 1 at time 0 the probability is 1, I can start with that
and I can solve those three equations with the initial condition and I can get the solution
that is a one way.
Since it is a finite state CTMC there are many ways to get the time dependent solution
basically you have to solve the system of different differential equations with the
initial conditions.
Here I am using the Eigen value method that means find the Eigen values for the Q matrix.
Therefore, use the Eigen value and Eigen vector concept and you get the P11 of t with the
unknown k1, k2, k3 and to find the unknown to k1, k2, k3 use the initial condition.
Here I am using the initial condition as well as the Q matrix values the q 11 that means
the element corresponding to the 1, 1 that is nothing but the P dash of 1, 1 of 0.
Similarly, if I go for Q square matrix and q11 of 2.
The element in the 1, 1 in the Q square matrix that is nothing but P double dash of 1, 1
of 0.
Therefore now I can use these three initial conditions to get the unknown values k1, k2
and k3.
So once I know the k1, k2, k3, I can substitute that for the P11 of t is equal to this much.
Similarly, I can go for finding the P1, 2 of t and P1, 3 of t.
I do not want P1, 3 in the same way because once I know the P1, 1 of t and P1, 2 of t.
So P 1, 3 of t is nothing but 1 minus of that those two probabilities because there is summation
of probabilities is equal to 1.
So this is the other way of getting the time dependent solution the transition probability
of system being in the state j given that it was in the state I at time 0.
Suppose the CTMC has the finite state space then I can use the exponential matrix also
to get the time dependent solution that what I have given this way.
So start with the forward equation.
Therefore, the solution is going to be P of t is equal to P of 0 e power Qt.
P of t is a matrix P of 0 is the matrix e power Qt that is also again going to be a
matrix exponential matrix therefore I am writing it e power Qt is nothing but Q is the matrix
and t is the real value.
So if greater than or equal to 0 therefore e power Qt is going to be I matrix.
I matrix is nothing but the identical matrix of order whatever the state number plus the
summation n is equal to 1 to infinity of Q power n times t power n divided by n factorial.
So the whole thing is going to be exponential matrix and using that you can get the P of
t.
I am not going detailed for how to compute this e power Qt and so on, but whenever you
have CTMC the finite space through this method also one can get the time dependent solution.
So with this I have completed the examples for the CTMC to find out the time dependent
or transient probabilities.
Now, I am moving into the limiting distribution the way we discuss the limiting distribution
for the DTMC the same concept can be used for the CTMC also.
The change is instead of one step transition probability matrix here we have to use the
Infinitesimal Generator Matrix in a different way.
So I am first giving the Ergodic Theorem, whenever the CTMC is irreducible that means
all the states are communicating with all other states.
Since all the states are communicating with all other states.
So if one is of the particular type it is a positive recurrent then all the other states
are going to be a positive recurrent.
If one is going to be a null recurrent then all the other states also going to be a null
recurrent.
So here I am making the assumption the CTMC is irreducible as well as all the states are
positive recurrent then the limiting distribution always exist.
Suppose, it is independent of the initial state, it need not be a independent of initial
states suppose the same thing is independent of initial state then I can write that limiting
probability is Pij of t.
Since it is independent of i.
I can write it as the pi j.
Then I can form a vector and since it is a limiting distribution it is a probability
distribution.
Therefore, the probabilities are these probabilities are always greater than or equal to 0.
And the summation of probability is going to be 1.
It would not be less than 1 that is the Ergodic theorem says.
Whenever you have a irreducible CTMC with all the states are positive recurrent then
as t tends to infinity the system has the distribution limiting distribution.
If it is independent of initial states, then you can label with the pi j as the probabilities.
And this probability distribution satisfies it is a probability mass function therefore
it satisfies the probability mass function conditions.
That means whenever you have a dynamical system in which it is irreducible model and all the
states are positive recurrent that means the mean recurrence time is going to be finite
value.
Then that system is called it as a Ergodic system or the Ergodic concept can be used
therefore as t tends to infinity you can get the limiting distribution.
If it is independent of initial state means, whatever be the state, you are going to do
it for the discrete end stimulation for the dynamical system that is for a Ergodic system
then the initial condition state does not matter to get the limiting distribution.
Later we are going to give some few examples how to find out the limiting distribution.
I am explaining the stationary distribution also.
The stationary distribution the way I have discuss the DTMC the CTMC also same.
So I have a vector if the vector satisfies these three conditions; probabilities therefore
greater than or equal to zero, summation is equal to 1 and you should be able to solve
the solution and get the pi.
It is a homogenous situation.
So you need a second condition to have the nonnzero probabilities.
So if you solve pi Q equal to 0 along with the summation of pi j is equal to 1 and if
this pi j exist then the CTMC has the stationary distribution.
The similar way I have discussed the stationary distribution for the DTMC model also instead
of pi Q is equal to 0 we had a pi P is equal to pi.
So if any vector satisfies that pi P is equal to pi and summation of pi i is equal to 1
and all the pi i are greater than or equal to 0.
Then that is going to be a stationary distribution for DTMC.
The same way if pi Q is equal to 0 and summation of pi j is equal to 1, pi j are greater than
or equal to 0.
If this is satisfied by any vector, then that is going to be the stationary distribution
for a Time Homogenous CTMC.
Every time we are discussing the default CTMC that is the time homogenous CTMC.
The main result for the stationary distribution whenever you have an irreducible positive
recurrent CTMC the stationary distribution exists and that is going to be unique.
Whenever the CTMC is a positive recurrent as well as irreducible there is no need of
periodicity in the CTMC whereas the same stationary distribution for the DTMC we have included
one more condition that is aperiodic, but for the CTMC there is no periodicity for the
state.
Therefore, as long as the system is irreducible and the positive recurrent then the stationary
distribution exist and it is unique and by solving these equations you can get the unique
stationary distribution.