This stochastic process in this, we are going to discuss the Module 4 Discrete time Markov
Chain and this is a lecture 1, in this lecture, I am going to discuss the introduction about
the discrete time Markov chain, then followed by the definition and the important one concept
called one step transition probability matrix.
So this lecture I am going to cover the Introduction, Definition, Transition probability matrix
and few simple examples also, consider a random experiment of tossing a coin infinitely many
times each trial there are two possible outcomes namely head or tail.
Assume that the probability of head, that probability you assume that is p and the probability
of tail occurring in each trial that you assume it as 1 - p, you assume that the p lies between
0 to 1, denote for the nth trail, because you are tossing a coin infinitely many times
for the nth trial, you denote the a random variable Xn is the random variable whose values
are 0 or 1 with the probability.
The probability of Xn takes a value 0 that is same as in the nth trial you are getting
the tail that probabilities 1 - p and the probability of Xn takes a value 1 that probability
is make it as p for the head appears and already you assume that and the probability is lies
between 0 to 1, thus you have a sequence of random variable X1, X2 and so on and this
will form a stochastic process.
And assume that all the Xi's are mutually independent
random variable, so this is a random experiment in which we are tossing a coin infinitely
many times and for any nth trial, you define the random variable Xn with the probability
it takes a value 0 the probability 1 - p and takes a value 1 with the probability p and
that is equivalent of appearing head at the probability p and occurring the tail with
the probability 1 – p.
Now I am going to define another random variable that is a partial sum of first n random variables.
An n Xi's, so the Sn will be sum of a first n random variables therefore the sum Sn gives
the number of heads appears in the first n trials, it can be observed that S(n + 1) is
same as Sn + X(n + 1), since Sn is the partial sum of a first n trials outcome so the S(n
+ 1) is nothing but Sn + X(n + 1).
you can also observed that since Sn is the sum of a first n random variables and Sn +
X(n+1) and also all the Xi's are mutually independent variables.
Sn is independent with X(n+1), that means here the S(n + 1) th random variable is a
the combination of a two independent random variables whereas the Sn is the till nth trial
how many heads you appear plus whether it is a head or tail accordingly these values
is going to be 0 or 1, therefore if you see the sample path of S(n + 1), it will be incremented
by 1 if X(n + 1) takes a value 1 or it would have been the same value earlier if this X(n
+ 1) takes a value 0.
And also you can observe that S(n + 1) is a depends on Sn and only on it, it is not
a depends on S(n – 1) or S(n – 2) so on, because it is accumulated the number of trials
values over the n, therefore S(n + 1) is depends on Sn and only on it, the Sn for different
values of n this will form a stochastic process and now you can come to the conclusion the
probability of this is a stochastic process.
The probability of S(n + 1) suppose this values is k + 1 given that Sn was k that means the
S(n + 1) value would have been 1, therefore the appearance of a the head appears in the
(n + 1) th trial and probability is going to p. similarly you can make out suppose S(n
+ 1) value will be k such that Sn is also k then that is possible with a (n + 1) th
trial you got the tail, therefore that probability is 1 – p, this is satisfied for all n.
So you can make out this is satisfied for all n is greater than or equal to, even I
can go for n is greater than or equal to 1 not only these.
Similarly, I can come to the conclusion the probability of (Sn + 1 )is equal to k given
that S1 was i1, S2 was i2 and so on Sn was k that is also can be proved the probability
of S(n + 1) is equal to k, given that Sn is equal to k that is same as what is the probability
that value was the same k in the subsequent trials that is possible of appearing a tail
in the (n + 1) th trial.
Therefore, that the appearance of the tail in the (n + 1) th trial the probability is
1 - p or I can use the notation q, that means the probability of (n + 1) th trial that distribution
given that I know the value till the nth trial that is same as the distribution of (n + 1)
th trial, given with the only the distribution nth distributions not the earlier distributions
and this property is called memoryless property.
The stochastic process the Sn satisfies the memoryless property or the other word called
Markov property, the distribution of (n + 1) given that the distribution of first random
variable, second random variable, the nth random variable that is same as the conditional
distribution of (n + 1) th random variable given that with the nth random variable only
and this property is called a memoryless or Markov property.
The stochastic process the Sn satisfying the Markov property or memoryless property is
called Markov process, the stochastic process satisfying the memoryless property or Markov
property is called Markov process, in this example the stochastic process Sn is the discrete
time, discrete state stochastic process.
Now I can give based on the state space and the parameter space I can classify the Markov
process or I can give the name of the Markov process in an easy way based on the state
space as well as the parameter space.
So when the state space S, the S is the state space this is nothing but the collection of
all possible values of the stochastic process, if this is of the discrete type that means
the collection of a elements in the state space S is going to be a finite or countably
infinite then we say the states spaces of the discrete type.
So whenever the stochastic process satisfying the Markov property then the stochastic process
is called the Markov process or you can say whenever the state space is a discrete then
you can say the corresponding stochastic process we can call it as a Markov chain, whenever
the state spaces are discrete.
Now based on the parameter space T, parameter space is nothing but the possible values of
T whether it is going to be a finite or countably infinite then it is going to be a discrete
parameter space or discrete time or it is going to be uncountably many values then it
is going to be called it as a continuous type, so whenever T is going to be a discrete type
then the Markov chain is going to be called it as discrete time Markov chain.
Whenever the parameters which is going to be of the continuous type that means the possible
values of T is going to be uncountably many then we say continuous time Markov chain,
so in this example the Sn, the possible values of Sn is also going to the state space is
going to be discrete type and the parameter space is also going to be discrete type therefore
the given example the sun is going to be the discrete time Markov chain.
So in this module, we are going to study the discrete time Markov chain then next module,
module five we are going to discuss the continuous time Markov chain, so in general whenever
the stochastic process satisfying the Markov property it will be called Markov process.
So based on the state space the Markov process called as Markov chain and the based on the
parameter space it is called discrete time Markov chain or continuous time Markov chain
accordingly, discrete type or continuous type.