WEBVTT
Kind: captions
Language: en
00:00:22.020 --> 00:00:28.340
This stochastic process in this, we are going
to discuss the Module 4 Discrete time Markov
00:00:28.340 --> 00:00:34.940
Chain and this is a lecture 1, in this lecture,
I am going to discuss the introduction about
00:00:34.940 --> 00:00:41.160
the discrete time Markov chain, then followed
by the definition and the important one concept
00:00:41.160 --> 00:00:46.460
called one step transition probability matrix.
00:00:46.460 --> 00:00:53.370
So this lecture I am going to cover the Introduction,
Definition, Transition probability matrix
00:00:53.370 --> 00:01:07.220
and few simple examples also, consider a random
experiment of tossing a coin infinitely many
00:01:07.220 --> 00:01:14.000
times each trial there are two possible outcomes
namely head or tail.
00:01:14.000 --> 00:01:23.729
Assume that the probability of head, that
probability you assume that is p and the probability
00:01:23.729 --> 00:01:41.100
of tail occurring in each trial that you assume
it as 1 - p, you assume that the p lies between
00:01:41.100 --> 00:01:54.880
0 to 1, denote for the nth trail, because
you are tossing a coin infinitely many times
00:01:54.880 --> 00:02:04.390
for the nth trial, you denote the a random
variable Xn is the random variable whose values
00:02:04.390 --> 00:02:06.920
are 0 or 1 with the probability.
00:02:06.920 --> 00:02:13.700
The probability of Xn takes a value 0 that
is same as in the nth trial you are getting
00:02:13.700 --> 00:02:21.420
the tail that probabilities 1 - p and the
probability of Xn takes a value 1 that probability
00:02:21.420 --> 00:02:27.760
is make it as p for the head appears and already
you assume that and the probability is lies
00:02:27.760 --> 00:02:37.010
between 0 to 1, thus you have a sequence of
random variable X1, X2 and so on and this
00:02:37.010 --> 00:02:39.920
will form a stochastic process.
00:02:39.920 --> 00:02:57.280
And assume that all the Xi's
are mutually independent
00:02:57.280 --> 00:03:06.340
random variable, so this is a random experiment
in which we are tossing a coin infinitely
00:03:06.340 --> 00:03:14.160
many times and for any nth trial, you define
the random variable Xn with the probability
00:03:14.160 --> 00:03:20.180
it takes a value 0 the probability 1 - p and
takes a value 1 with the probability p and
00:03:20.180 --> 00:03:27.130
that is equivalent of appearing head at the
probability p and occurring the tail with
00:03:27.130 --> 00:03:30.630
the probability 1 – p.
00:03:30.630 --> 00:03:41.410
Now I am going to define another random variable
that is a partial sum of first n random variables.
00:03:41.410 --> 00:03:50.580
An n Xi's, so the Sn will be sum of a first
n random variables therefore the sum Sn gives
00:03:50.580 --> 00:03:59.920
the number of heads appears in the first n
trials, it can be observed that S(n + 1) is
00:03:59.920 --> 00:04:11.460
same as Sn + X(n + 1), since Sn is the partial
sum of a first n trials outcome so the S(n
00:04:11.460 --> 00:04:19.359
+ 1) is nothing but Sn + X(n + 1).
00:04:19.359 --> 00:04:29.630
you can also observed that since Sn is the
sum of a first n random variables and Sn +
00:04:29.630 --> 00:04:34.110
X(n+1) and also all the Xi's are mutually
independent variables.
00:04:34.110 --> 00:04:43.529
Sn is independent with X(n+1), that means
here the S(n + 1) th random variable is a
00:04:43.529 --> 00:04:51.479
the combination of a two independent random
variables whereas the Sn is the till nth trial
00:04:51.479 --> 00:04:58.520
how many heads you appear plus whether it
is a head or tail accordingly these values
00:04:58.520 --> 00:05:06.009
is going to be 0 or 1, therefore if you see
the sample path of S(n + 1), it will be incremented
00:05:06.009 --> 00:05:13.740
by 1 if X(n + 1) takes a value 1 or it would
have been the same value earlier if this X(n
00:05:13.740 --> 00:05:16.819
+ 1) takes a value 0.
00:05:16.819 --> 00:05:27.699
And also you can observe that S(n + 1) is
a depends on Sn and only on it, it is not
00:05:27.699 --> 00:05:35.870
a depends on S(n – 1) or S(n – 2) so on,
because it is accumulated the number of trials
00:05:35.870 --> 00:05:46.139
values over the n, therefore S(n + 1) is depends
on Sn and only on it, the Sn for different
00:05:46.139 --> 00:05:56.740
values of n this will form a stochastic process
and now you can come to the conclusion the
00:05:56.740 --> 00:06:05.539
probability of this is a stochastic process.
00:06:05.539 --> 00:06:18.589
The probability of S(n + 1) suppose this values
is k + 1 given that Sn was k that means the
00:06:18.589 --> 00:06:24.979
S(n + 1) value would have been 1, therefore
the appearance of a the head appears in the
00:06:24.979 --> 00:06:32.449
(n + 1) th trial and probability is going
to p. similarly you can make out suppose S(n
00:06:32.449 --> 00:06:43.270
+ 1) value will be k such that Sn is also
k then that is possible with a (n + 1) th
00:06:43.270 --> 00:06:54.870
trial you got the tail, therefore that probability
is 1 – p, this is satisfied for all n.
00:06:54.870 --> 00:07:00.650
So you can make out this is satisfied for
all n is greater than or equal to, even I
00:07:00.650 --> 00:07:05.580
can go for n is greater than or equal to 1
not only these.
00:07:05.580 --> 00:07:12.440
Similarly, I can come to the conclusion the
probability of (Sn + 1 )is equal to k given
00:07:12.440 --> 00:07:26.349
that S1 was i1, S2 was i2 and so on Sn was
k that is also can be proved the probability
00:07:26.349 --> 00:07:36.210
of S(n + 1) is equal to k, given that Sn is
equal to k that is same as what is the probability
00:07:36.210 --> 00:07:44.169
that value was the same k in the subsequent
trials that is possible of appearing a tail
00:07:44.169 --> 00:07:45.999
in the (n + 1) th trial.
00:07:45.999 --> 00:07:50.379
Therefore, that the appearance of the tail
in the (n + 1) th trial the probability is
00:07:50.379 --> 00:08:04.029
1 - p or I can use the notation q, that means
the probability of (n + 1) th trial that distribution
00:08:04.029 --> 00:08:10.809
given that I know the value till the nth trial
that is same as the distribution of (n + 1)
00:08:10.809 --> 00:08:19.229
th trial, given with the only the distribution
nth distributions not the earlier distributions
00:08:19.229 --> 00:08:24.159
and this property is called memoryless property.
00:08:24.159 --> 00:08:35.830
The stochastic process the Sn satisfies the
memoryless property or the other word called
00:08:35.830 --> 00:08:46.490
Markov property, the distribution of (n +
1) given that the distribution of first random
00:08:46.490 --> 00:08:50.680
variable, second random variable, the nth
random variable that is same as the conditional
00:08:50.680 --> 00:08:57.190
distribution of (n + 1) th random variable
given that with the nth random variable only
00:08:57.190 --> 00:09:02.120
and this property is called a memoryless or
Markov property.
00:09:02.120 --> 00:09:10.930
The stochastic process the Sn satisfying the
Markov property or memoryless property is
00:09:10.930 --> 00:09:23.410
called Markov process, the stochastic process
satisfying the memoryless property or Markov
00:09:23.410 --> 00:09:32.440
property is called Markov process, in this
example the stochastic process Sn is the discrete
00:09:32.440 --> 00:09:44.160
time, discrete state stochastic process.
00:09:44.160 --> 00:09:56.190
Now I can give based on the state space and
the parameter space I can classify the Markov
00:09:56.190 --> 00:10:03.710
process or I can give the name of the Markov
process in an easy way based on the state
00:10:03.710 --> 00:10:08.060
space as well as the parameter space.
00:10:08.060 --> 00:10:16.460
So when the state space S, the S is the state
space this is nothing but the collection of
00:10:16.460 --> 00:10:27.100
all possible values of the stochastic process,
if this is of the discrete type that means
00:10:27.100 --> 00:10:32.640
the collection of a elements in the state
space S is going to be a finite or countably
00:10:32.640 --> 00:10:36.850
infinite then we say the states spaces of
the discrete type.
00:10:36.850 --> 00:10:42.710
So whenever the stochastic process satisfying
the Markov property then the stochastic process
00:10:42.710 --> 00:10:48.300
is called the Markov process or you can say
whenever the state space is a discrete then
00:10:48.300 --> 00:11:01.650
you can say the corresponding stochastic process
we can call it as a Markov chain, whenever
00:11:01.650 --> 00:11:05.720
the state spaces are discrete.
00:11:05.720 --> 00:11:15.950
Now based on the parameter space T, parameter
space is nothing but the possible values of
00:11:15.950 --> 00:11:22.850
T whether it is going to be a finite or countably
infinite then it is going to be a discrete
00:11:22.850 --> 00:11:28.280
parameter space or discrete time or it is
going to be uncountably many values then it
00:11:28.280 --> 00:11:38.260
is going to be called it as a continuous type,
so whenever T is going to be a discrete type
00:11:38.260 --> 00:11:44.440
then the Markov chain is going to be called
it as discrete time Markov chain.
00:11:44.440 --> 00:11:50.160
Whenever the parameters which is going to
be of the continuous type that means the possible
00:11:50.160 --> 00:12:00.550
values of T is going to be uncountably many
then we say continuous time Markov chain,
00:12:00.550 --> 00:12:07.380
so in this example the Sn, the possible values
of Sn is also going to the state space is
00:12:07.380 --> 00:12:12.840
going to be discrete type and the parameter
space is also going to be discrete type therefore
00:12:12.840 --> 00:12:18.530
the given example the sun is going to be the
discrete time Markov chain.
00:12:18.530 --> 00:12:25.930
So in this module, we are going to study the
discrete time Markov chain then next module,
00:12:25.930 --> 00:12:33.970
module five we are going to discuss the continuous
time Markov chain, so in general whenever
00:12:33.970 --> 00:12:40.450
the stochastic process satisfying the Markov
property it will be called Markov process.
00:12:40.450 --> 00:12:45.820
So based on the state space the Markov process
called as Markov chain and the based on the
00:12:45.820 --> 00:12:50.491
parameter space it is called discrete time
Markov chain or continuous time Markov chain
00:12:50.491 --> 00:12:53.980
accordingly, discrete type or continuous type.