This is Stochastic Processes, Module 8: Renewal Processes, Lecture 2: Generalized Renewal
Processes and Renewal Limit Theorems.
In the Lecture 1, we have discussed the renewal processes definition and its properties.
Followed by renewal process definition, we have discussed the renewal function, and then
we have discussed the renewal equation, and also we have seen few examples in the Lecture
1.
In the Lecture 2, we are planning to discuss the reward renewal process.
As a special case, we are going to discuss the Markov reward models.
Then we are going to discuss two different renewal processes, that is alternative renewal
process and delayed renewal process.
With this we are completing the generalized renewal processes.
The second half, we are going to discuss the Central Limit Theorem, Long-run properties
and three important renewal limit theorems.
What is a reward renewal process?
Let X1 be the time to the first renewal and let Xn be the time between (n-1)th renewal
and nth renewal.
The same definition which we have used for the renewal process and also you assume that
Xi's are i.i.d. random variables with the CDF capital F of x.
Since Xi's are the inter-arrival times, the mean will be a positive.
Mean exists and it will be positive.
Let Rn be the nth reward at time t of the nth renewal.
For each renewal, you are attaching the reward at the time of the renewal.
Usually, Rn may depend on Xn.
Now we are defining a new random variable R(t) function of t, that is nothing but the
summation of Rn's where n is running from 1 to N(t) where N(t) is a renewal process.
The collection of R(t) for t greater than or equal to zero will be called it as a Reward
Renewal Process.
That R(t) is nothing but a reward earned by time t.
So from the renewal process, we are attaching the reward for each renewal and by defining
R(t) is equal to summation of n is equal to 1 to N(t) Rn, that will be call it as a Reward
Renewal Process.
Note that unlike the Xn, each Rn may take a negative values as well as the positive
values.
The Xi's are nothing but the inter-arrival time of the renewals whereas the rewards may
take negative values as well as positive values.
See the sample path of a reward renewal process.
So the x-axis is the time.
The y-axis is N(t).
And these are all the time points in which the renewals takes place.
The first renewal takes place, second renewal and so on.
So this is the inter-arrival time and attaching the reward Ri to the each Xn and these Ri's
may be negative or positive, then the collection of Ri's with this form Rt is equal to summation
n is equal to 1 to N(t) Rn will be the reward renewal process.
So it is very difficult to show the sample path of R(t).
So here we are showing the sample path of renewal process with the rewards attached
with the each Xi's.
Now we are moving into the simple example of reward renewal process.
Consider an age replacement model.
In this model, a component that is used continuously with replacement -- replacements.
Let X be the lifetime of the component, which is random with the distribution function F.
The component is replaced by a new one upon failure or at a fixed time period capital
T, whichever comes first.
The replacement policy is called an age replacement because the component is replaced by a new
one upon failure or at a fixed time period T.
The cost of a new component is c1 and the additional cost is incurred by a failure is
c2.
Our interest is to find the long-run average cost.
Let N(t) be the number of replacements of components by time.
N(t) is a renewal process.
Let R(t) be the amount of cost incurred by time t.
So this is a reward renewal process.
So the long-run average cost can be expressed as the ratio of expectation of R by expectation
of X that we are going to conclude later.
Now we are using the long-run average cost is the expectation of R divided by expectation
of X. Expectation of R is nothing but expectation
of cost during a cycle and expectation of X is nothing but expectation of the cycle
length.
Now one can find the expectation of cycle length, that is nothing but the expectation
of the cycle length is a random variable, which is nothing but the minimum of X or T.
Here the component is replaces either upon a failure or at each capital T. Time between
replacement is called a cycle.
That is nothing but the integration 0 to t (1 - F(x))dx.
Given that, the reward or cost during the cycle R, therefore, R will be c1 if X is greater
than T, if it fails after T. If it fails before T, then there is an additional cost c2.
Therefore, the cost will be c1 + c2 if failure occurs before the fixed time T.
Hence, the expected value that is the expected cost is either c1 with the probability X is
greater than T or c2 plus -- c1 + c2 with the probability X is less than or equal to
T. So that is nothing but c1 + c2 times F(T).
Hence, the long-run average cost is nothing but expectation of R divided by expectation
of X.
That is nothing but expectation of cost during a cycle divided by expectation of cycle length.
Substitute the values.
You will get the long-run average cost.
Now we are moving into the special case of a reward renewal process that is a Markov
Reward Model.
A Markov Reward Model is a labeled continuous-time Markov chain augmented with a state reward
and impulse reward structures.
The state reward structure is a function r that assigns to each state a reward rs such
that if t time units are spent in the state s, a reward of rs times t is acquired.
The rewards that are defined in the state reward structure can be interpreted in various
ways.
They can be recorded as the gain or benefit acquired by staying in some state and they
can also be regarded as the cost spent by staying in some state, and this type of Markov
reward model is called a rate-based Markov reward model.
The impulse reward structure, on the other hand, is a function of -- function t that
assigns to each transition from s to s' where s and s' are belonging to the state space
S, and the spending time in transition of s to s' is positive, and the reward t of s
to s' such that if the transition from s to s' occurs, a reward of t(s, s') is acquired.
Similar to the state reward structure, the impulse reward structure can be interpreted
in various ways.
An impulse reward can be considered as the cost of taking a transition or the gain that
is acquired by taking the transition.
Markov reward models are commonly used for the performance, dependability and performability
analysis of computer and communication systems.
In general, the reward rate is assigned on the basis of desired measures.
Let Z(t) is nothing but rX(t) be the instantaneous reward rate of the Markov reward model at
time t.
Then the expected instantaneous reward rate at time t is given by: expectation of Z(t),
that is nothing but summation of ri Pi(t).
The expected reward rate in steady state is nothing but expectation of Z is summation
of ri pi.
Suppose in the perform -- availability model, our interest is to find out the availability
of the system, then one can assign the rewards for the up states are 1 and rewards for the
down states are zeros.
Then the expected reward rate will be the availability for the system by making a summation
of ri's and probability of being in those states.
The different types of measures namely steady-state measures, transient measures, cumulative measures
and performability measures are supported by Markov reward models.
For instant, steady-state measures are computed from the steady-state behavior of the Markov
chain.
So for the availability model, if you know the steady-state behavior of the Markov chain
such as a steady-state probability of being in this system, then by assigning the rewards
1 to the up states and 0s to the down states, one can get the steady-state availability
of the system.
Similarly, reverting the rewards, you can get the steady-state unavailability of the
system also.
Under the assumption that the steady-state distribution of X(t), one can find the steady-state
measures by multiplying ri's with the pi's where i is -- i belonging to the state space
S. Similarly, the cumulative measures denoted
by Y(t) express the overall gain that is received from a system over some finite time interval.
When transient measures are integrated over the time interval 0 to t, then Y(t) can be
given as integration 0 to t rX(s) ds.