On jump Markov processes. This is just a place-holder at this stage.

Keywords: something or other

Contents

1 Preliminaries

1. “Probability Distributions,” Connexions module m43336,
2. “Introduction to Markov Processes,” Connexionx module m44014,
3. “Continuous Markov Processes,” Connexions module m44258 (in parallel with “Integral of a Markov Process,” module 44376),
4. “Integral of a Markov Process,” Connexions module m44376 (in parallel with “Continuous Markov Processes,” module 44258).

2 Introduction and Summary

A jump Markov process proceeds by the Markov system performing finite, that is, not infinitesimal jumps on moving from $x\left(t\right)$ to $x\left(t+dt\right)$. We will denote the jump by $\Delta x$. Although we will be able to demonstrate that the Markov propagator density function for such a process can be fully characterized by two functions, as was also the case with the continuous Markov processes, on account of this finiteness we will no longer be able to truncate Jump Krammers-Moyal equations don’t trunctate. Kramers-Moyal equations. There is no Fokker-Planck equivalent for the jump Markov process.

Because the Kramers-Moyal equations don’t truncate, there is another machinery, the master equations, which are differential-integral equations, that is preferably used in this context. Although still intractable analytically, the master equations provide us with tools for numerical investigations of jump processes.

The jump Markov process propagator density looks as follow

where

• $a\left(x,t\right)dt$ is the probability that the Markov process at $x$ at time $t$ will jump within the next $dt$, that is, it will jump during the time interval $\left[t,t+dt\right]$ and
• $w\left(\Delta x|\left(x,t\right)\right)d\Delta x$ is the probability that having jumped the process will land between $\Delta x$ and $\Delta x+d\Delta x$ away from $x$, that is between $x+\Delta x$ and $x+\Delta x+d\Delta x$ in absolute terms.

The same propagator density is sometimes expressed as

where $W\left(\Delta x|\left(x,t\right)\right)$ is called the consolidated characterizing function of a jump Markov process. It is the probability that the Markov process at $x$ at $t$ will jump in the next $dt$ by between $\Delta x$ and $\Delta x+d\Delta x$ away from $x$. Functions $a\left(x,t\right)$ and $w\left(\Delta x|\left(x,t\right)\right)$ can be expressed in terms of $W\left(\Delta x|\left(x,t\right)\right)$ as follows

The master equations can be expressed in terms of the consolidated characterizing functions in which case they look as follows (forward and backward):

Unlike continuous Markov processes, the jump processes can be characterized by a yet another function, called the next jump density function

The function is the probability density that a jump Markov process that is at $x$ at $t$ will perform its next jump at about $\Delta t$ past $t$, that is, between $t+\Delta t$ and $t+\Delta t+d\Delta t$ landing between $x+\Delta x$ and $x+\Delta x+d\Delta x$, or, to put it in other words, that the next jump will happen within $\left[\Delta t,\Delta t+d\Delta t\right]$ after $t$ landing the system within $\left[\Delta x,\Delta x+d\Delta x\right]$ away from $x$. This function can be expressed in terms of $a\left(x,t\right)$ and $w\left(x^{\prime }|\left(x,t\right)\right)$ as follows

This will simplify for temporally homogeneous processes to

Completely homogeneous jump Markov processes are particularly interesting, because they’re the simplest possible and so we can say something about them. The two characterizing functions, $a$ and $w$, become

with the next jump density function simplifying to

An important thing to notice is that whereas we could characterize completely homogeneous continuous Markov processes (they are called Wiener processes) by two constants, here we have one constant only and one irreducible function of the jump itself.

It is interesting to consider $w\left(\Delta x\right)$ given by exponential, Gaussian and Cauchy-Lorentz distributions. The systems so defined become quite tractable and, more importantly, applicable to a range of physical phenomena, for example, diffusion and Brownian motion.

In this module we are also going to take another look at quantum mechanics, asking if quantum mechanics can be simulated by jump Markov processes. There was a vigorous discussion about this in the literature.

It all started with a paper by Hardy, Home, Squires and Whitaker, “Realism and the quantum-mechanical two-state oscillator,” published on the 1st of April 1992 in Physical Review A, vol. 45, no. 7, pp. 4267–4270. Nearly two years later Gillespie referred to this paper in his article “Why quantum mechanics cannot be formulated as a Markov process,” published in March 1994 in Physical Review A, vol. 49, no. 3, pp. 1607–1612. A year after that, in May 1995, Garbaczewski and Olkiewicz responded with “Why quantum dynamics can be formulated as a Markov process,” published in Physical Review A, vol. 51, no. 5, pp. 3445–3453, to which Gillespie responded in June 1996 in “Comment on ‘Why quantum dynamics can be formulated as a Markov process’”, published in Physical Review A, vol. 53, no. 6, pp. 4602–4604. Not to be outdone, Gorbaczewski and Olkiewicz published their riposte in August 1996, “Comment on ‘Why quantum mechanics cannot be formulated as a Markov Process’” in Physical Review A, vol. 54, no. 2, pp. 1733–1736. In the same issue of Physical Review A, following the article by Gorbaczewski and Olkiewicz, Gillespie presented his final one-page argument, pp. 1737–1738. But the last word in this exchange belonged to Hardy, Home, Squires and Whitaker, who in their “Comment on ‘Why quantum mechanics cannot be formulated as a Markov process’”, published in October 1997 in Physical Review A, vol. 56, no. 4, pp. 3301–3303, pointed out that they did not claim their process described in the original 1992 paper to be a Markov process. They demonstrated a concrete model that satisfied their requirements (of 1992) and pointed out various problems with Gillespie’s own arguments.

The important moral of the story is that not every stochastic jump process must be a Markov process to begin with. Markovianism is a quite special requirement that may not always apply. Another moral is that some reasoning that Gillespie is so fond of in his papers and in his book (which here we faithfully follow, because it’s an excellent introduction to Markov processes) is not necessarily unquestionable. In particular, the trick of dividing $dt$ into $dt∕n$ does stretch things a bit. Shouldn’t we treat $dt$ as indivisible instead? In effect the Gillespie’s trick leads to smoothness conditions that may be stronger than necessary. Gillespie’s $dt$ is not infinitesimal enough.

To understand this fascinating discussion though we must master the formalism of jump Markov processes first, and so we begin$\dots$

3 The Jump Propagator

A jump process found at $x$ at $t$ will most likely stay at $x$ for a while, then it’ll suddenly jump away from $x$. We can’t normally say how long it’s going to stay at $x$, but we are interested in processes for which a probability exists that the jump will occur within the next $\Delta t$. Not precisely at $\Delta t$ after $t$, mind you, but within $\Delta t$ following $t$. And so

is this probability. The probability function $q$ does not have to exist, but if it does not, then there’s not much else we can say about such processes. So we stick to this assumption. There’s one other thing we can obviously assume, namely that $q\left(0|\left(x,t\right)\right)=0$, meaning that the process is stuck at $x$ at $t$. We will also assume that $q\left(\Delta t|\left(x,t\right)\right)$ is a smooth function of times $t$ and $\Delta t$. These two assumptions, in combination with the assumption that $q$ exists in the first place, go quite far already. Whether we assume too much by doing so will transpire later, when we attempt to apply the theory to known physical processes. But for the time being, the assumptions are purely operational: we need them so that we can develop a tractable theory. It will turn out eventually that the assumption of Markovianism demands a certain degree of smoothness of $q$, although it will also turn out that this is a sufficient, not a necessary, condition.

Now, once the process has jumped, it’ll land somewhere, and where it lands may be described by another probability

This is the probability that once the process has jumped it’ll land at between some finite $\Delta x$ and $\Delta x+d\Delta x$ away from $x$, that is, between $x+\Delta x$ and $x+\Delta x+d\Delta x$. And here we assume also that $w\left(\Delta x|\left(x,t\right)\right)$ is a smooth function of time $t$, again so that the theory is tractable, but it’ll transpire down the road that the assumption of Markovianism demands a certain degree of smoothness of $w$, although this will be a sufficient, not a necessary, condition too.

Both functions $q$ and $wd\Delta x$ must be non-negative, because they are probabilities. Additionally $wd\Delta x$ being probability density must integrate to $1$ over $\Delta x$.

The proof of this equation is rooted in the Markov definition of the underlying process and in the proportional function lemma, that was proved in module m44258:

We prove equation (13) as follows.

In view of our comments on the discussion between Gillespie, Hardy and his collaborators, and Gorbaczewski and Olkiewicz, it is important that we recognize that, in this case, we have assumed the smoothness of $q$ from the beginning. Whether this assumption is more than is strictly required by Markovianism remains to be seen. The reasoning here is quite similar to how we demonstrated in m44258 that

for the continuous Markov processes, wherefrom we deduced that the propagator density function in this case had to be a Gaussian. The assumptions behind equation (13) preclude, for example, a sharp spike in probability $q$ at the end of $dt$, or indeed, within a finite time interval $\Delta t$ thereafter.

Given the meaning of $q$, the meaning of $a\left(x,t\right)dt$ is the probability that the process in $x$ at $t$ will jump away from $x$ in the next $dt$. Furthermore, because $dt$ is infinitesimal, we expect only one jump to occur within $dt$ or none at all. The probability of two jumps occurring would be proportional to ${\left(dt\right)}^2$.

The jump Markov process propagator density function can now be written in the following form

which we read as follows. On the left side we have the probability of the Markov process finding itself removed by $\Delta x$ from $x$ which it occupied at $t$ upon having advanced the time by $dt$. As $\Delta x$ is finite, this is unmistakably a jump propagator. On the right side of the equation, we express the same as the probability that the process in $x$ at $t$ will jump in the next $dt$ times the probability that having jumped it will land by a finite $\Delta x$ away from $x$ or—and this is what the plus stands for—the probability that the process will not jump, in which case it will stay at $x$, which is to say, it will displace by $\Delta x=0$, which is what is signified here by the delta function. So, the process will jump, or it will not.

Dividing both sides by $d\Delta x$ yields

In the following, we’re going to look more closely at two issues. First, we’ll derive a formula for $q\left(\Delta t|\left(x,t\right)\right)$ in terms of $a\left(x,t\right)$. Second, we’re going to see that equation (20) satisfies the Chapman-Komogorov equation to the first order in $dt$, wherefrom we’ll also obtain a more precise idea as to how smooth the functions $a\left(x,t\right)$ and $w\left(\Delta x|\left(x,t\right)\right)$ have to be.

Expression for $q\left(\Delta t|\left(x,t\right)\right)$

$$
We cannot infer from equation (13) that

$$q\left(\Delta t|\left(x,t\right)\right)={\int \; <!--nolimits-->}_0^{\Delta t}a\left(x,t\right)dt$$

(21)

It is more tricky, because we deal here with probabilities and they have their own special rules. Let $\neg q\left(\Delta t|\left(x,t\right)\right)$ be the probability that the system that’s in $x$ at $t$ will not jump away from $x$ in $\Delta t$. This probability, of course, is

$$\neg q\left(\Delta t|\left(x,t\right)\right)=1-q\left(\Delta t|\left(x,t\right)\right).$$

(22)

The probability that the system still will not jump in the next $dt$ is $\neg q\left(\Delta t+dt|\left(x,t\right)\right)$ and it can be decomposed into the product of the probability that the system will not jump in $\Delta t$ and the probability that it will not jump in the following $dt$ either:

$$\begin{array}{cc}\begin{array}{rl}\neg q\left(\Delta t+dt|\left(x,t\right)\right)& =\neg q\left(\Delta t|\left(x,t\right)\right)\times \neg q\left(dt|\left(x,t+\Delta t\right)\right)\\ & =\neg q\left(\Delta t|\left(x,t\right)\right)\left(1-q\left(dt|\left(x,t+\Delta t\right)\right)\right)\\ & =\neg q\left(\Delta t|\left(x,t\right)\right)\left(1-a\left(x,t+\Delta t\right)dt\right),\end{array}& \end{array}$$

(23)

wherefrom

$$\frac{d\neg q\left(\Delta t|\left(x,t\right)\right)}{\neg q\left(\Delta t|\left(x,t\right)\right)}=-a\left(x,t+\Delta t\right)dt.$$

(24)

This then integrates to

$$\neg q\left(\Delta t|\left(x,t\right)\right)=e^{-{\int \; <!--nolimits-->}_0^{\Delta t}a\left(x,t+t^{\prime }\right)d{t}^{\prime }}.$$

(25)

Now we drop the $\neg$, reverting to $q$ itself, which yields

$$q\left(\Delta t|\left(x,t\right)\right)=1-e^{-{\int \; <!--nolimits-->}_0^{\Delta t}a\left(x,t+t^{\prime }\right)d{t}^{\prime }}.$$

(26)

For the very small $\Delta t\approx dt$ we should expect $a\left(x,t\right)dt$ to be small too. The exponential function then will be approximated by

$$1-a\left(x,t\right)dt$$

(27)

and equation (26) becomes (13), our starting point. We see here that the infinitesimal relationship given by equation (13) itself was not enough to reconstruct the correct relationship between $q$ and $a$ for the finite $\Delta t$. Additional information, provided by equation (23), was needed to get it right.

Smoothness of $a$ and $w$

$$
In order to obtain smoothness conditions on $a$ and $w$ we need to inspect what is required of these two functions to ensure that the propagator density function $\Pi \left(\Delta x|dt|\left(x,t\right)\right)$ satisfies the Chapman-Kolmogorov condition

$$\begin{array}{cc}\begin{array}{rl}& \Pi \left(\Delta x|dt|\left(x,t\right)\right)=\\ & {\int \; <!--nolimits-->}_{\Omega \left(\Delta x^{\prime }\right)}\Pi \left(\Delta x-\Delta x^{\prime }|\left(1-\lambda \right)dt|\left(x+\Delta x^{\prime },t+\lambda dt\right)\right)\\ & \Pi \left(\Delta x^{\prime }|\lambda dt|\left(x,t\right)\right)d\Delta x^{\prime }.\end{array}& \end{array}$$

(28)

This condition assumes that the infinitesimal $dt$ can be split into two sub-infinitesimals, $\left[0,\lambda dt\right]$ and $\left[\lambda dt,dt\right]$ and that the process itself must be Markov-divisible on top of these two intervals. This is a weighty assumption, especially given that $\lambda$ is an arbitrary real number in $\left[0,1\right]$. It will result in further continuity conditions down the road.

We begin by substituting equation (20) in equation (28). The first $\Pi$ under the integral becomes

$$\begin{array}{cc}\begin{array}{rl}& a\left(x+\Delta x^{\prime },t+\lambda dt\right)\left(1-\lambda \right)dtw\left(\Delta x-\Delta x^{\prime }|\left(x+\Delta x^{\prime },t+\lambda dt\right)\right)\\ & +\left(1-a\left(x+\Delta x^{\prime },t+\lambda dt\right)\left(1-\lambda \right)dt\right)\delta \left(\Delta x-\Delta x^{\prime }\right).\end{array}& \end{array}$$

(29)

The second $\Pi$ is

$$a\left(x,t\right)\lambda dtw\left(\Delta x^{\prime }|\left(x,t\right)\right)+\left(1-a\left(x,t\right)\lambda dt\right)\delta \left(\Delta x^{\prime }\right).$$

(30)

We have to multiply them now, but we only keep terms linear in $dt$, because $dt$ is infinitesimal after all. The first component in each sum is proportional to $dt$, so we can neglect first times first right away. But the second component of each sum contains a term that is not proportional to $dt$, which is the $\delta$ term itself. In summary, we’ll end up with the first component of the first sum times the delta from the second sum, plus the first component of the second sum times the delta from the first sum, plus a term that’ll contain a product of the two deltas. This last term looks as follows

$$\delta \left(\Delta x-\Delta x^{\prime }\right)\delta \left(\Delta x^{\prime }\right)\left(1-a\left(x+\Delta x^{\prime },t+\lambda dt\right)\left(1-\lambda \right)dt-a\left(x,t\right)\lambda dt\right)$$

(31)

to which we add

$$\begin{array}{cc}\begin{array}{rl}& a\left(x+\Delta x^{\prime },t+\lambda dt\right)\left(1-\lambda \right)dtw\left(\Delta x-\Delta x^{\prime }|\left(x+\Delta x^{\prime },t+\lambda dt\right)\right)\delta \left(\Delta x^{\prime }\right)\\ & +a\left(x,t\right)\lambda dtw\left(\Delta x^{\prime }|\left(x,t\right)\right)\delta \left(\Delta x-\Delta x^{\prime }\right).\end{array}& \end{array}$$

(32)

This is to be integrated over $\Omega \left(\Delta x^{\prime }\right)$. The deltas make the integration easy. $\delta \left(\Delta x^{\prime }\right)$ simply converts $\Delta x^{\prime }$ to zero and $\delta \left(\Delta x-\Delta x^{\prime }\right)$ converts $\Delta x^{\prime }$ to $\Delta x$, which yields

$$\begin{array}{cc}\begin{array}{rl}& a\left(x,t+\lambda dt\right)\left(1-\lambda \right)dtw\left(\Delta x|\left(x,t+\lambda dt\right)\right)+a\left(x,t\right)\lambda dtw\left(\Delta x|\left(x,t\right)\right)\\ & +\left(1-a\left(x,t+\lambda dt\right)\left(1-\lambda \right)dt-a\left(x,t\right)\lambda dt\right)\delta \left(\Delta x\right).\end{array}& \end{array}$$

(33)

We want this to be equal to

$$a\left(x,t\right)dtw\left(\Delta x|\left(x,t\right)\right)+\left(1-a\left(x,t\right)dt\right)\delta \left(\Delta x\right).$$

(34)

The requirement imposes certain conditions on $a$ and $w$. For example, comparing the $\delta$ terms yields

$$a\left(x,t+\lambda dt\right)\left(1-\lambda \right)+a\left(x,t\right)\lambda =a\left(x,t\right).$$

(35)

This must hold in the $dt\to 0$ limit.

If we assume that

$$a\left(x,t+\lambda dt\right)=a\left(x,t\right)+\frac{\partial a\left(x,t\right)}{\partial t}\lambda dt+\mathcal{𝒪}\left({\left(\lambda dt\right)}^2\right)$$

(36)

then

$$\begin{array}{cc}\begin{array}{rl}& a\left(x,t+\lambda dt\right)\left(1-\lambda \right)+a\left(x,t\right)\lambda \\ & =\left(1-\lambda \right)\left(a\left(x,t\right)+\frac{\partial a\left(x,t\right)}{\partial t}\lambda dt\right)+\lambda a\left(x,t\right)+\mathcal{𝒪}\left({\left(\lambda dt\right)}^2\right)\\ & =a\left(x,t\right)+\left(1-\lambda \right)\frac{\partial a\left(x,t\right)}{\partial t}\lambda dt+\mathcal{𝒪}\left({\left(\lambda dt\right)}^2\right)\\ & \underset{dt\to 0}{\overset{}{\to }}a\left(x,t\right).\end{array}& \end{array}$$

(37)

We add a similar assumption regarding $w$, that is,

$$\begin{array}{cc}\begin{array}{rl}& w\left(\Delta x|\left(x,t+\lambda dt\right)\right)=\\ & w\left(\Delta x|\left(x,t\right)\right)+\frac{\partial w\left(\Delta x|\left(x,t\right)\right)}{\partial t}\lambda dt+\mathcal{𝒪}\left({\left(\lambda dt\right)}^2\right)\end{array}& \end{array}$$

(38)

to demonstrate similarly that

$$\begin{array}{cc}\begin{array}{rl}& a\left(x,t+\lambda dt\right)\left(1-\lambda \right)w\left(\Delta x|\left(x,t+\lambda dt\right)\right)+a\left(x,t\right)\lambda w\left(\Delta x|\left(x,t\right)\right)\\ & \underset{dt\to 0}{\overset{}{\to }}a\left(x,t\right)w\left(\Delta x|\left(x,t\right)\right).\end{array}& \end{array}$$

(39)

But by now we can see it even without the explicit computation. If both $a$ and $w$ are to be expanded as per equations (36) and (38), the first term, the one that does not involve $dt$, is simply $a\left(x,t\right)w\left(\Delta x|\left(x,t\right)\right)$. The terms that are proportional to $\lambda$ cancel out. All other terms have $dt$ in them, so they all vanish in the limit $dt\to 0$.

Consequently, we see that assumptions given by equations (36) and (38) are sufficient to ensure that the propagator in the form of equation (20) satisfies the Chapman-Kolmogorov condition given by equation (28). However, we have not proven that they are necessary. There may exist less restrictive conditions that would still ensure the satisfaction of equation (28).

Functions $a\left(x,t\right)$ and $w\left(\Delta x|\left(x,t\right)\right)$ are called the characterizing functions of the jump Markov process. Whereas the continuous Markov process is also described by two functions (see module m44258, where we call them $\mu \left(x,t\right)$, the drift function, and $D\left(x,t\right)$, the diffusion function), the functions depend on two variables $x$ and $t$. Here, for jump processes, we have three variables. The third one is $\Delta x$, the finite jump length. The resulting description is going to be more elaborate. We will also show that in a certain limit jump processes become continuous processes.

Looking at equation (20) we see that the two functions enter the expression for the propagator in a special way, namely through the product of $a$ and $w$ and then through $a$ itself in the second summand, the one proportional to the Dirac delta. Because $w\left(x^{\prime }|\left(x,t\right)\right)$ is the probability density in $x^{\prime }$, it must integrate to 1. In turn, $a\left(x,t\right)$ does not depend on $x^{\prime }$. We can therefore define

for which we’ll find that

and

Function $W\left(x^{\prime }|\left(x,t\right)\right)$ is called the consolidated characterizing function of the jump Markov process and it can be used instead of $a$ and $w$ to encode the jump process propagator density as follows

The expression

is the probability that the system in $x$ at $t$ will jump in the next $dt$ by between $x^{\prime }$ and $x^{\prime }+d{x}^{\prime }$ away from $x$.

Now, we switch to one dimention, $x\to x$ and $x^{\prime }\to x^{\prime }$.

Once we have the jump propagator density function, we can calculate propagator moments, namely

Here we have made use of

to kill the second term, the one with $1-a$. From the above then

Defining the $n^{th}$ moments of $w$ and $W$, the consolidated characterizing function of the jump process, by

and

we can rewrite equation (47) as

For the above definitions to be meaningful, the corresponding integrals must be convergent: the moments ${\Pi }_n$ exist if the corresponding moments of $w$ (or $W$) exist.

4 The Jump Kramers-Moyal Equations

The Kramers-Moyal equations (forward and backward) were derived in module m44014 from the Kolmogorov equations. They allowed us to express the time derivatives of the Markov process probability density function $P\left(\left(x,t\right)|\left(x_0,t_0\right)\right)$ in terms of the propagator density moments, thus providing us with some sort of evolutionary equations, not unlike the Schrödinger equation of quantum mechanics—though, in general, as we’ll see in the case of the jump processes, they may not be explicitly solvable. In the case of the continuous Markov processes, the forward Kramers-Moyal equation turned out to be the familiar Fokker-Planck equation that with some additional non-Markovian assumptions could be converted into the Schrödinger equation—the “non-Markovian” phrase here being important.

The forward Kramers-Moyal equation is

Substituting equation (50) yields

The backward Kramers-Moyal equation is

Substituting equation (50) yields

The Kramers-Moyal equations for the jump Markov processes are partial differential equations of the infinite order in the $x$ or $x_0$ variable. Unlike in the continuous Markov process case, they cannot be truncated in general, so they are nothing but trouble. They can be truncated approximately when the jump Markov process can be considered a continuous one, approximately too, in which case they turn into the Fokker-Planck equations. We are going to demonstrate this in the following.

4.1 The Truncation

To see that the Kramers-Moyal equations do not truncate we make use of the concavity of $x^n$ for $x\ge 0$. Concavity means that $x^n$ curves away from a tangent line for any $x\in \left[0,\infty \right]$. In other words, any tangent provides us with a low estimate for $x^n$.

Let us then consider $x=a>0$. The slope of the tangent at $x=a$ is

The equation for the tangent line is

where $\alpha$ is as above and $\beta$ is given by the requirement that $f\left(a\right)=a^n$, that is