Transform Methods

Three basic transforms We define each of three transforms, determine some key properties, and use them to study various probability distributions associated with random variables. In the section on integral transforms, we show their relationship to well known integral transforms. These have been studied extensively and used in many other applications, which makes it possible to utilize the considerable literature on these transforms. Definition. The moment generating functionM_X for random variable X (i.e., for its distribution) is the function M X ( s ) = E [ e s X ] ( s is a real or complex parameter) The characteristic functionφ_X for random variable X is φ X ( u ) = E [ e i u X ] ( i 2 = - 1 , u is a real parameter) The generating functiongX(s) for a nonnegative, integer-valued random variable X is g X ( s ) = E [ s X ] = ∑ k s k P ( X = k ) The generating function E[sX] has meaning for more general random variables, but its usefulness is greatest for nonnegative, integer-valued variables, and we limit our consideration to that case. The defining expressions display similarities which show useful relationships. We note two which are particularly useful. M X ( s ) = E [ e s X ] = E [ ( e s ) X ] = g X ( e s ) and φ X ( u ) = E [ e i u X ] = M X ( i u ) Because of the latter relationship, we ordinarily use the moment generating function instead of the characteristic function to avoid writing the complex unit i. When desirable, we convert easily by the change of variable. The integral transform character of these entities implies that there is essentially a one-to-one relationship between the transform and the distribution. Moments The name and some of the importance of the moment generating function arise from the fact that the derivatives of M_X evaluateed at s=0 are the moments about the origin. Specifically M X ( k ) ( 0 ) = E [ X k ] , provided the k th moment exists Since expectation is an integral and because of the regularity of the integrand, we may differentiate inside the integral with respect to the parameter. M X ' ( s ) = d d s E [ e s X ] = E d d s e s X = E [ X e s X ] Upon setting s=0, we have MX'(0)=E[X]. Repeated differentiation gives the general result. The corresponding result for the characteristic function is φ(k)(0)=ikE[Xk]. The exponential distributionThe density function is fX(t)=λe-λt for t≥0. M X ( s ) = E [ e s X ] = ∫ 0 ∞ λ e - ( λ - s ) t d t = λ λ - s M X ' ( s ) = λ ( λ - s ) 2 M X ' ' ( s ) = 2 λ ( λ - s ) 3 E [ X ] = M X ' ( 0 ) = λ λ 2 = 1 λ E [ X 2 ] = M X ' ' ( 0 ) = 2 λ λ 3 = 2 λ 2 From this we obtain Var [X]=2/λ2-1/λ2=1/λ2. The generating function does not lend itself readily to computing moments, except that g X ' ( s ) = ∑ k = 1 ∞ k s k - 1 P ( X = k ) so that g X ' ( 1 ) = ∑ k = 1 ∞ k P ( X = k ) = E [ X ] For higher order moments, we may convert the generating function to the moment generating function by replacing s with e^s, then work with M_X and its derivatives. The Poisson <m:math overflow="scroll"><m:mrow><m:mo>(</m:mo><m:mi>μ</m:mi><m:mo>)</m:mo></m:mrow></m:math> distributionP(X=k)=e-μμkk!,k≥0, so that g X ( s ) = e - μ ∑ k = 0 ∞ s k μ k k ! = e - μ ∑ k = 0 ∞ ( s μ ) k k ! = e - μ e μ s = e μ ( s - 1 ) We convert to M_X by replacing s with e^s to get MX(s)=eμ(es-1). Then M X ' ( s ) = e μ ( e s - 1 ) μ e s M X ' ' ( s ) = e μ ( e s - 1 ) [ μ 2 e 2 s + μ e s ] so that E [ X ] = M X ' ( 0 ) = μ , E [ X 2 ] = M X ' ' ( 0 ) = μ 2 + μ , and Var [ X ] = μ 2 + μ - μ 2 = μ These results agree, of course, with those found by direct computation with the distribution. Operational properties We refer to the following as operational properties. (T1)If Z=aX+b, then MZ(s)=ebsMX(as),φZ(u)=eiubφX(au),gZ(s)=sbgX(sa) For the moment generating function, this pattern follows from E[e(aX+b)s]=sbsE[e(as)X] Similar arguments hold for the other two. (T2)If the pair {X,Y} is independent, then MX+Y(s)=MX(s)MY(s),φX+Y(u)=φX(u)φY(u),gX+Y(s)=gX(s)gY(s) For the moment generating function, esX and esY form an independent pair for each value of the parameter s. By the product rule for expectation E[es(X+Y)]=E[esXesY]=E[esX]E[esY] Similar arguments are used for the other two transforms. A partial converse for (T2) is as follows: (T3)If MX+Y(s)=MX(s)MY(s), then the pair {X,Y} is uncorrelated. To show this, we obtain two expressions for E[(X+Y)2], one by direct expansion and use of linearity, and the other by taking the second derivative of the moment generating function. E[(X+Y)2]=E[X2]+E[Y2]+2E[XY]MX+Y''(s)=[MX(s)MY(s)]''=MX''(s)MY(s)+MX(s)MY''(s)+2MX'(s)MY'(s) On setting s=0 and using the fact that MX(0)=MY(0)=1, we have E[(X+Y)2]=E[X2]+E[Y2]+2E[X]E[Y] which implies the equality E[XY]=E[X]E[Y]. Note that we have not shown that being uncorrelated implies the product rule. We utilize these properties in determining the moment generating and generating functions for several of our common distributions. Some discrete distributions Indicator function X = I E P ( E ) = p g X ( s ) = s 0 q + s 1 p = q + p s M X ( s ) = g X ( e s ) = q + p e s Simple random variableX=∑i=1ntiIAi (primitive form) P(Ai)=piMX(s)=∑i=1nestipi Binomial(n,p). X=∑i=1nIEiwith{IEi:1≤i≤n}iidP(Ei)=p We use the product rule for sums of independent random variables and the generating function for the indicator function. gX(s)=∏i=1n(q+ps)=(q+ps)nMX(s)=(q+pes)n Geometric(p). P(X=k)=pqk∀k≥0E[X]=q/p We use the formula for the geometric series to get gX(s)=∑k=0∞pqksk=p∑k=0∞(qs)k=p1-qsMX(s)=p1-qes Negative binomial(m,p) If Y_m is the number of the trial in a Bernoulli sequence on which the mth success occurs, and Xm=Ym-m is the number of failures before the mth success, then P(Xm=k)=P(Ym-m=k)=C(-m,k)(-q)kpmwhereC(-m,k)=-m(-m-1)(-m-2)⋯(-m-k+1)k! The power series expansion about t=0 shows that (1+t)-m=1+C(-m,1)t+C(-m,2)t2+⋯for-1<t<1 Hence MXm(s)=pm∑k=0∞C(-m,k)(-q)kesk=p1-qesm Comparison with the moment generating function for the geometric distribution shows that Xm=Ym-m has the same distribution as the sum of m iid random variables, each geometric (p). This suggests that the sequence is characterized by independent, successive waiting times to success. This also shows that the expectation and variance of X_m are m times the expectation and variance for the geometric. Thus E[Xm]=mq/pand Var [Xm]=mq/p2 Poisson(μ)P(X=k)=e-μμkk!∀k≥0 In , above, we establish gX(s)=eμ(s-1) and MX(s)=eμ(es-1). If {X,Y} is an independent pair, with X∼ Poisson (λ) and Y∼ Poisson (μ), then Z=X+Y∼ Poisson (λ+μ). Follows from (T1) and product of exponentials. Some absolutely continuous distributions Uniform on (a,b)fX(t)=1b-aa<t<bMX(s)=∫estfX(t)dt=1b-a∫abestdt=esb-esas(b-a) Symmetric triangular(-c,,c)fX(t)=I[-c,0)(t)c+tc2+I[0,c](t)c-tc2MX(s)=1c2∫-c0(c+t)estdt+1c2∫0c(c-t)estdt=ecs+e-cs-2c2s2=ecs-1cs·1-e-cscs=MY(s)MZ(-s)=MY(s)M-Z(s) where M_Y is the moment generating function for Y∼ uniform (0,c) and similarly for M_Z. Thus, X has the same distribution as the difference of two independent random variables, each uniform on (0,c). Exponential(λ)fX(t)=λe-λt,t≥0 In example 1, above, we show that MX(s)=λλ-s. Gamma(α,λ)fX(t)=1Γ(α)λαtα-1e-λtt≥0MX(s)=λαΓ(α)∫0∞tα-1e-(λ-s)tdt=λλ-sα For α=n, a positive integer, MX(s)=λλ-sn which shows that in this case X has the distribution of the sum of n independent random variables each exponential (λ). Normal(μ,σ2). The standardized normal, Z∼N(0,1)MZ(s)=12π∫-∞∞este-t2/2dt Now st-t22=s22-12(t-s)2 so that MZ(s)=es2/212π∫-∞∞e-(t-s)2/2dt=es2/2 since the integrand (including the constant 1/2π) is the density for N(s,1). X=σZ+μ implies by property (T1) MX(s)=esμeσ2s2/2=expσ2s22+sμ Affine combination of independent normal random variablesSuppose {X,Y} is an independent pair with X∼N(μX,σX2) and Y∼N(μY,σY2). Let Z=aX+bY+c. Then Z is normal, for by properties of expectation and variance μ Z = a μ X + b μ Y + c and σ Z 2 = a 2 σ X 2 + b 2 σ Y 2 and by the operational properties for the moment generating function M Z ( s ) = e s c M X ( a s ) M Y ( b s ) = exp ( a 2 σ X 2 + b 2 σ Y 2 ) s 2 2 + s ( a μ X + b μ Y + c ) = exp σ Z 2 s 2 2 + s μ Z The form of M_Z shows that Z is normally distributed. Moment generating function and simple random variables Suppose X=∑i=1ntiIAi in canonical form. That is, A_i is the event {X=ti} for each of the distinct values in the range of X, with pi=P(Ai)=P(X=ti). Then the moment generating function for X is M X ( s ) = ∑ i = 1 n p i e s t i The moment generating function M_X is thus related directly and simply to the distribution for random variable X. Consider the problem of determining the sum of an independent pair {X,Y} of simple random variables. The moment generating function for the sum is the product of the moment generating functions. Now if Y=∑j=1mujIBj, with P(Y=uj)=πj, we have M X ( s ) M Y ( s ) = ∑ i = 1 n p i e s t i ∑ j = 1 m π j e s u j = ∑ i , j p i π j e s ( t i + u j ) The various values are sums ti+uj of pairs (ti,uj) of values. Each of these sums has probability piπj for the values corresponding to ti,uj. Since more than one pair sum may have the same value, we need to sort the values, consolidate like values and add the probabilties for like values to achieve the distribution for the sum. We have an m-function mgsum for achieving this directly. It produces the pair-products for the probabilities and the pair-sums for the values, then performs a csort operation. Although not directly dependent upon the moment generating function analysis, it produces the same result as that produced by multiplying moment generating functions. Distribution for a sum of independent simple random variablesSuppose the pair {X,Y} is independent with distributions X = [ 1 3 5 7 ] Y = [ 2 3 4 ] P X = [ 0 . 2 0 . 4 0 . 3 0 . 1 ] P Y = [ 0 . 3 0 . 5 0 . 2 ] Determine the distribution for Z=X+Y.

X = [1 3 5 7];
Y = 2:4;
PX = 0.1*[2 4 3 1];
PY = 0.1*[3 5 2];
[Z,PZ] = mgsum(X,Y,PX,PY);
disp([Z;PZ]')
    3.0000    0.0600
    4.0000    0.1000
    5.0000    0.1600
    6.0000    0.2000
    7.0000    0.1700
    8.0000    0.1500
    9.0000    0.0900
   10.0000    0.0500
   11.0000    0.0200

This could, of course, have been achieved by using icalc and csort, which has the advantage that other functions of X and Y may be handled. Also, since the random variables are nonnegative, integer-valued, the MATLAB convolution function may be used (see ). By repeated use of the function mgsum, we may obtain the distribution for the sum of more than two simple random variables. The m-functions mgsum3 and mgsum4 utilize this strategy. The techniques for simple random variables may be used with the simple approximations to absolutely continuous random variables. Difference of uniform distributionThe moment generating functions for the uniform and the symmetric triangular show that the latter appears naturally as the difference of two uniformly distributed random variables. We consider X and Y iid, uniform on [0,1].

tappr
Enter matrix [a b] of x-range endpoints  [0 1]
Enter number of x approximation points  200
Enter density as a function of t  t<=1
Use row matrices X and PX as in the simple case
[Z,PZ] = mgsum(X,-X,PX,PX);
plot(Z,PZ/d)               % Divide by d to recover f(t)
%  plotting details   ---  see

The generating function The form of the generating function for a nonnegative, integer-valued random variable exhibits a number of important properties. X = ∑ k = 0 ∞ k I A i (canonical form) p k = P ( A k ) = P ( X = k ) g X ( s ) = ∑ k = 0 ∞ s k p k As a power series in s with nonnegative coefficients whose partial sums converge to one, the series converges at least for |s|≤1. The coefficients of the power series display the distribution: for value k the probability pk=P(X=k) is the coefficient of s^k. The power series expansion about the origin of an analytic function is unique. If the generating function is known in closed form, the unique power series expansion about the origin determines the distribution. If the power series converges to a known closed form, that form characterizes the distribution, For a simple random variable (i.e., pk=0 for k>n), g_X is a polynomial. The Poisson distributionIn , above, we establish the generating function for X∼ Poisson (μ) from the distribution. Suppose, however, we simply encounter the generating function g X ( s ) = e m ( s - 1 ) = e - m e m s From the known power series for the exponential, we get g X ( s ) = e - m ∑ k = 0 ∞ ( m s ) k k ! = e - m ∑ k = 0 ∞ s k m k k ! We conclude that P ( X = k ) = e - m m k k ! , 0 ≤ k which is the Poisson distribution with parameter μ=m. For simple, nonnegative, integer-valued random variables, the generating functions are polynomials. Because of the product rule (T2), the problem of determining the distribution for the sum of independent random variables may be handled by the process of multiplying polynomials. This may be done quickly and easily with the MATLAB convolution function. Sum of independent simple random variablesSuppose the pair {X,Y} is independent, with g X ( s ) = 1 10 ( 2 + 3 s + 3 s 2 + 2 s 5 ) g Y ( s ) = 1 10 ( 2 s + 4 s 2 + 4 s 3 ) In the MATLAB function convolution, all powers of s must be accounted for by including zeros for the missing powers.

gx = 0.1*[2 3 3 0 0 2];      % Zeros for missing powers 3, 4
gy = 0.1*[0 2 4 4];          % Zero  for missing power 0
gz = conv(gx,gy);
a = ['       Z         PZ'];
b = [0:8;gz]';
disp(a)
       Z         PZ          % Distribution for Z = X + Y
disp(b)
         0         0
    1.0000    0.0400
    2.0000    0.1400
    3.0000    0.2600
    4.0000    0.2400
    5.0000    0.1200
    6.0000    0.0400
    7.0000    0.0800
    8.0000    0.0800

If mgsum were used, it would not be necessary to be concerned about missing powers and the corresponding zero coefficients.

Integral transforms We consider briefly the relationship of the moment generating function and the characteristic function with well known integral transforms (hence the name of this chapter). Moment generating function and the Laplace transform When we examine the integral forms of the moment generating function, we see that they represent forms of the Laplace transform, widely used in engineering and applied mathematics. Suppose F_X is a probability distribution function with FX(-∞)=0. The bilateral Laplace transform for F_X is given by ∫ - ∞ ∞ e - s t F X ( t ) d t The Laplace-Stieltjes transform for F_X is ∫ - ∞ ∞ e - s t F X ( d t ) Thus, if M_X is the moment generating function for X, then MX(-s) is the Laplace-Stieltjes transform for X (or, equivalently, for F_X). The theory of Laplace-Stieltjes transforms shows that under conditions sufficiently general to include all practical distribution functions M X ( - s ) = ∫ - ∞ ∞ e - s t F X ( d t ) = s ∫ - ∞ ∞ e - s t F X ( t ) d t Hence 1 s M X ( - s ) = ∫ - ∞ ∞ e - s t F X ( t ) d t The right hand expression is the bilateral Laplace transform of F_X. We may use tables of Laplace transforms to recover F_X when M_X is known. This is particularly useful when the random variable X is nonnegative, so that FX(t)=0 for t<0. If X is absolutely continuous, then M X ( - s ) = ∫ - ∞ ∞ e - s t f X ( t ) d t In this case, MX(-s) is the bilateral Laplace transform of f_X. For nonnegative random variable X, we may use ordinary tables of the Laplace transform to recover f_X. Use of Laplace transformSuppose nonnegative X has moment generating function M X ( s ) = 1 ( 1 - s ) We know that this is the moment generating function for the exponential (1) distribution. Now, 1 s M X ( - s ) = 1 s ( 1 + s ) = 1 s - 1 1 + s From a table of Laplace transforms, we find 1/s is the transform for the constant 1 (for t≥0) and 1/(1+s) is the transform for e-t,t≥0, so that FX(t)=1-e-tt≥0, as expected. Laplace transform and the densitySuppose the moment generating function for a nonnegative random variable is M X ( s ) = λ λ - s α From a table of Laplace transforms, we find that for α>0, Γ ( α ) ( s - a ) α is the Laplace transform of t α - 1 e a t t ≥ 0 If we put a=-λ, we find after some algebraic manipulations f X ( t ) = λ α t α - 1 e - λ t Γ ( α ) , t ≥ 0 Thus, X∼ gamma (α,λ), in keeping with the determination, above, of the moment generating function for that distribution. The characteristic function Since this function differs from the moment generating function by the interchange of parameter s and iu, where i is the imaginary unit, i2=-1, the integral expressions make that change of parameter. The result is that Laplace transforms become Fourier transforms. The theoretical and applied literature is even more extensive for the characteristic function. Not only do we have the operational properties (T1) and (T2) and the result on moments as derivatives at the origin, but there is an important expansion for the characteristic function. An expansion theorem If E[|X|n]<∞, then φ ( k ) ( 0 ) = i k E [ X k ] , for 0 ≤ k ≤ n and φ ( u ) = ∑ k = 0 n ( i u ) k k ! E [ X k ] + o ( u n ) as u → 0 We note one limit theorem which has very important consequences. A fundamental limit theorem Suppose {Fn:1≤n} is a sequence of probability distribution functions and {φn:1≤n} is the corresponding sequence of characteristic functions. If F is a distribution function such that Fn(t)→F(t) at every point of continuity for F, and φ is the characteristic function for F, then φn(u)→φ(u)∀u If φn(u)→φ(u) for all u and φ is continuous at 0, then φ is the characteristic function for distribution function F such that Fn(t)→F(t)ateachpointofcontinuityofF — □