Linear Regression

Suppose that a pair {X,Y}{X,Y} of random variables has a joint distribution. A value X(ω)X(ω) is observed. It is desired to estimate the corresponding value Y(ω)Y(ω). Obviously there is no rule for determining Y(ω)Y(ω) unless Y is a function of X. The best that can be hoped for is some estimate based on an average of the errors, or on the average of some function of the errors.

Suppose X(ω)X(ω) is observed, and by some rule an estimate Y^(ω)Y^(ω) is returned. The error of the estimate is Y(ω)-Y^(ω)Y(ω)-Y^(ω). The most common measure of error is the mean of the square of the error

The choice of the mean square has two important properties: it treats positive and negative errors alike, and it weights large errors more heavily than smaller ones. In general, we seek a rule (function) r such that the estimate Y^(ω)Y^(ω) is rX(ω)rX(ω). That is, we seek a function r such that

The problem of determining such a function is known as the regression problem. In the unit on Regression, we show that this problem is solved by the conditional expectation of Y, given X. At this point, we seek an important partial solution.

The regression line of Y on X

We seek the best straight line function for minimizing the mean squared error. That is, we seek a function r of the form u=r(t)=at+bu=r(t)=at+b. The problem is to determine the coefficients a, b such that

We write the error in a special form, then square and take the expectation.

Standard procedures for determining a minimum (with respect to a) show that this occurs for

Thus the optimum line, called the regression line of Y on X, is

The second form is commonly used to define the regression line. For certain theoretical purposes, this is the preferred form. But for calculation, the first form is usually the more convenient. Only the covariance (which requres both means) and the variance of X are needed. There is no need to determine Var [Y] Var [Y] or ρ.

jdemo1
jcalc
Enter JOINT PROBABILITIES (as on the plane)  P
Enter row matrix of VALUES of X  X
Enter row matrix of VALUES of Y  Y
 Use array operations on matrices X, Y, PX, PY, t, u, and P
EX = total(t.*P)
EX =   0.6420
EY = total(u.*P)
EY =   0.0783
VX = total(t.^2.*P) - EX^2
VX =   3.3016
CV = total(t.*u.*P) - EX*EY
CV =  -0.1633
a = CV/VX
a  =  -0.0495
b = EY - a*EX
b  =   0.1100           % The regression line is u = -0.0495t + 0.11

Example 3: Distribution of Example 5 from "Random Vectors and MATLAB" and Example 12 from "Function of Random Vectors"

The pair {X,Y}{X,Y} has joint density fXY(t,u)=637(t+2u)fXY(t,u)=637(t+2u) on the region 0≤t≤20≤t≤2, 0≤u≤max{1,t}0≤u≤max{1,t} (see Figure Figure 1). Determine the regression line of Y on X. If the value X(ω)=1.7X(ω)=1.7 is observed, what is the best mean-square linear estimate of Y(ω)Y(ω)?

Figure 1: Regression line for Example 3.

ANALYTIC SOLUTION

E [ X ] = 6 37 ∫ 0 1 ∫ 0 1 ( t 2 + 2 t u ) d u d t + 6 37 ∫ 1 2 ∫ 0 t ( t 2 + 2 t u ) d u d t = 50 / 37 E [ X ] = 6 37 ∫ 0 1 ∫ 0 1 ( t 2 + 2 t u ) d u d t + 6 37 ∫ 1 2 ∫ 0 t ( t 2 + 2 t u ) d u d t = 50 / 37 (10)

The other quantities involve integrals over the same regions with appropriate integrands, as follows:

Table 1

Quantity	Integrand	Value
E [ X 2 ] E [ X 2 ]	t 3 + 2 t 2 u t 3 + 2 t 2 u	779/370
E [ Y ] E [ Y ]	t u + 2 u 2 t u + 2 u 2	127/148
E [ X Y ] E [ X Y ]	t 2 u + 2 t u 2 t 2 u + 2 t u 2	232/185

Then

Var [ X ] = 779 370 - 50 37 2 = 3823 13690 Cov [ X , Y ] = 232 185 - 50 37 · 127 148 = 1293 13690 Var [ X ] = 779 370 - 50 37 2 = 3823 13690 Cov [ X , Y ] = 232 185 - 50 37 · 127 148 = 1293 13690 (11)

and

a = Cov [ X , Y ] / Var [ X ] = 1293 3823 ≈ 0 . 3382 , b = E [ Y ] - a E [ X ] = 6133 15292 ≈ 0 . 4011 a = Cov [ X , Y ] / Var [ X ] = 1293 3823 ≈ 0 . 3382 , b = E [ Y ] - a E [ X ] = 6133 15292 ≈ 0 . 4011 (12)

The regression line is u=at+bu=at+b. If X(ω)=1.7X(ω)=1.7, the best linear estimate (in the mean square sense) is Y^(ω)=1.7a+b=0.9760Y^(ω)=1.7a+b=0.9760 (see Figure 1 for an approximate plot).

APPROXIMATION

tuappr
Enter matrix [a b] of X-range endpoints  [0 2]
Enter matrix [c d] of Y-range endpoints  [0 2]
Enter number of X approximation points  400
Enter number of Y approximation points  400
Enter expression for joint density  (6/37)*(t+2*u).*(u<=max(t,1))
Use array operations on X, Y, PX, PY, t, u, and P
EX = total(t.*P)
EX =  1.3517                   % Theoretical = 1.3514
EY = total(u.*P)
EY =  0.8594                   % Theoretical = 0.8581
VX = total(t.^2.*P) - EX^2
VX =  0.2790                   % Theoretical = 0.2793
CV = total(t.*u.*P) - EX*EY
CV =  0.0947                   % Theoretical = 0.0944
a = CV/VX
a  =  0.3394                   % Theoretical = 0.3382
b = EY - a*EX
b  =  0.4006                   % Theoretical = 0.4011
y = 1.7*a + b
y  =  0.9776                   % Theoretical = 0.9760

An interpretation of ρ²

The analysis above shows the minimum mean squared error is given by

If ρ=0ρ=0, then E[(Y-Y^)2]=σY2E[(Y-Y^)2]=σY2, the mean squared error in the case of zero linear correlation. Then, ρ² is interpreted as the fraction of uncertainty removed by the linear rule and X. This interpretation should not be pushed too far, but is a common interpretation, often found in the discussion of observations or experimental results.

More general linear regression

Consider a jointly distributed class. {Y,X1,X2,⋯,Xn}{Y,X1,X2,⋯,Xn}. We wish to deterimine a function U of the form

If U satisfies this minimum condition, then E[(Y-U)V]=0E[(Y-U)V]=0, or, equivalently

To see this, set W=Y-UW=Y-U and let d2=E[W2]d2=E[W2]. Now, for any α

If we select the special

This implies E[WV]2≤0E[WV]2≤0, which can only be satisfied by E[WV]=0E[WV]=0, so that

On the other hand, if E[(Y-U)V]=0E[(Y-U)V]=0 for all V of the form above, then E[(Y-U)2]E[(Y-U)2] is a minimum. Consider

Since U-VU-V is of the same form as V, the last term is zero. The first term is fixed. The second term is nonnegative, with zero value iff U-V=0a.s.U-V=0a.s. Hence, E[(Y-V)2]E[(Y-V)2] is a minimum when V=UV=U.

If we take V to be 1,X1,X2,⋯,Xn1,X1,X2,⋯,Xn, successively, we obtain n+1n+1 linear equations in the n+1n+1 unknowns a0,a1,⋯,ana0,a1,⋯,an, as follows.

For each i=1,2,⋯,ni=1,2,⋯,n, we take (2)-E[Xi]·(1)(2)-E[Xi]·(1) and use the calculating expressions for variance and covariance to get

These n equations plus equation (1) may be solved alagebraically for the a_i.

In the important special case that the X_i are uncorrelated (i.e., Cov [Xi,Xj]=0 Cov [Xi,Xj]=0 for i≠ji≠j), we have

and

In particular, this condition holds if the class {Xi:1≤i≤n}{Xi:1≤i≤n} is iid as in the case of a simple random sample (see the section on "Simple Random Samples and Statistics).

Examination shows that for n=1n=1, with X1=XX1=X, a0=ba0=b, and a1=aa1=a, the result agrees with that obtained in the treatment of the regression line, above.