Linear Regression Model

Population regression equation

$$\begin{array}{rl}y& =f(x_1,x_2,\dots ,x_K)+\epsilon \\ & =x_1{\beta }_1+x_2{\beta }_2+\cdots +x_K{\beta }_K+\epsilon \end{array}$$

• Deterministic part and random part
• Linear relationship

Sample regression equation

$$y_{𝑖}=x_{𝑖1}{\beta }_1+x_{𝑖2}{\beta }_2+\dots +x_{𝑖𝐾}{\beta }_{𝐾}+{\epsilon }_{𝑖}$$

• $y_{𝑖}$: Observed value of the interested variable;
• $x_{𝑖1}{\beta }_1+x_{𝑖2}{\beta }_2+\dots +x_{𝑖𝐾}$: a deterministic part;
• ${\epsilon }_{𝑖}$: A random part;
• matrix form : $y=X\beta +\epsilon$

Target. Estimate unknown parameters, make predictions.

Assumptions

Assumption 1 (Full Rank). $X$ is an $n\times K$ matrix with rank $K$, i.e., there are no exact linear relationships among the variables.

Assumption 2 (Mean independence). The expected value of disturbance is zero conditional on the observation, i.e.,

$$E[\epsilon |X]=\left[\begin{array}{c}E[{\epsilon }_1|X]\\ E[{\epsilon }_2|X]\\ ⋮\\ E[{\epsilon }_n|X]\end{array}\right]=0.$$

∵ $𝐶𝑜𝑣(𝑋,\epsilon )=𝐶𝑜{𝑣}_{𝑋}(𝑋,𝐸(\epsilon |𝑋))$ ∴ Mean independence implies that $𝐶𝑜𝑣(\epsilon ,𝑋)=0$.

Assumption 3 (Homoscedasticity). The variances and covariances of the disturbances

$$Var[{\epsilon }_i|X]={\sigma }^2,\text{ for all }i=1,\dots ,n$$$$Cov[{\epsilon }_i,{\epsilon }_j|X]=0\text{, for all }i\ne j$$

which is summarized as $E[\epsilon {\epsilon }^{\prime }|X]={\sigma }^{2}I$.

Least square regression

Population regression:

$$y_i=x_i\beta +{\epsilon }_i$$

Sample regression:

$$y_i=x_{i}b+e_i$$

The least squares coefficient vector minimizes the sum of squared residuals:

$$\sum _{i=1}^n{e}_i^2=\sum _{i=1}^n(y_i-x_i^{\prime }b{)}^2$$

and the solution is

$$b=(X^{\prime }X{)}^{-1}{X}^{\prime }y$$

Proposition

1. The least squares residuals sum to zero.
2. $\overline{y}={\overline{x}}^{\prime }b$.
3. $\overline{y}=\overline{\hat{y}}$.

It is important to note that none of these results need hold if the regression does not contain a constant term.

Least squares partitions the vector $y$ into two orthogonal parts,

$$y=𝑃y+𝑀y=\mathrm{projection}+\mathrm{residual}$$$$y=X(X^{\prime }X{)}^{-1}{X}^{\prime }y+[I-X(X^{\prime }X{)}^{-1}{X}^{\prime }]y$$

Partitioned regression

Suppose that the regression involves two sets of variables, ${𝑋}_1$ and ${𝑋}_2$. Thus,

$$y=X\beta +\epsilon =X_1{\beta }_1+X_2{\beta }_2+\epsilon .$$

The normal equations are

$$\left[\begin{array}{cc}{X}_1^{\prime }{X}_1& X_1^{\prime }{X}_2\\ X_2^{\prime }{X}_1& X_2^{\prime }{X}_2\end{array}\right]\left[\begin{array}{c}{b}_1\\ b_2\end{array}\right]=\left[\begin{array}{c}{X}_1^{\prime }y\\ X_2^{\prime }y\end{array}\right]$$

Theorem (Orthogonal Partitioned Regression). In the multiple linear least squares regression of $y$ on two sets of variables ${𝑋}_1$ and ${𝑋}_2$, if the two sets of variables are orthogonal, then the separate coefficient vectors can be obtained by separate regressions of $y$ on ${𝑋}_1$ alone and $y$ on ${𝑋}_2$ alone.

Theorem (Frisch-Waugh-Lovell Theorem). In the linear least squares regression of vector $y$ on two sets of variables, ${𝑋}_1$ and ${𝑋}_2$, the subvector $b_2$ is the set of coefficients obtained when the residuals from a regression of $y$ on ${𝑋}_1$ alone are regressed on the set of residuals obtained when each column of ${𝑋}_2$ is regressed on ${𝑋}_1$.

Theorem (Change in the Sum of Squares When a Variable is Added to a Regression). If ${𝑒}^{\prime }𝑒$ is the sum of squared residuals when $y$ is regressed on $X$ and ${𝑢}^{\prime }𝑢$ is the sum of squared residuals when $y$ is regressed on $X$ and $z$, then

$$u^{\prime }u=e^{\prime }e-c^2(z_{\ast }^{\prime }{z}_{\ast })\le e^{\prime }e,$$

where $c$ is the coefficient on $z$ in the long regression of $y$ on $[X,z]$ and ${𝑧}_{\ast }=𝑀𝑧$ is the vector of residuals when $z$ is regressed on $X$.

Goodness of fit

We want to know how the variation of $y$ is explained by the variation of $x$:

$$y_i-\overline{y}=\hat{y}-\overline{y}+e_i=(x_i-\overline{x}{)}^{\prime }b+e$$

We can obtain a measure of how well the regression line fits the data by using the

$$\mathrm{coefficient}\mathrm{of}\mathrm{determination}:\frac{SSR}{SST}=\frac{b^{\prime }{X}^{\prime }{M}^{0}Xb}{y^{\prime }{M}^{0}y}=1-\frac{e^{\prime }e}{y^{\prime }{M}^{0}y}.$$

Theorem (Change in $R^{\mathrm{𝟐}}$ When a Variable is Added to a Regression). Let ${𝑅}_{𝑋𝑧}^2$ be the coefficient of determination in the regression of $y$ on $X$ and an additional variable $z$, let ${𝑅}_{𝑋}^2$ be the same for the regression of $y$ on $X$ alone, and let $r_{y𝑧}^{\ast }$ be the partial correlation between $y$ and $z$, controlling for $X$. Then

$${𝑅}_{𝑋𝑧}^2={𝑅}_{𝑋}^2+(1-{𝑅}_{𝑋}^2){𝑟}_{y𝑧}^{\ast 2},$$

• where the partial correlation $r_{yz}^{\ast }$ is the simple correlation between $y_{\ast }$ and ${𝑧}_{\ast }$ , where the square of the partial correlation coefficient is$$r_{yz}^{\ast 2}=\frac{(z_{\ast }^{\prime }{y}_{\ast }{)}^2}{(z_{\ast }^{\prime }{z}_{\ast })(y_{\ast }^{\prime }{y}_{\ast })}.$$

The adjusted ${𝑅}^2$ (for degrees of freedom), which in corporates a penalty for these results is computed as follows

$${\overline{R}}^2=1-\frac{e^{\prime }e/(n-K)}{y^{\prime }{M}^{0}y/(n-1)}.$$

The connection between $R^2$ and ${\overline{R}}^2$ is

$${\overline{R}}^2=1-\frac{n-1}{n-K}(1-R^2).$$