The Least Squares Estimator

Finite Sample Properties

Unbiased Estimation

$$𝐸[b]={𝐸}_x\{𝐸[b|X]\}={𝐸}_x[\beta ]=\beta .$$

The interpretation of this result is that for any particular set of observations $X$, the least squares estimator has expectation $\beta$. Therefore, when we average this over the possible values of $X$, we find the unconditional mean is $\beta$ as well.

Omission of Relevant Variables

The estimation is biased when a relevant variable is omitted in the regression:

$$𝐸[b_1|X]={\beta }_1+{𝑃}_{1,2}{\beta }_2$$

where

$${𝑃}_{1,2}=({𝑋}_1^{\prime }{𝑋}_1{)}^{-1}{𝑋}_1^{\prime }{𝑋}_2.$$

Inclusion of Irrelevant Variables

Inclusion of irrelevant variables will not affect the biasness of the relevant variables. However, it has the cost of have a higher covariance matrix, which decrease the efficiency of the estimation.

$$E[(\begin{array}{c}b\\ c\end{array})|X,z]=(\begin{array}{c}\beta \\ \gamma \end{array})==(\begin{array}{c}\beta \\ 0\end{array}).$$

Variance of the Least Squares Estimator

Since $𝐸[b]=\beta$ and $𝐸[\epsilon {\epsilon }^{\prime }|X]={\sigma }^2𝐼$,we have

$$\begin{array}{rl}\mathrm{Var}[b\mid X]& =E[(b-\beta )(b-\beta {)}^{\prime }\mid X]\\ & =E[{\left(X^{\prime }X\right)}^{-1}{X}^{\prime }\epsilon {\epsilon }^{\prime }X{\left(X^{\prime }X\right)}^{-1}\mid X]\\ & ={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }\mathrm{E}[\epsilon {\epsilon }^{\prime }\mid X]X{\left(X^{\prime }X\right)}^{-1}\\ & ={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }\left({\sigma }^{2}I\right)X{\left(X^{\prime }X\right)}^{-1}\\ & ={\sigma }^2{\left(X^{\prime }X\right)}^{-1}\end{array}$$

Theorem (Gauss Markov Theorem). In the linear regression model with regressor matrix $X$, the least squares estimator $b$ is the minimum variance linear unbiased estimator of $\beta$. For any vector of constants $w$, the minimum variance linear unbiased estimator of $w^{\prime }\beta$ in the regression model is $w^{\prime }b$, where $b$ is the least squares estimator.

Estimating the Variance of the Least Squares Estimator

We don’t use

$${\hat{\sigma }}^2=\frac{1}{n}\sum _{i=1}^n{e}_i^2,$$

but use

$$s^2=\frac{e^{\prime }e}{n-K}.$$$$Est.Var[b|X]=\frac{e^{\prime }e}{n-K}(X^{\prime }X{)}^{-1}.$$

If we assume the disturbances are normally distributed, then the estimator $b$ has normal distribution as well,

$$b|X\sim N[\beta ,{\sigma }^2(X^{\prime }X{)}^{-1}].$$

Large Sample Properties

Consistency of the Estimator

Assume

$${\mathrm{plim}}_{n\to \mathrm{\infty }}\frac{X^{\prime }X}{n}=Q,$$

where $Q$ is a positive definite matrix, we have

$$\mathrm{plim}b=\beta .$$

Consistency and Unbiasedness

Consider a sample $x_1,\dots ,x_n$ from a $N(\mu ,{\sigma }^2)$ population, and we want to estimate $\mu$.

• Unbiased but not consistent. $x_1$ is an unbiased estimator of $\mu$ since $E[x_1]=\mu$. But, $x_1$ is not consistent since its distribution does not become more concentrated around $\mu$ as the sample size increases, it’s always $N(\mu ,{\sigma }^2)$
• Consistent but not unbiased. $\tilde{x}=\frac{1}{n-1}\sum _{i=1}^n{x}_i$. Since $E[\tilde{x}]=\frac{n}{n-1}\mu \ne \mu$, $\tilde{x}$ is an biased estimator. When $n\to \mathrm{\infty }$ , $E[\tilde{x}]\to \mu$, so $\tilde{x}$ is a consistent estimator.

Asymptotic Normality of the Estimator

Theorem (Asymptotic Distribution of $b$ with Independent Observations). If $\{{\epsilon }_i\}$ are independently distributed with mean zero and finite variance ${\sigma }^2$ and $x_{ik}$ is such that the Grenander conditions are met, then

$$b\stackrel{a}{\sim }N[\beta ,\frac{{\sigma }^2}{n}{Q}^{-1}].$$

Grenander Conditions

1. For each column of $X$, $x_k$, if ${𝑑}_{nk}^2=x_k^{\prime }{x}_k$, then $\underset{n\to \mathrm{\infty }}{lim}{d}_{nk}^2=+\mathrm{\infty }$.
2. $\underset{n\to \mathrm{\infty }}{lim}\frac{x_{ik}^2}{d_{nk}^2}=0$ for all $i=1,2,\dots ,n$.
3. Let ${𝑅}_n$ be the sample correlation matrix of the columns of $X$, excluding the constant term if there is one. Then $\underset{n\to \mathrm{\infty }}{lim}{R}_n=𝐶$, a positive definite matrix.

Interval Estimation

The ratio

$$t_K=\frac{b_k-{\beta }_k}{\sqrt{s^2{S}^{kk}}},$$

where $S^{kk}$ denotes the $k$th element of $(X^{\prime }X{)}^{-1}$, has a $t$ distribution with $n-K$ degrees of freedom, and a confidence interval for ${\beta }_k$ can be formed using

$$Prob[b_k-t_{(1-\alpha /2),[n-K]}\sqrt{s^2{S}^{kk}}\le {\beta }_k\le b_k+t_{(1-\alpha /2),[n-K]}\sqrt{s^2{S}^{kk}}]=1-\alpha .$$

Prediction

The prediction variance is

$$𝑉𝑎𝑟[e^0|X,x_0]={\sigma }^2+{x^0}^{\prime }[{\sigma }^2(X^{\prime }X{)}^{-1}]x^0,$$

and the prediction interval is

$${\hat{y}}^0\pm t_{(1-\alpha /2),[n-K]}se(e^0).$$