Hypothesis Tests and Model Selection

General Linear Hypothesis

The linear regression

$$y=X\beta +\epsilon$$

has restrictions

$$\begin{array}{c}{r}_{11}{\beta }_1+r_{12}{\beta }_2+\cdots +r_{1K}{\beta }_K=q_1\\ r_{21}{\beta }_1+r_{22}{\beta }_2+\cdots +r_{2K}{\beta }_K=q_2\\ \cdots \\ r_{J1}{\beta }_1+r_{J2}{\beta }_2+\cdots +r_{JK}{\beta }_K=q_J\end{array}$$

or in the matrix form

$$R\beta =q$$

We might have the hypothesis that

$$\begin{array}{l}{H}_0:R\beta -q=0\\ H_1:R\beta -q\ne 0\end{array}$$

There are two types of errors.

1. Definition (size of a test). Type I error: The null hypothesis is correct, but we rejct it.
2. Definition (power of a test). Type II error: The null hypothesis is incorrect, but we don't rejct it.

Two Approaches to Testing Hypotheses

Wald tests. If the hypothesis is correct, the sample discrepancy, $𝑅𝑏-𝑞$ should be close to zero.

Fit based tests. A measure of how much ${𝑅}^2$ falls when we impose the restrictions.

Wald Tests

The $ t$ Test

Statistic: $t$ distribution with $n-K$ degree of freedom,

$$t_k=\frac{b_k-{\beta }_k^0}{\sqrt{s^2{S}^{kk}}}$$

$1-\alpha$ confidence interval

$$\mathrm{Prob}\{-t_{(1-\frac{\alpha }{2}),[n-K]}^{\ast }<t_k<+t_{(1-\frac{\alpha }{2}),[n-K]}^{\ast }\}.$$

The $ F$ Statistic

$$F[J,n-K\mid X]=\frac{(Rb-q{)}^{\prime }{\left\{R\left[s^2{\left(X^{\prime }X\right)}^{-1}\right]R^{\prime }\right\}}^{-1}(Rb-q)}{J}$$

For testing a single restriction, the $t$ statistic is the square root of the $F$ statistic

$$t^2=\frac{(\hat{q}-q{)}^2}{\mathrm{Var}(\hat{q}-q\mid X)}=F[1,n-K]$$

Fit Tests

The Restricted Least Square Estimator

$𝑏$ and ${𝑏}_{\ast }$ are the unrestricted and restricted estimators, then

$$b_{\ast }=b+(X^{\prime }X{)}^{-1}{R}^{\prime }[R(X^{\prime }X{)}^{-1}{R}^{\prime }{]}^{-1}[q-Rb].$$

Remark 1. 如果 $b$ 满足约束条件，那么$b_{\ast }=b$;如果不满足约束条件，则全部参数都会发生变化。

Remark 2. 除非$b_{\ast }=b$, 不然我们总有

$$\text{SSE(restricted)}>\text{SSE(unrestricted)}$$

Remark 3

$$\begin{array}{rl}\mathrm{Var}[b_{\ast }|X]=& \mathrm{Var}[b|X]\\ & -\text{a nonnegatvie definite matrix }\end{array}$$

The Loss of Fit

For a coefficient restriction

$$t_z^2=\frac{(R_{Xz}^2-R_X^2)/1}{(1-R_{Xz}^2)/(n-K)}$$$$r_{yz}^{\ast 2}=\frac{t_z^2}{t_z^2+(n-K)}$$

And for general restrictions

$$F[J,n-K]=\frac{(R^2-R_{\ast }^2)/J}{(1-R^2)/(n-K)}$$

Testing the Significance of the Regression

Setting ${𝑅}_{\ast }^2=0$ in the general statistic

$$F[J,n-K]=\frac{(R^2-R_{\ast }^2)/J}{(1-R^2)/(n-K)},$$

we have

$$F[J,n-K]=\frac{R^2/(K-1)}{(1-R^2)/(n-K)}.$$