Maximum Likelihood Estimation

The Likelihood Function

The probability density function, or pdf, for a random variable $y$, conditioned on a set of parameters $\theta$ is denoted $f(y|\theta ).$

The joint density, or likelihood function is

$$𝑓({𝑦}_1,\dots ,{𝑦}_{𝑛}|\theta )=\prod _{i=1}^{𝑛}𝑓({𝑦}_{𝑖}|\theta )=𝐿(\theta |𝑦).$$

It is usually simpler to work with the log of the likelihood function:

$$\ln 𝐿(\theta |y)=\sum _{i=1}^n\ln 𝑓({𝑦}_{𝑖}|\theta ).$$

MLE estimation: choose $\theta$ to maximize log likelihood function.

The necessary condition is

$$\frac{\partial \ln 𝐿(\theta |𝑦,𝑋)}{\partial \theta }=0,$$

which is called the likelihood equation.

Properties of MLE

Under regularity, the maximum likelihood estimator (MLE) has the following asymptotic properties:

1. Consistency: $\mathrm{plim}\hat{\theta }={\theta }_0$.
2. Asymptotic normality: $\hat{\theta }\stackrel{a}{\sim }N[{\theta }_0,\{𝐼({\theta }_0){\}}^{-1}]$, where$$𝐼({\theta }_0)=-{𝐸}_0[\frac{{\partial }^2\ln 𝐿}{\partial {\theta }_0\partial {\theta }_0^{\prime }}].$$
3. Asymptotic efficiency: $\hat{\theta }$ is asymptotically efficient and achieves the Cramer Rao lower bound for consistent estimators.

For each observations, we have log-density $\ln 𝑓({𝑦}_{𝑖}|\theta )$.Denote ${𝑔}_{𝑖}=\frac{\partial \ln 𝑓({𝑦}_{𝑖}|\theta )}{\partial \theta }$ and ${𝐻}_{𝑖}=\frac{{\partial }^2\ln 𝑓({𝑦}_{𝑖}|\theta )}{\partial \theta \partial {\theta }^{\prime }}$, $i=1,\dots ,n$. Then we have

$$𝑔=\frac{\partial \ln 𝐿(\theta |𝑦)}{\partial \theta }=\sum _{i=1}^{𝑛}\frac{\partial \ln 𝑓({𝑦}_{𝑖}|\theta )}{\partial \theta }=\sum _{i=1}^{𝑛}{𝑔}_{𝑖},$$

and

$$\text{Hessian Matrix }𝐻=\frac{{\partial }^2\ln 𝐿(\theta |𝑦)}{\partial \theta \partial {\theta }^{\prime }}=\sum _{𝑖=1}^{𝑛}\frac{{\partial }^2\ln 𝑓({𝑦}_{𝑖}|\theta )}{\partial \theta \partial {\theta }^{\prime }}=\sum _{𝑖=1}^{𝑛}{𝐻}_{𝑖}.$$

Information Matrix Equality

Information matrix equality:

$$\begin{gathered} \mathrm{Var}\left[\frac{\partial \ln 𝐿({\theta }_0|𝑦)}{\partial {\theta }_0}\right]={𝐸}_0[(\frac{\partial \ln 𝐿({\theta }_0|𝑦)}{\partial {\theta }_0})(\frac{\partial \ln 𝐿({\theta }_0|𝑦)}{\partial {\theta }_0^{\prime }})] \\ =-{𝐸}_0[\frac{{\partial }^2\ln 𝐿({\theta }_0|𝑦)}{\partial {\theta }_0\partial {\theta }_0^{\prime }}] \end{gathered}$$

Consistency

Let ${\theta }_0$ be the true value of the parameter, $\hat{\theta }$ is the MLE, and $\theta$ be any other estimator for $\mathrm{\Theta }$. Then MLE $\hat{\theta }$ is consistent, i.e.,

$$\mathrm{plim}\hat{\theta }={\theta }_0.$$

Asymptotic Normality

The MLE $\hat{\theta }$ has an asymptotic normal distribution,

$$\sqrt{n}(\hat{\theta }-{\theta }_0)\stackrel{d}{\to }𝑁[0,\left\{{-{𝐸}_0[\frac{1}{𝑛}𝐻({\theta }_0)]}^{-1}\right\}],$$

so we have

$$\hat{\theta }\stackrel{a}{\sim }𝑁[{\theta }_0,\{𝐼({\theta }_0){\}}^{-1}].$$

Asymptotic Efficiency

Cramer-Rao Lower Bound

Assuming that the density of ${𝑦}_{𝑖}$ satisfies the regularity conditions, the asymptotic variance of a consistent and asymptotically normally distributed estimator of the parameter vector ${\theta }_0$ will always be at least as large as

$$[𝐼({\theta }_0){]}^{-1}=(-{𝐸}_0[\frac{{\partial }^2\ln 𝐿({\theta }_0)}{\partial {\theta }_0\partial {\theta }_0^{\prime }}]{)}^{-1}.$$

Hypothesis Test

Likelihood Ratio Test:

$$-2\ln \frac{{\hat{L}}_R}{{\hat{L}}_U}\sim {\chi }^2(J).$$

Wald Test:

$$W=[𝑐(\hat{\theta })-𝑞{]}^{\prime }(\mathrm{Asy}.\mathrm{Var}[c(\hat{\theta })-𝑞{]}^{-1}[𝑐(\hat{\theta })-𝑞]\sim {\chi }^2(𝐽).$$

Lagrange Multiplier Test:

$$\mathrm{LM}=(\frac{\partial \ln L({\hat{\theta }}_{𝑅})}{\partial {\hat{\theta }}_{𝑅}}{)}^{\prime }[𝐼({\hat{\theta }}_{𝑅}){]}^{-1}(\frac{\partial \ln 𝐿({\hat{\theta }}_{𝑅})}{\partial {\hat{\theta }}_{𝑅}}{)}^{\prime }\sim {\chi }^2(J),$$

where $J$ is the number of restrictions.