11 Maximum Likelihood Estimate and Estimator

Formally, let $\mathbf{X}=(X_1,X_2,\dots ,X_n)$) be a random sample from a distribution with parameter $\theta$. Suppose we have observed the values of $X$ as $x=(x_1,x_2,\dots ,x_n)$. A maximum likelihood estimate (MLE) of $\theta$, denoted as ${\hat{\theta }}_{MLE}$, is a value of $\theta$ that maximises the likelihood function:

$${\hat{\theta }}_{\text{MLE}}=\arg \underset{\theta }{\max }L(\theta |x)$$

Maximum Likelihood Estimator:

A Maximum Likelihood Estimator of the parameter $\theta$, denoted as ${\hat{\Theta }}_{\text{MLE}}$, is a random variable

$${\hat{\Theta }}_{\text{MLE}}={\hat{\Theta }}_{\text{MLE}}(X)$$

whose value is given by ${\hat{\theta }}_{\text{MLE}}$ when $X=x$

Cost Function for computing the MLE

Cost Function : A function that maps a set of events into a number representing the “cost” of that event occurring, also called the loss function or objective function. For computing the MLE, there is one-to-one mapping between the likelihood function and the cost function: Given some data $D$.

$$J(\theta ,D)=-\log L(\theta |D)$$

Why Negative Logarithm ?

• Convention - Software for minimization problems.
• Convenience - Logarithms simplify multiplication into addition. This makes differentiation easier.
• Numerical stability - Product of small probabilities can converge to zero, causing computational issues due to machine precision limits.