Likelihood based fitting

"Fitting" simply means estimating some parameters of a model (or really a set of models) based on data. Likelihood-based fitting does this through the likelihood function.

In frequentist frameworks, this typically means doing maximum likelihood estimation. In bayesian frameworks, usually posterior distributions of the parameters are calculated from the likelihood.

Fitting Frameworks

Likelihood fits typically either follow a frequentist framework of maximum likelihood estimation, or a bayesian framework of updating estimates to find posterior distributions given the data.

Maximum Likelihood fits

A maximum likelihood fit means finding the values of the model parameters $(\overrightarrow{\mu },\overrightarrow{\nu })$ which maximize the likelihood, $\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\nu })$ The values which maximize the likelihood, are the parameter estimates, denoted with a "hat" ($\hat{}$):

$$(\overrightarrow{\hat{\mu }},\overrightarrow{\hat{\nu }})\equiv \underset{\overrightarrow{\mu },\overrightarrow{\nu }}{\mathrm{argmax}}\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\nu })$$

These values provide point estimates for the parameter values.

Because the likelihood is equal to the probability of observing the data given the model, the maximum likelihood estimate finds the parameter values for which the data is most probable.

Bayesian Posterior Calculation

In a bayesian framework, the likelihood represents the probability of observing the data given the model and some prior probability distribution over the model parameters.

The prior probability of the parameters, $\pi (\overrightarrow{\mathrm{\Phi }})$, are updated based on the data to provide a posterior distributions

$$p(\overrightarrow{\mathrm{\Phi }};\mathrm{data})=\frac{p(\mathrm{data};\overrightarrow{\mathrm{\Phi }})\pi (\overrightarrow{\mathrm{\Phi }})}{\int p(\mathrm{data};{\overrightarrow{\mathrm{\Phi }}}^{\prime })\pi ({\overrightarrow{\mathrm{\Phi }}}^{\prime })\mathrm{d}{\overrightarrow{\mathrm{\Phi }}}^{\prime }}=\frac{\mathcal{L}(\overrightarrow{\mathrm{\Phi }})\pi (\overrightarrow{\mathrm{\Phi }})}{\int \mathcal{L}({\overrightarrow{\mathrm{\Phi }}}^{\prime })\pi ({\overrightarrow{\mathrm{\Phi }}}^{\prime })\mathrm{d}{\overrightarrow{\mathrm{\Phi }}}^{\prime }}$$

The posterior distribution $p(\overrightarrow{\mathrm{\Phi }};\mathrm{data})$ defines the updated belief about the parameters $\overrightarrow{\mathrm{\Phi }}$.

Methods for considering subsets of models

Often, one is interested in some particular aspect of a model. This may be for example information related to the parameters of interest, but not the nuisance parameters. In this case, one needs a method for specifying precisely what is meant by a model considering only those parameters of interest.

There are several methods for considering sub models which each have their own interpretations and use cases.

Conditioning

Conditional Sub-models can be made by simply restricting the values of some parameters. The conditional likelihood of the parameters $\overrightarrow{\mu }$ conditioned on particular values of the parameters $\overrightarrow{\nu }$ is:

$$\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\nu })\stackrel{\mathrm{conditioned}\mathrm{on}\overrightarrow{\nu }={\overrightarrow{\nu }}_0}{\to }\mathcal{L}(\overrightarrow{\mu })=\mathcal{L}(\overrightarrow{\mu },{\overrightarrow{\nu }}_0)$$

Profiling

The profiled likelihood $\mathcal{L}(\overrightarrow{\mu })$ is defined from the full likelihood, $\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\nu })$, such that for every point $\overrightarrow{\mu }$ it is equal to the full likelihood at $\overrightarrow{\mu }$ maximized over $\overrightarrow{\nu }$.

$$\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\nu })\stackrel{\mathrm{profiling}\overrightarrow{\nu }}{\to }\mathcal{L}(\overrightarrow{\mu })=\underset{\overrightarrow{\nu }}{max}\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\nu })$$

In some sense, the profiled likelihood is the best estimate of the likelihood at every point $\overrightarrow{\mu }$, it is sometimes also denoted with a double hat notation $\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\hat{\hat{\nu }}}(\overrightarrow{\mu }))$.

Marginalization

Marginalization is a procedure for producing a probability distribution $p(\overrightarrow{\mu };\mathrm{data})$ for a set of parameters $\overrightarrow{\mu }$, which are only a subset of the parameters in the full distribution $p(\overrightarrow{\mu },\overrightarrow{\nu };\mathrm{data})$. The marginal probability density $p(\overrightarrow{\mu })$ is defined such that for every point $\overrightarrow{\mu }$ it is equal to the probability at $\overrightarrow{\mu }$ integrated over $\overrightarrow{\nu }$.

$$p(\overrightarrow{\mu },\overrightarrow{\nu })\stackrel{\mathrm{marginalizing}\overrightarrow{\nu }}{\to }p(\overrightarrow{\mu })={\int }_{\overrightarrow{\nu }}p(\overrightarrow{\mu },\overrightarrow{\nu })$$

The marginalized probability $p(\overrightarrow{\mu })$ is the probability for the parameter values $\overrightarrow{\mu }$ taking into account all possible values of $\overrightarrow{\nu }$.

Marginalized likelihoods can also be defined, by their relationship to the probability distributions.

Parameter Uncertainties

Parameter uncertainties describe regions of parameter values which are considered reasonable parameter values, rather than single estimates. These can be defined either in terms of frequentist confidence regions or bayesian credibility regions.

In both cases the region is defined by a confidence or credibility level $CL$, which quantifies the meaning of the region. For frequentist confidence regions, the confidence level $CL$ describes how often the confidence region will contain the true parameter values if the model is a sufficiently accurate approximation of the truth. For bayesian credibility regions, the credibility level $CL$ describes the bayesian probability that the true parameter value is in that region for under the given model.

The confidence or credibility regions are described by a set of points $\{\overrightarrow{\mu }{\}}_{\mathrm{CL}}$ which meet some criteria. In most situations of interest, the credibility region or confidence region for a single parameter, $\mu$, is effectively described by an interval:

$$\{\mu {\}}_{\mathrm{CL}}=[{\mu }_{\mathrm{CL}}^{-},{\mu }_{\mathrm{CL}}^{+}]$$

Typically indicated as:

$$\mu =X_{-\mathrm{down}}^{+\mathrm{up}}$$

or, if symmetric intervals are used:

$$\mu =X\pm \mathrm{unc}.$$

Frequentist Confidence Regions

Frequentist confidence regions are random variables of the observed data. These are very often the construction used to define the uncertainties reported on a parameter.

If the same experiment is repeated multiple times, different data will be osbserved each time and a different confidence set $\{\overrightarrow{\mu }{\}}_{\mathrm{CL}}^i$ will be found for each experiment. If the data are generated by the model with some set of values ${\overrightarrow{\mu }}_{\mathrm{gen}}$, then the fraction of the regions $\{\overrightarrow{\mu }{\}}_{\mathrm{CL}}^i$ which contain the values ${\overrightarrow{\mu }}_{\mathrm{gen}}$ will be equal to the confidence level $\mathrm{CL}$. The fraction of intervals which contain the generating parameter value is referred to as the "coverage".

From first principles, the intervals can be constructed using the Neyman construction.

In practice, the likelihood can be used to construct confidence regions for a set of parameters $\overrightarrow{\mu }$ by using the profile likelikhood ratio:

$$\mathrm{\Lambda }\equiv \frac{\mathcal{L}(\overrightarrow{\mu },\overrightarrow{\hat{\nu }}(\overrightarrow{\mu }))}{\mathcal{L}(\overrightarrow{\hat{\mu }},\overrightarrow{\hat{\nu }})}$$

i.e. the ratio of the profile likelihood at point $\overrightarrow{\mu }$ to the maxmimum likelihood. For technical reasons, the negative logarithm of this quantity is typically used in practice.

Each point $\overrightarrow{\mu }$ can be tested to see if it is in the confidence region, by checking the value of the likelihood ratio at that point and comparing it to the expected distribution if that point were the true generating value of the data.

$$\{\overrightarrow{\mu }{\}}_{\mathrm{CL}}=\{\overrightarrow{\mu }:-\log (\mathrm{\Lambda })<{\gamma }_{\mathrm{CL}}(\overrightarrow{\mu })\}$$

The cutoff value ${\gamma }_{\mathrm{CL}}$ must be chosen to match this desired coverage of the confidence set.

Under some conditions, the value of ${\gamma }_{\mathrm{CL}}$ is known analytically for any desired confidence level, and is independent of $\overrightarrow{\mu }$, which greatly simplifies estimating confidence regions.

Constructing Frequentist Confidence Regions in Practice

When a single fit is performed by some numerical minimization program and parameter values are reported along with some uncertainty values, they are usually reported as frequentist intervals. The MINUIT minimizer which evaluates likelihood functions has two methods for estimating parameter uncertainties.

These two methods are the most commonly used methods for estimating confidence regions in a fit; they are the minos method, and the hessian method. In both cases, Wilk's theorem is assumed to hold at all points in parameter space, such that ${\gamma }_{\mathrm{CL}}$ is independent of $\overrightarrow{\mu }$.

When ${\gamma }_{\mathrm{CL}}$ is independent of $\overrightarrow{\mu }$ the problem simplifies to finding the boundaries where $-\log (\mathrm{\Lambda })={\gamma }_{\mathrm{CL}}$. This boundary point is referred to as the "crossing", i.e. where $-\log (\mathrm{\Lambda })$ crosses the threshold value.

The Minos method for estimating confidence regions

In the minos method, once the best fit point $\overrightarrow{\hat{\mu }}$ is determined, the confidence region for any parameter ${\mu }_i$ can be found by moving away from its best fit value ${\hat{\mu }}_i$. At each value of ${\mu }_i$, the other parameters are profiled, and $-\log \mathrm{\Lambda }$ is calculated.

Following this procedure, ${\mu }_i$ is searched for the boundary of the confidence regions, where $-\log \mathrm{\Lambda }={\gamma }_{\mathrm{CL}}$.

The search is performed in both directions, away from the best fit value of the parameter and the two crossings are taken as the borders of the confidence region.

This procedure has to be followed sepately for each parameter ${\mu }_i$ for which a confidence interval is calculated.

The Hessian method for estimating confidence regions

The Hessian method relies on the second derivatives (i.e. the hessian) of the likelihood at the best fit point.

By assuming that the shape of the likelihood function is well described by its second-order approximation, the values at which $-\log (\mathrm{\Lambda })={\gamma }_{\mathrm{CL}}$ can be calculated analytically without the need for a seach

$${\mu }_i^{\mathrm{crossing}}-\hat{\mu }\propto {\left(\frac{{\partial }^2\mathcal{L}(\overrightarrow{\hat{\mu }})}{\partial {\mu }_i^2}\right)}^{-2}$$

By computing and then inverting the full hessian matrix, all individual confidence regions and the full covariance matrix are determined. By construction, this method always reports symmetric confidence intervals, as it assumes that the likelihood is well described by a second order expansion.

Bayesian Credibility Regions

Often the full posterior probability distribution is summarized in terms of some credible region which contains some specified portion of the posterior probability of the parameter.

$$\{\overrightarrow{\mu }{\}}_{\mathrm{CL}}=\{\overrightarrow{\mu }:\overrightarrow{\mu }\in \mathrm{\Omega },{\int }_{\mathrm{\Omega }}p(\overrightarrow{\mu };\mathrm{data})=\mathrm{CL}\}$$

The credible region represents a region in which the bayesian probability of the parameter being in that region is equal to the chosen Credibility Level.