|
In practical applications, we need to determine the values of hyperparameters, such as the an optimal order of polynomial $M$ (model complexity) or the regularization coefficient $\lambda$. If data is plentiful, one approach is simply to test different settings on a validation set and select the one with the best predictive performance. If the model design is iterated many times, then some over-fitting to the validation data can occur and we may need a third test set on which the performance is finally evaluated.
If the data for training and testing is limited, one solution is to use cross-validation, which allows a proportion $(S-1)/S$ of the available data to be used for training while making use of all of the data to assess performance. When data is particularly scarce, we may adopt the leave-one-out technique where $S=N$
One major drawback of cross-validation is that the number of training runs is increased by a factor of $S$. Further, exploring combinations of settings could require a number of training runs that is exponential in the number of parameters. We therefore need a measure of performance which depends only on the training data and which does not suffer from bias due to over-fitting.
Historically various 'information criteria' have been proposed that attempt to correct for the bias of maximum likelihood by the addition of a penalty term to compensate for the over-fitting of more complex models. For example, the Akaike information criterion, or AIC, chooses the model which the quantity
\[ \ln p(\mathcal{D} \mid \mathbf{w}_\text{ML}) - M \]
is the largest. A variant of this quantity, the Bayesian information criterion, or BIC, will be discussed in Section 4.4.1. Such criteria do not take account of the uncertainty in the model parameters, however, they usually favour overly simple models in practice. In Section 3.4, we turn to a fully Bayesian approach where complexity penalties arise in a natural and principled way.
|