What is … conformal prediction
Conformal prediction is a machine learning framework for uncertainty quantification. It produces statistically valid prediction regions for any underlying point predictor only assuming exchangeability of the data [1]. This is in contrast to traditional point prediction frameworks, which provide a single best estimate for a target variable and often do not quantify the uncertainty associated with that estimate.
Conformal prediction was originally designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. It requires a user-specified significance level for which the algorithm should produce its predictions. This significance level restricts the frequency of errors that the algorithm is allowed to make.
From the blog post [2] some advantages of conformal prediction are:
- Guaranteed coverage: Prediction sets generated by conformal prediction come with coverage guarantees of the true outcome. Conformal prediction does not depend on a well-calibrated model and coverage can also be guaranteed across classes or subgroups.
- Easy to use: Conformal prediction approaches can be implemented from scratch with just a few lines of code.
- Model-agnostic: Conformal prediction works with any machine learning model. It uses the normal outputs of whatever your preferred model is.
- Distribution-free: Conformal prediction makes no assumptions about underlying distributions of data, i.e. it is a non-parametric method.
- No retraining required: Conformal prediction can be used without retraining your model. It is another way of looking at and using, model outputs.