Janne Merilinna, VTT Technical Research Centre of Finland, Finland

In practical applications, model accuracy alone is insufficient; quantifying model uncertainty is crucial, particularly in mission-critical scenarios involving life, money, or reputation. In this paper, we propose a novel method called MACAU (Model-based AleatoriC and epistemic uncertAinty qUantification) and implement it in the LightGBM gradient-boosting framework. MACAU enables the quantification of both aleatoric and epistemic uncertainties in Random Forest (RF). Additionally, MACAU offers enhanced novelty detection capabilities, particularly valuable for identifying out-of-distribution (OOD) samples. We compare MACAU with other RF- or gradient boosted trees-based methods, including RF-native between-variance, quantile regression, inductive conformal prediction, exogeneous model for uncertainty estimation using the Gaussian negative log-likelihood method, Natural Gradient Boosting, and CatBoost. Our evaluation is conducted on both synthetic and real-world regression cases. The results demonstrate the effectiveness of MACAU in quantifying model uncertainty, as measured by the Continuous Ranked Probability Score, as well as detecting OOD samples, as measured by the ROCAUC

Machine Learning, Epistemic and Aleatoric Uncertainty, Out-of-Distribution Detection