ECAI 2004 Conference Paper

[PDF] [full paper] [prev] [tofc] [next]

Reliability and typicalness for estimating confidence level of individual classifications

Matjaz Kukar

In the past decades machine learning algorithms have been successfully used in many problems, and are emerging as valuable data analysis tools. However, for a serious practical use, their serious impediment is, that more often than not, they cannot produce good (unbiased) assessments of their predictions' quality. In last years, several approaches for estimating reliability or confidence of individual classifiers have emerged, many of them building upon algorithmic theory of randomness, such as (historically ordered) (1) transduction-based confidence estimation, (2) typicalness-based confidence estimation, and (3) transductive reliability estimation. However, all of these approaches have weaknesses, (1) and (3) are tightly bound with particular classifiers, for (2) the interpretation of reliability estimations is not consistent with statistical confidence levels. In the paper we propose a joint approach that compensates the described weaknesses by integrating typicalness-based confidence estimation and transductive reliability estimation in a joint confidence machine. By our approach the machine produces true confidence values (in a usual statistical sense, e.g., a confidence level of 95% means that in 95% the predicted class is also a true class), as well as provides us with a general approach that can be used with (almost) any underlying classifier We perform a series of tests with several different machine learning algorithms in several problem domains. Our experiments show that the proposed approach integrates the "best of both worlds" in terms of discriminating performance, generality and comprehensiveness.

Keywords: confidence, reliability, transduction

Citation: Matjaz Kukar: Reliability and typicalness for estimating confidence level of individual classifications . In R.López de Mántaras and L.Saitta (eds.): ECAI2004, Proceedings of the 16th European Conference on Artificial Intelligence, IOS Press, Amsterdam, 2004, pp.1043-1044.

[prev] [tofc] [next]

ECAI-2004 is organised by the European Coordinating Committee for Artificial Intelligence (ECCAI) and hosted by the Universitat Politècnica de València on behalf of Asociación Española de Inteligencia Artificial (AEPIA) and Associació Catalana d'Intel-ligència Artificial (ACIA).