ECAI 2004 Conference Paper

[full paper]

Avoiding data overfitting in scientific discovery: Experiments in functional genomics

Dragan Gamberger, Nada Lavrac

Functional genomics is a typical scientific discovery domain characterized by a very large number of attributes (genes) relative to the number of examples (observations). The danger of data overfitting is crucial in such domains. This work presents an approach which can help in avoiding data overfitting in supervised inductive learning of short rules that are appropriate for human interpretation. The approach is based on the subgroup discovery rule learning framework, enhanced by methods of restricting the hypothesis search space by exploiting the relevance of features that enter the rule construction process as well as their combinations that form the rules. A multi-class functional genomics problem of classifying fourteen cancer types based on more than 16000 gene expression values is used to illustrate the methodology.

Keywords: scientific discovery, feature relevancy, learning from small datasets, functional genomics

Citation: Dragan Gamberger, Nada Lavrac: Avoiding data overfitting in scientific discovery: Experiments in functional genomics. In R.López de Mántaras and L.Saitta (eds.): ECAI2004, Proceedings of the 16th European Conference on Artificial Intelligence, IOS Press, Amsterdam, 2004, pp.470-474.