Statistical Comparisons of Multiple Classifiers

Dechang Chen; Xiuzhen Cheng

Statistical Comparisons of Multiple Classifiers

Dechang Chen^*, Xiuzhen Cheng

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

This paper discusses the issue of comparing multiple classifiers, applied to the same test dataset of a classification problem. Assume that the output is 0 if a classifier correctly classifies a test feature point and the output is 1 otherwise. Then all the outputs from a given classifier constitute a sample of 0 and 1, and all the samples are correlated. From these dependent samples, we use Cochran 's Q statistic, as an overall test statistic, to detect whether or not the error rates of the classifiers are significantly different. When the null hypothesis that the error rates are equal is rejected, a thorough analysis of the nature of the error rates, such as the ranking of the error rates, is undertaken. For this purpose, we employ the Scheffé and Bonferroni multiple comparison procedures, based on dependent samples. We also use examples to demonstrate how to make these statistical comparisons.

Original language	English
Title of host publication	Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA 03
Editors	H.R. Arabnia, E.B. Kozerenko
Pages	97-101
Number of pages	5
State	Published - 2003
Externally published	Yes
Event	Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA'03 - Las Vegas, NV, United States Duration: 23 Jun 2003 → 26 Jun 2003

Publication series

Name	Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications

Conference

Conference	Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA'03
Country/Territory	United States
City	Las Vegas, NV
Period	23/06/03 → 26/06/03

Keywords

Cochran's Q statistic
Hypothesis testing
Multiple comparison procedure
Pattern recognition

Cite this

@inproceedings{248d87268187481b9698960378837266,

title = "Statistical Comparisons of Multiple Classifiers",

abstract = "This paper discusses the issue of comparing multiple classifiers, applied to the same test dataset of a classification problem. Assume that the output is 0 if a classifier correctly classifies a test feature point and the output is 1 otherwise. Then all the outputs from a given classifier constitute a sample of 0 and 1, and all the samples are correlated. From these dependent samples, we use Cochran 's Q statistic, as an overall test statistic, to detect whether or not the error rates of the classifiers are significantly different. When the null hypothesis that the error rates are equal is rejected, a thorough analysis of the nature of the error rates, such as the ranking of the error rates, is undertaken. For this purpose, we employ the Scheff{\'e} and Bonferroni multiple comparison procedures, based on dependent samples. We also use examples to demonstrate how to make these statistical comparisons.",

keywords = "Cochran's Q statistic, Hypothesis testing, Multiple comparison procedure, Pattern recognition",

author = "Dechang Chen and Xiuzhen Cheng",

year = "2003",

language = "English",

isbn = "1932415114",

series = "Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications",

pages = "97--101",

editor = "H.R. Arabnia and E.B. Kozerenko",

booktitle = "Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA 03",

note = "Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA'03 ; Conference date: 23-06-2003 Through 26-06-2003",

}

Chen, D & Cheng, X 2003, Statistical Comparisons of Multiple Classifiers. in HR Arabnia & EB Kozerenko (eds), Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA 03. Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, pp. 97-101, Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA'03, Las Vegas, NV, United States, 23/06/03.

Statistical Comparisons of Multiple Classifiers. / Chen, Dechang; Cheng, Xiuzhen.
Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA 03. ed. / H.R. Arabnia; E.B. Kozerenko. 2003. p. 97-101 (Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Statistical Comparisons of Multiple Classifiers

AU - Chen, Dechang

AU - Cheng, Xiuzhen

PY - 2003

Y1 - 2003

N2 - This paper discusses the issue of comparing multiple classifiers, applied to the same test dataset of a classification problem. Assume that the output is 0 if a classifier correctly classifies a test feature point and the output is 1 otherwise. Then all the outputs from a given classifier constitute a sample of 0 and 1, and all the samples are correlated. From these dependent samples, we use Cochran 's Q statistic, as an overall test statistic, to detect whether or not the error rates of the classifiers are significantly different. When the null hypothesis that the error rates are equal is rejected, a thorough analysis of the nature of the error rates, such as the ranking of the error rates, is undertaken. For this purpose, we employ the Scheffé and Bonferroni multiple comparison procedures, based on dependent samples. We also use examples to demonstrate how to make these statistical comparisons.

AB - This paper discusses the issue of comparing multiple classifiers, applied to the same test dataset of a classification problem. Assume that the output is 0 if a classifier correctly classifies a test feature point and the output is 1 otherwise. Then all the outputs from a given classifier constitute a sample of 0 and 1, and all the samples are correlated. From these dependent samples, we use Cochran 's Q statistic, as an overall test statistic, to detect whether or not the error rates of the classifiers are significantly different. When the null hypothesis that the error rates are equal is rejected, a thorough analysis of the nature of the error rates, such as the ranking of the error rates, is undertaken. For this purpose, we employ the Scheffé and Bonferroni multiple comparison procedures, based on dependent samples. We also use examples to demonstrate how to make these statistical comparisons.

KW - Cochran's Q statistic

KW - Hypothesis testing

KW - Multiple comparison procedure

KW - Pattern recognition

UR - http://www.scopus.com/inward/record.url?scp=1642415965&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:1642415965

SN - 1932415114

T3 - Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications

SP - 97

EP - 101

BT - Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA 03

A2 - Arabnia, H.R.

A2 - Kozerenko, E.B.

T2 - Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications, MLMTA'03

Y2 - 23 June 2003 through 26 June 2003

ER -

Statistical Comparisons of Multiple Classifiers

Abstract

Publication series

Conference

Keywords

Fingerprint

Cite this