2005 MIREX Contest Results - Symbolic Genre Classification (Contest wiki)

Goal: To classify MIDI recordings into genre categories.

Dataset: Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold crossvalidated with each algorithm tested using identical training and testing data splits.

OVERALL
Rank	Participant	Mean Hierarchical Classification Accuracy	Mean Raw Classification Accuracy
1	McKay & Fujinaga	77.17%	65.28%
2	Basili, Serafini, & Stellato (NB)	72.08%	58.53%
3	Li, M.	67.57%	55.90%
4	Basili, Serafini, & Stellato (J48)	67.14%	53.14%
5	Ponce de Leon & Inesta	37.76%	26.52%

38 Classes
Rank	Participant	Hierarchical Classification Accuracy	Hierarchical Classification Accuracy Std	Raw Classification Accuracy	Raw Classification Accuracy Std	Runtime (s)	Machine	Confusion Matrix Files
1	McKay & Fujinaga	64.33%	1.04	46.11%	1.51	3 days	R	MF_38eval.txt
2	Basili, Serafini, & Stellato (NB)	62.60%	0.26	45.05%	0.55	N/A	N/A	BST_NB_38eval.txt
3	Basili, Serafini, & Stellato (J48)	57.61%	1.14	40.95%	1.35	N/A	N/A	BST_J48_38eval.txt
4	Li, M.	54.91%	0.66	39.79%	0.87	15,948	G	L_38eval.txt
5	Ponce de Leon & Inesta	24.84%	1.40	15.26%	1.13	821	L	PI_38eval.txt

9 Classes
Rank	Participant	Hierarchical Classification Accuracy	Hierarchical Classification Accuracy Std	Raw Classification Accuracy	Raw Classification Accuracy Std	Runtime (s)	Machine	Confusion Matrix Files
1	McKay & Fujinaga	90.00%	0.60	84.44%	1.41	18,375	R	MF_9eval.txt
2	Basili, Serafini, & Stellato (NB)	81.56%	0.76	72.00%	0.88	N/A	N/A	BST_NB_9eval.txt
3	Li, M.	80.22%	1.47	72.00%	2.31	3,777	G	L_9eval.txt
4	Basili, Serafini, & Stellato (J48)	76.67%	1.11	65.33%	1.65	N/A	N/A	BST_J48_9eval.txt
5	Ponce de Leon & Inesta	50.67%	1.26	37.78%	2.30	197	L	PI_9eval.txt

Maintained by :J Stephen Downie
Comments to : jdownie at uiuc dot edu

Last modified: 28 September 2005