2005 MIREX Contest Results - Audio Drum Detection (Contest wiki)

Goal: To detect the occurences of drum events in polyphonic audio.

Dataset: At least 50 files of both live and sequenced music, with many genres encompassed and various degrees of drum audio contained in the files. Three collections of music were used: Christian Dittmar (CD), Koen Tanghe (KT) and Masataka Goto (MG). Participants were evaluated against music from each individual collection, and then the three collection scores are averaged to produce a composite score.

OVERALL
Rank	Participant	Total Average Classification F-measure	Total Overall Onset Precision	Total Overall Onset Recall	Total Overall Onset F-measure	BD Average F-measure	HH Average F-measure	SD Average F-measure	Runtime (s)	Machine
1	Yoshii, Goto, & Okuno	0.670	64.92%	67.02%	0.659	0.728	0.574	0.702	8534	B 0
2	Tanghe, Degroeve, & De Baets 3	0.611	63.30%	71.19%	0.670	0.688	0.601	0.555	1337	Y
3	Tanghe, Degroeve, & De Baets 4	0.609	62.57%	71.09%	0.666	0.686	0.590	0.562	1342	Y
4	Tanghe, Degroeve, & De Baets 1	0.599	60.02%	72.45%	0.657	0.677	0.588	0.542	1350	Y
5	Dittmar, C.	0.588	65.68%	63.38%	0.645	0.606	0.585	0.581	673	R
6	Paulus, J.	0.499	59.61%	64.86%	0.621	0.527	0.587	0.430	1137	L
7	Gillet & Richard 2	0.443	77.09%	40.63%	0.532	0.598	0.334	0.428	21248	F
8	Gillet & Richard 1	0.391	69.84%	37.98%	0.492	0.533	0.343	0.317	21997	F

CHRISTIAN DITTMAR COLLECTION
Rank	Participant	Total Average Classification F-measure	Total Overall Onset Precision	Total Overall Onset Recall	Total Overall Onset F-measure	BD Average F-measure	HH Average F-measure	SD Average F-measure
1	Dittmar, C.	0.753	77.73%	72.56%	0.751	0.783	0.696	0.790
2	Yoshii, Goto, & Okuno	0.690	64.25%	62.75%	0.660	0.714	0.533	0.811
3	Tanghe, Degroeve, & De Baets 3	0.595	61.85%	64.85%	0.633	0.685	0.568	0.548
4	Tanghe, Degroeve, & De Baets 4	0.589	62.45%	64.22%	0.628	0.668	0.555	0.559
5	Tanghe, Degroeve, & De Baets 1	0.580	57.78%	65.94%	0.616	0.669	0.553	0.533
6	Paulus, J.	0.440	55.82%	54.36%	0.551	0.430	0.497	0.424
7	Gillet & Richard 2	0.401	77.22%	30.16%	0.434	0.658	0.156	0.364
8	Gillet & Richard 1	0.339	66.33%	30.57%	0.418	0.466	0.279	0.265

KOEN TANGHE COLLECTION
Rank	Participant	Total Average Classification F-measure	Total Overall Onset Precision	Total Overall Onset Recall	Total Overall Onset F-measure	BD Average F-measure	HH Average F-measure	SD Average F-measure
1	Yoshii, Goto, & Okuno	0.617	53.06%	66.30%	0.589	0.686	0.481	0.652
2	Tanghe, Degroeve, & De Baets 3	0.546	52.67%	68.21%	0.594	0.613	0.485	0.525
3	Tanghe, Degroeve, & De Baets 4	0.541	51.90%	67.98%	0.589	0.612	0.479	0.523
4	Tanghe, Degroeve, & De Baets 1	0.533	47.48%	69.24%	0.577	0.602	0.467	0.511
5	Dittmar, C.	0.494	51.20%	51.51%	0.514	0.509	0.418	0.535
6	Paulus, J.	0.425	55.82%	55.10%	0.555	0.444	0.489	0.412
7	Gillet & Richard 2	0.273	66.68%	28.22%	0.397	0.393	0.178	0.269
8	Gillet & Richard 1	0.259	58.48%	26.11%	0.361	0.375	0.196	0.210

MASATAKA GOTO COLLECTION (50 songs from RWC Music Database: RWC-MDB-P-2001)
Rank	Participant	Total Average Classification F-measure	Total Overall Onset Precision	Total Overall Onset Recall	Total Overall Onset F-measure	BD Average F-measure	HH Average F-measure	SD Average F-measure
1	Yoshii, Goto, & Okuno	0.716	76.13%	69.16%	0.725	0.776	0.661	0.710
2	Tanghe, Degroeve, & De Baets 4	0.685	72.32%	75.83%	0.740	0.766	0.691	0.599
3	Tanghe, Degroeve, & De Baets 3	0.683	72.74%	75.65%	0.742	0.763	0.701	0.585
4	Tanghe, Degroeve, & De Baets 1	0.673	69.86%	77.12%	0.733	0.753	0.693	0.574
5	Gillet & Richard 2	0.630	81.60%	53.00%	0.643	0.774	0.517	0.599
6	Dittmar, C.	0.617	71.37%	67.78%	0.695	0.631	0.675	0.544
7	Paulus, J.	0.597	62.90%	75.47%	0.686	0.648	0.695	0.449
8	Gillet & Richard 1	0.544	76.11%	48.80%	0.595	0.715	0.479	0.436

Maintained by :J Stephen Downie
Comments to : jdownie at uiuc dot edu

Last modified: 5 September 2005