2005 MIREX Contest Results - Audio Onset Detection (Contest wiki)

Goal: The evaluation and comparison of onset detection algorithms applied to audio music recordings

Dataset: 85 audio files (14.8 minutes total) from 9 classes: complex, poly pitched, solo bars and bells, solo brass, solo drum, solo plucked strings, solo singing voice, solo sustained strings, solo winds

OVERALL
Rank	Participant	Overall Average F-measure	Overall Average Precision	Overall Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled	Mean Absolute Distance	Mean Distance	Runtime (s)	Machine
1	Lacoste & Eck 2	80.07%	79.27%	83.70%	7974	1776	1525	210	53	26.82	6.05	4.85	0.65	0.17	0.0115	0.00613	4713	G
2	Lacoste & Eck 1	78.35%	77.69%	83.27%	7884	2317	1615	202	60	26.55	7.91	5.12	0.62	0.19	0.0115	0.00572	1022	G
3	Ricard, J.	74.80%	81.36%	73.70%	7099	1581	2400	200	3	23.97	5.18	7.70	0.63	0.01	0.0138	0.00593	154	R
4	Brossier, P.	74.72%	74.07%	81.95%	7699	3101	1800	204	54	25.81	10.71	5.86	0.62	0.15	0.0111	-0.00384	50	B0
5	Röbel, A. 2	74.64%	83.93%	71.00%	6555	1026	2944	163	169	22.62	3.46	9.05	0.51	0.52	0.0084	0.00380	159	L
6	Collins, N.	72.10%	87.96%	68.26%	6174	629	3325	168	35	21.27	2.13	10.40	0.52	0.12	0.0069	-0.00120	12	I
7	Röbel, A. 1	69.57%	79.16%	68.60%	6215	1419	3284	154	253	21.40	5.05	10.27	0.48	0.88	0.0087	0.00525	158	L
8	Pertusa, Klapuri, & Iñesta	58.92%	60.01%	61.62%	5704	4548	3795	217	61	19.41	15.08	12.25	0.73	0.18	0.0276	-0.02209	56	F
9	West, K.	48.77%	48.50%	56.29%	5424	7119	4075	146	0	18.46	24.05	13.21	0.46	0.0	0.0138	-0.00499	179	F

Note: Overall Average F-Measure, Overall Average Precision and Overall Average Recall are weighted by number of files in each class;
Total Correct, Total False Positive, Total False Negative, Total Merged, and Total Doubled are over ground-truths and classes;
Average Correct, Average False Positive, Average False Negative, Average Merged, and Average Doubled are averaged over ground-truths, then weighted average across classes;

For the following tables: Average F-Measure, Average Precision and Average Recall are accross all Ground-truths and Files in each class;
Total Correct, Total False Positive, Total False Negative, Total Merged, and Total Doubled are accross all Ground-truths and Files in each class;
Average Correct, Average False Positive, Average False Negative, Average Merged, and Average Doubled are accross all Ground-truths, total over Files in each class;

COMPLEX (15 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Lacoste & Eck 2	78.85%	81.37%	77.82%	2836	584	723	109	23	567.20	116.80	144.60	21.80	4.60
2	Lacoste & Eck 1	77.02%	78.92%	77.46%	2786	749	773	107	32	557.20	149.80	154.60	21.40	6.40
3	Brossier, P.	76.16%	77.55%	77.89%	2793	927	766	115	42	558.60	185.40	153.20	23.00	8.40
4	Ricard, J.	71.90%	77.51%	68.20%	2465	650	1094	97	3	493.60	130.00	218.80	19.40	0.60
5	Röbel, A. 2	62.84%	82.39%	53.58%	1968	357	1591	85	92	393.60	71.40	318.20	17.00	18.40
6	Collins, N.	60.25%	86.14%	51.77%	1878	212	1681	87	13	375.60	42.40	336.20	17.40	2.60
7	Röbel, A. 1	59.76%	79.58%	52.66%	1897	328	1662	80	70	379.40	65.60	332.40	16.00	14.00
8	Pertusa, Klapuri, & Iñesta	50.16%	51.69%	51.22%	1884	1756	1675	80	40	376.80	351.20	355.30	16.00	8.00
9	West, K.	47.13%	47.45%	51.52%	1794	2466	1765	72	0	358.80	493.20	353.00	14.40	0.00

POLY-PITCHED (10 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Lacoste & Eck 2	86.31%	86.71%	88.73%	764	100	95	26	2	254.67	33.33	31.67	8.67	0.67
2	Lacoste & Eck 1	85.93%	85.65%	88.94%	764	115	95	28	5	254.67	38.33	31.67	9.33	1.67
3	Ricard, J.	83.26%	90.12%	80.11%	677	82	182	27	0	225.67	27.33	60.67	9.00	0.00
4	Brossier, P.	80.88%	76.95%	89.41%	759	192	100	27	4	253.00	64.00	33.33	9.00	1.33
5	Röbel, A. 2	76.24%	88.93%	69.02%	593	58	266	19	4	197.67	19.33	88.67	6.33	1.33
6	Collins, N.	75.70%	89.95%	69.98%	570	54	289	19	0	190.00	18.00	96.33	6.33	0.00
7	Röbel, A. 1	69.29%	75.00%	72.87%	616	281	243	19	23	205.33	93.67	81.00	6.33	7.67
8	Pertusa, Klapuri, & Iñesta	59.37%	60.03%	58.30%	479	328	380	15	3	159.67	109.33	126.67	5.00	1.00
9	West, K.	39.98%	34.71%	54.01%	483	906	376	18	0	161.00	302.00	125.33	6.00	0.00

SOLO-BARS-AND-BELLS (4 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Collins, N.	99.28%	98.91%	99.67%	321	3	3	0	0	107.00	1.00	1.00	0.00	0.00
2	Röbel, A. 1	97.92%	96.15%	100.00%	324	12	0	0	0	108.00	4.00	0.00	0.00	0.00
3	Röbel, A. 2.	90.34%	100.00%	84.07%	297	0	27	0	0	99.00	0.00	9.00	0.00	0.00
4	Ricard, J.	87.17%	81.79%	97.00%	297	18	27	6	0	99.00	6.00	9.00	2.00	0.00
5	Lacoste & Eck 2	86.55%	82.66%	98.33%	309	42	15	3	0	103.00	14.00	5.00	1.00	0.00
6	Lacoste & Eck 1	86.37%	81.67%	99.00%	315	45	9	0	0	105.00	15.00	3.00	0.00	0.00
7	Brossier, P.	73.97%	66.08%	99.33%	318	198	6	1	0	106.00	66.00	2.00	0.33	0.00
8	Pertusa, Klapuri, & Iñesta	60.22%	60.98%	68.39%	231	78	93	18	0	77.00	26.00	31.00	6.00	0.00
9	West, K.	34.58%	36.49%	51.89%	189	324	135	1	0	63.00	108.00	45.00	0.33	0.00

SOLO-BRASS (2 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Ricard, J.	72.66%	74.24%	71.51%	184	17	29	0	0	61.33	5.67	9.67	0.00	0.00
2	Lacoste & Eck 2	70.25%	66.80%	76.49%	192	36	21	0	1	64.00	12.00	7.00	0.00	0.33
3	Collins, N.	69.09%	71.71%	67.26%	170	40	43	0	8	56.67	13.33	14.33	0.00	2.67
4	Röbel, A. 2.	68.32%	66.15%	70.93%	189	39	24	0	3	3.00	13.00	8.00	0.00	1.00
5	Lacoste & Eck 1	67.88%	63.55%	77.77%	193	50	20	0	3	64.33	16.67	6.67	0.00	1.00
6	Brossier, P.	64.88%	57.15%	79.55%	196	77	17	0	0	65.33	35.67	5.67	0.00	0.00
7	Röbel, A. 1	61.87%	58.33%	68.20%	162	66	51	0	6	54.00	22.00	17.00	0.00	2.00
8	Pertusa, Klapuri, & Iñesta	54.41%	51.46%	59.38%	120	96	93	0	1	40.00	32.00	31.00	0.00	0.33
9	West, K.	33.94%	31.97%	42.96%	106	182	107	2	0	35.33	60.67	35.67	0.67	0.00

SOLO-DRUM (30 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Collins, N.	92.31%	95.92%	90.28%	2668	86	240	51	3	889.33	28.67	80.00	17.00	1.00
2	Lacoste & Eck 2	91.40%	92.25%	91.76%	2706	186	202	56	15	902.00	62.00	67.33	16.67	5.00
3	Ricard, J.	90.97%	96.46%	87.55%	2590	74	318	54	0	863.33	24.67	106.00	18.00	0.00
4	Röbel, A. 2.	89.96%	92.17%	89.06%	2668	188	240	49	27	889.33	62.67	80.00	16.33	9.00
5	Lacoste & Eck 1	89.91%	92.79%	88.53%	2634	174	274	55	0	878.00	58.00	91.33	18.33	0.00
6	Röbel, A. 1	86.29%	88.84%	86.44%	2565	339	343	46	76	855.00	113.00	114.33	15.33	25.33
7	Brossier, P.	86.28%	90.90%	85.79%	2452	215	456	48	2	817.33	71.67	152.00	16.00	0.67
8	Pertusa, Klapuri, & Iñesta	77.22%	82.24%	74.52%	2167	362	741	93	1	722.33	120.67	247.00	31.00	0.33
9	West, K.	71.61%	75.59%	72.16%	2162	724	746	41	0	720.67	241.33	248.67	13.67	0.00

SOLO-PLUCKED-STRING (9 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Röbel, A. 2	84.20%	85.82%	82.99%	350	58	81	5	12	1116.67	19.33	27.00	1.67	4.00
2	Lacoste & Eck 1	83.49%	77.82%	93.27%	396	156	35	5	2	132.00	52.00	11.67	1.67	0.67
3	Collins, N.	81.97%	77.78%	88.09%	380	136	51	7	9	126.67	45.33	17.00	2.33	3.00
4	Lacoste & Eck 2	81.04%	75.23%	90.20%	387	153	44	10	2	129.00	51.00	14.67	3.33	0.67
5	Brossier, P.	79.99%	73.93%	90.97%	384	123	47	5	1	128.00	41.00	15.67	1.67	0.33
6	Ricard, J.	77.85%	88.33%	73.71%	330	42	101	6	0	110.00	14.00	33.67	2.00	0.00
7	Röbel, A. 1	77.58%	75.21%	83.14%	361	197	70	6	63	120.33	65.67	23.33	2.00	21.00
8	Pertusa, Klapuri, & Iñesta	67.74%	75.63%	71.68%	317	172	114	5	2	105.67	57.33	38.00	1.67	0.67
9	West, K.	39.85%	35.08%	57.54%	252	495	179	3	0	84.00	165.0	59.67	1.00	0.00

SOLO-SINGING-VOICE (5 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Lacoste & Eck 2	45.33%	35.83%	64.46%	143	286	86	1	5	47.67	95.33	28.67	0.33	1.67
2	Röbel, A. 1	42.69%	52.76%	36.85%	81	78	148	0	2	27.00	26.00	49.33	0.00	0.67
3	Röbel, A. 2	40.68%	39.85%	42.14%	96	153	133	1	6	32.00	51.00	44.33	0.33	2.00
4	Lacoste & Eck 1	34.35%	23.14%	68.49%	152	529	77	1	0	50.67	176.33	25.67	0.33	0.00
5	Collins, N.	29.34%	59.44%	19.85%	44	28	185	1	0	14.67	9.33	61.67	0.33	0.00
6	Ricard, J.	27.59%	20.71%	45.22%	98	391	131	1	0	32.67	130.33	43.67	0.33	0.00
7	Brossier, P.	22.16%	13.73%	57.74%	130	848	99	1	1	43.33	282.67	33.00	0.33	0.33
8	West, K.	12.07%	7.84%	26.74%	62	778	167	1	0	20.67	259.33	55.67	0.33	0.00
9	Pertusa, Klapuri, & Iñesta	11.12%	6.86%	31.12%	66	885	163	0	2	22.00	295.00	54.33	0.00	0.67

SOLO-SUSTAINED-STRINGS (6 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Brossier, P.	57.92%	57.33%	64.34%	504	249	206	6	3	168.00	83.00	68.67	2.00	1.00
2	Lacoste & Eck 2	56.68%	57.49%	59.70%	473	208	237	5	5	157.67	69.33	79.00	1.67	1.67
3	Lacoste & Eck 1	52.87%	52.44%	58.63%	460	278	247	5	16	154.33	92.67	82.33	1.67	5.33
4	Pertusa, Klapuri, & Iñesta	39.79%	36.47%	53.01%	365	598	345	5	9	121.67	199.33	115.00	1.67	3.00
5	Ricard, J.	38.45%	71.33%	32.63%	305	100	405	7	0	101.67	33.33	135.00	2.33	0.00
6	Röbel, A. 2	36.18%	79.12%	26.95%	201	39	509	3	0	67.00	13.00	169.67	1.00	0.00
7	West, K.	32.12%	29.27%	41.41%	312	696	398	6	0	104.00	232.00	132.67	2.00	0.00
8	Röbel, A. 1	17.35%	73.87%	10.64%	89	13	621	2	0	29.67	4.33	207.00	0.67	0.00
9	Collins, N.	14.74%	90.74%	8.47%	47	7	663	2	0	15.67	2.33	221.00	0.67	0.00

SOLO-WIND (4 files)
Rank	Participant	Average F-measure	Average Precision	Average Recall	Total Correct	Total False Positives	Total False Negatives	Total Merged	Total Doubled	Average Correct	Average False Positives	Average False Negatives	Average Merged	Average Doubled
1	Röbel, A. 2	66.01%	66.38%	67.93%	193	134	73	1	25	64.33	44.67	24.33	0.33	8.33
2	Lacoste & Eck 2	58.75%	54.38%	67.18%	164	181	102	0	0	54.67	60.33	34.00	0.00	0.00
3	Lacoste & Eck 1	56.48%	48.77%	71.54%	181	221	85	1	2	60.33	73.67	28.33	0.33	0.67
4	Brossier, P.	52.08%	44.83%	69.87%	163	272	103	1	1	54.33	90.67	34.33	0.33	0.33
5	Röbel, A. 1	51.13%	58.72%	46.59%	120	105	146	1	13	40.00	35.00	48.67	0.33	4.33
6	Collins, N.	47.57%	81.71%	35.40%	96	63	170	1	2	32.00	21.00	56.67	0.33	0.67
7	Ricard, J.	38.57%	38.98%	49.38%	153	207	113	2	0	51.00	69.00	37.67	0.67	0.00
8	Pertusa, Klapuri, & Iñesta	25.59%	20.62%	34.91%	75	273	191	1	3	25.00	91.00	63.67	0.33	1.00
9	West, K.	18.11%	13.91%	28.35%	64	548	202	2	0	21.33	182.67	67.33	0.67	0.00

Maintained by :J Stephen Downie
Comments to : jdownie at uiuc dot edu

Last modified: 28 September 2005