University of Illinois · Graduate School of Library and Information Science · ISRL

2005 MIREX Contest Results - Audio Onset Detection (Contest wiki)


Goal: The evaluation and comparison of onset detection algorithms applied to audio music recordings

Dataset: 85 audio files (14.8 minutes total) from 9 classes: complex, poly pitched, solo bars and bells, solo brass, solo drum, solo plucked strings, solo singing voice, solo sustained strings, solo winds


OVERALL
Rank Participant Overall Average F-measure Overall Average Precision Overall Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
Mean Absolute Distance
Mean Distance
Runtime (s) Machine
1 Lacoste & Eck 2
80.07%
79.27%
83.70%
7974
1776
1525
210
53
26.82
6.05
4.85
0.65
0.17
0.0115
0.00613
4713
G
2 Lacoste & Eck 1 78.35%
77.69%
83.27%
7884
2317
1615
202
60
26.55
7.91
5.12
0.62
0.19
0.0115
0.00572
1022 G
3 Ricard, J.
74.80%
81.36%
73.70%
7099
1581
2400
200
3
23.97
5.18
7.70
0.63
0.01
0.0138
0.00593
154
R
4 Brossier, P.
74.72%
74.07%
81.95%
7699
3101
1800
204
54
25.81
10.71
5.86
0.62
0.15
0.0111
-0.00384
50 B0
5 Röbel, A. 2 74.64%
83.93%
71.00%
6555
1026
2944
163
169
22.62
3.46
9.05
0.51
0.52
0.0084
0.00380
159 L
6 Collins, N. 72.10%
87.96%
68.26%
6174
629
3325
168
35
21.27
2.13
10.40
0.52
0.12
0.0069
-0.00120
12
I
7 Röbel, A. 1
69.57%
79.16%
68.60%
6215
1419
3284
154
253
21.40
5.05
10.27
0.48
0.88
0.0087
0.00525
158 L
8 Pertusa, Klapuri, & Iñesta
58.92%
60.01%
61.62%
5704
4548
3795
217
61
19.41
15.08
12.25
0.73
0.18
0.0276
-0.02209
56 F
9
West, K.
48.77%
48.50%
56.29%
5424
7119
4075
146
0
18.46
24.05
13.21
0.46
0.0
0.0138
-0.00499
179
F


Note: Overall Average F-Measure, Overall Average Precision and Overall Average Recall are weighted by number of files in each class;
Total
Correct, Total False Positive, Total False Negative, Total Merged, and Total Doubled are over ground-truths and classes;
Average Correct, Average False Positive, Average False Negative, Average Merged, and Average Doubled are averaged over ground-truths, then weighted average across classes;


For the following tables: Average F-Measure, Average Precision and Average Recall are accross all Ground-truths and Files in each class;
Total
Correct, Total False Positive, Total False Negative, Total Merged, and Total Doubled are accross all Ground-truths and Files in each class;
Average Correct, Average False Positive, Average False Negative, Average Merged, and Average Doubled are accross all Ground-truths, total over Files in each class;

COMPLEX (15 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Lacoste & Eck 2
78.85%
81.37%
77.82%
2836
584
723
109
23
567.20
116.80
144.60
21.80
4.60
2 Lacoste & Eck 1 77.02%
78.92%
77.46%
2786
749
773
107
32
557.20
149.80
154.60
21.40
6.40
3 Brossier, P. 76.16%
77.55%
77.89%
2793
927
766
115
42
558.60
185.40
153.20
23.00
8.40
4 Ricard, J. 71.90%
77.51%
68.20%
2465
650
1094
97
3
493.60
130.00
218.80
19.40
0.60
5 Röbel, A. 2 62.84%
82.39%
53.58%
1968
357
1591
85
92
393.60
71.40
318.20
17.00
18.40
6 Collins, N. 60.25%
86.14%
51.77%
1878
212
1681
87
13
375.60
42.40
336.20
17.40
2.60
7 Röbel, A. 1
59.76%
79.58%
52.66%
1897
328
1662
80
70
379.40
65.60
332.40
16.00
14.00
8 Pertusa, Klapuri, & Iñesta 50.16%
51.69%
51.22%
1884
1756
1675
80
40
376.80
351.20
355.30
16.00
8.00
9
West, K.
47.13%
47.45%
51.52%
1794
2466
1765
72
0
358.80
493.20
353.00
14.40
0.00



POLY-PITCHED (10 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Lacoste & Eck 2
86.31%
86.71%
88.73%
764
100
95
26
2
254.67
33.33
31.67
8.67
0.67
2 Lacoste & Eck 1 85.93%
85.65%
88.94%
764
115
95
28
5
254.67
38.33
31.67
9.33
1.67
3 Ricard, J. 83.26%
90.12%
80.11%
677
82
182
27
0
225.67
27.33
60.67
9.00
0.00
4 Brossier, P. 80.88%
76.95%
89.41%
759
192
100
27
4
253.00
64.00
33.33
9.00
1.33
5 Röbel, A. 2 76.24%
88.93%
69.02%
593
58
266
19
4
197.67
19.33
88.67
6.33
1.33
6 Collins, N. 75.70%
89.95%
69.98%
570
54
289
19
0
190.00
18.00
96.33
6.33
0.00
7 Röbel, A. 1
69.29%
75.00%
72.87%
616
281
243
19
23
205.33
93.67
81.00
6.33
7.67
8 Pertusa, Klapuri, & Iñesta 59.37%
60.03%
58.30%
479
328
380
15
3
159.67
109.33
126.67
5.00
1.00
9
West, K.
39.98%
34.71%
54.01%
483
906
376
18
0
161.00
302.00
125.33
6.00
0.00


SOLO-BARS-AND-BELLS (4 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Collins, N.
99.28%
98.91%
99.67%
321
3
3
0
0
107.00
1.00
1.00
0.00
0.00
2 Röbel, A. 1 97.92%
96.15%
100.00%
324
12
0
0
0
108.00
4.00
0.00
0.00
0.00
3 Röbel, A. 2. 90.34%
100.00%
84.07%
297
0
27
0
0
99.00
0.00
9.00
0.00
0.00
4 Ricard, J.
87.17%
81.79%
97.00%
297
18
27
6
0
99.00
6.00
9.00
2.00
0.00
5 Lacoste & Eck 2
86.55%
82.66%
98.33%
309
42
15
3
0
103.00
14.00
5.00
1.00
0.00
6 Lacoste & Eck 1 86.37%
81.67%
99.00%
315
45
9
0
0
105.00
15.00
3.00
0.00
0.00
7 Brossier, P. 73.97%
66.08%
99.33%
318
198
6
1
0
106.00
66.00
2.00
0.33
0.00
8 Pertusa, Klapuri, & Iñesta 60.22%
60.98%
68.39%
231
78
93
18
0
77.00
26.00
31.00
6.00
0.00
9
West, K.
34.58%
36.49%
51.89%
189
324
135
1
0
63.00
108.00
45.00
0.33
0.00


SOLO-BRASS (2 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Ricard, J.
72.66%
74.24%
71.51%
184
17
29
0
0
61.33
5.67
9.67
0.00
0.00
2 Lacoste & Eck 2
70.25%
66.80%
76.49%
192
36
21
0
1
64.00
12.00
7.00
0.00
0.33
3 Collins, N. 69.09%
71.71%
67.26%
170
40
43
0
8
56.67
13.33
14.33
0.00
2.67
4 Röbel, A. 2. 68.32%
66.15%
70.93%
189
39
24
0
3
3.00
13.00
8.00
0.00
1.00
5 Lacoste & Eck 1 67.88%
63.55%
77.77%
193
50
20
0
3
64.33
16.67
6.67
0.00
1.00
6 Brossier, P. 64.88%
57.15%
79.55%
196
77
17
0
0
65.33
35.67
5.67
0.00
0.00
7 Röbel, A. 1 61.87%
58.33%
68.20%
162
66
51
0
6
54.00
22.00
17.00
0.00
2.00
8 Pertusa, Klapuri, & Iñesta 54.41%
51.46%
59.38%
120
96
93
0
1
40.00
32.00
31.00
0.00
0.33
9
West, K.
33.94%
31.97%
42.96%
106
182
107
2
0
35.33
60.67
35.67
0.67
0.00


SOLO-DRUM (30 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Collins, N.
92.31%
95.92%
90.28%
2668
86
240
51
3
889.33
28.67
80.00
17.00
1.00
2 Lacoste & Eck 2
91.40%
92.25%
91.76%
2706
186
202
56
15
902.00
62.00
67.33
16.67
5.00
3 Ricard, J. 90.97%
96.46%
87.55%
2590
74
318
54
0
863.33
24.67
106.00
18.00
0.00
4 Röbel, A. 2. 89.96%
92.17%
89.06%
2668
188
240
49
27
889.33
62.67
80.00
16.33
9.00
5 Lacoste & Eck 1 89.91%
92.79%
88.53%
2634
174
274
55
0
878.00
58.00
91.33
18.33
0.00
6 Röbel, A. 1 86.29%
88.84%
86.44%
2565
339
343
46
76
855.00
113.00
114.33
15.33
25.33
7 Brossier, P. 86.28%
90.90%
85.79%
2452
215
456
48
2
817.33
71.67
152.00
16.00
0.67
8 Pertusa, Klapuri, & Iñesta 77.22%
82.24%
74.52%
2167
362
741
93
1
722.33
120.67
247.00
31.00
0.33
9
West, K.
71.61%
75.59%
72.16%
2162
724
746
41
0
720.67
241.33
248.67
13.67
0.00



SOLO-PLUCKED-STRING (9 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Röbel, A. 2
84.20%
85.82%
82.99%
350
58
81
5
12
1116.67
19.33
27.00
1.67
4.00
2 Lacoste & Eck 1
83.49%
77.82%
93.27%
396
156
35
5
2
132.00
52.00
11.67
1.67
0.67
3 Collins, N. 81.97%
77.78%
88.09%
380
136
51
7
9
126.67
45.33
17.00
2.33
3.00
4 Lacoste & Eck 2 81.04%
75.23%
90.20%
387
153
44
10
2
129.00
51.00
14.67
3.33
0.67
5 Brossier, P. 79.99%
73.93%
90.97%
384
123
47
5
1
128.00
41.00
15.67
1.67
0.33
6 Ricard, J.
77.85%
88.33%
73.71%
330
42
101
6
0
110.00
14.00
33.67
2.00
0.00
7 Röbel, A. 1
77.58%
75.21%
83.14%
361
197
70
6
63
120.33
65.67
23.33
2.00
21.00
8 Pertusa, Klapuri, & Iñesta 67.74%
75.63%
71.68%
317
172
114
5
2
105.67
57.33
38.00
1.67
0.67
9
West, K.
39.85%
35.08%
57.54%
252
495
179
3
0
84.00
165.0
59.67
1.00
0.00



SOLO-SINGING-VOICE (5 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Lacoste & Eck 2
45.33%
35.83%
64.46%
143
286
86
1
5
47.67
95.33
28.67
0.33
1.67
2 Röbel, A. 1
42.69%
52.76%
36.85%
81
78
148
0
2
27.00
26.00
49.33
0.00
0.67
3 Röbel, A. 2 40.68%
39.85%
42.14%
96
153
133
1
6
32.00
51.00
44.33
0.33
2.00
4 Lacoste & Eck 1 34.35%
23.14%
68.49%
152
529
77
1
0
50.67
176.33
25.67
0.33
0.00
5 Collins, N.
29.34%
59.44%
19.85%
44
28
185
1
0
14.67
9.33
61.67
0.33
0.00
6 Ricard, J.
27.59%
20.71%
45.22%
98
391
131
1
0
32.67
130.33
43.67
0.33
0.00
7 Brossier, P. 22.16%
13.73%
57.74%
130
848
99
1
1
43.33
282.67
33.00
0.33
0.33
8 West, K.
12.07%
7.84%
26.74%
62
778
167
1
0
20.67
259.33
55.67
0.33
0.00
9
Pertusa, Klapuri, & Iñesta 11.12%
6.86%
31.12%
66
885
163
0
2
22.00
295.00
54.33
0.00
0.67



SOLO-SUSTAINED-STRINGS (6 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Brossier, P.
57.92%
57.33%
64.34%
504
249
206
6
3
168.00
83.00
68.67
2.00
1.00
2 Lacoste & Eck 2
56.68%
57.49%
59.70%
473
208
237
5
5
157.67
69.33
79.00
1.67
1.67
3 Lacoste & Eck 1 52.87%
52.44%
58.63%
460
278
247
5
16
154.33
92.67
82.33
1.67
5.33
4 Pertusa, Klapuri, & Iñesta 39.79%
36.47%
53.01%
365
598
345
5
9
121.67
199.33
115.00
1.67
3.00
5 Ricard, J.
38.45%
71.33%
32.63%
305
100
405
7
0
101.67
33.33
135.00
2.33
0.00
6 Röbel, A. 2 36.18%
79.12%
26.95%
201
39
509
3
0
67.00
13.00
169.67
1.00
0.00
7 West, K. 32.12%
29.27%
41.41%
312
696
398
6
0
104.00
232.00
132.67
2.00
0.00
8 Röbel, A. 1 17.35%
73.87%
10.64%
89
13
621
2
0
29.67
4.33
207.00
0.67
0.00
9
Collins, N. 14.74%
90.74%
8.47%
47
7
663
2
0
15.67
2.33
221.00
0.67
0.00



SOLO-WIND (4 files)
Rank Participant Average F-measure Average Precision Average Recall Total Correct Total False Positives Total False Negatives Total Merged Total Doubled
Average Correct
Average False Positives
Average False Negatives
Average Merged
Average Doubled
1 Röbel, A. 2
66.01%
66.38%
67.93%
193
134
73
1
25
64.33
44.67
24.33
0.33
8.33
2 Lacoste & Eck 2
58.75%
54.38%
67.18%
164
181
102
0
0
54.67
60.33
34.00
0.00
0.00
3 Lacoste & Eck 1 56.48%
48.77%
71.54%
181
221
85
1
2
60.33
73.67
28.33
0.33
0.67
4 Brossier, P.
52.08%
44.83%
69.87%
163
272
103
1
1
54.33
90.67
34.33
0.33
0.33
5 Röbel, A. 1
51.13%
58.72%
46.59%
120
105
146
1
13
40.00
35.00
48.67
0.33
4.33
6 Collins, N. 47.57%
81.71%
35.40%
96
63
170
1
2
32.00
21.00
56.67
0.33
0.67
7 Ricard, J.
38.57%
38.98%
49.38%
153
207
113
2
0
51.00
69.00
37.67
0.67
0.00
8 Pertusa, Klapuri, & Iñesta 25.59%
20.62%
34.91%
75
273
191
1
3
25.00
91.00
63.67
0.33
1.00
9
West, K. 18.11%
13.91%
28.35%
64
548
202
2
0
21.33
182.67
67.33
0.67
0.00



Maintained by :J Stephen Downie
Comments to : jdownie at uiuc dot edu
Last modified: 28 September 2005