Difference between revisions of "2007:Audio Onset Detection Results"

From MIREX Wiki
(→‎Introduction)
(→‎Individual Results)
 
(17 intermediate revisions by 7 users not shown)
Line 1: Line 1:
−
[[Category: Results]]
 
 
==Introduction==
 
==Introduction==
−
These are the results for the 2007 running of the Audio Onset Detection task set. For background information about this task set please refer to the [[Audio Onset Detection]] page.
+
These are the results for the 2007 running of the Audio Onset Detection task set. For background information about this task set please refer to the [[2007:Audio Onset Detection]] page.
  
 
The aim of the Audio Onset Detection task is to find the time locations at which all musical events in a recording begin. The dataset consists of 85 recordings across 9 different "classes" (e.g. solo drums, polyphonic pitched, etc.). For each sound file, ground truth annotations produced by 3-5 listeners were used for the evaluation. Each algorithm was tested across 10-20 different parameterizations (e.g. thresholds) in order to produce Precision vs. Recall Operating Characteristic (P-ROC) curves. The primary evauluation metric used was the F1-Measure (the equal weighted harmonic mean of precision and recall).  
 
The aim of the Audio Onset Detection task is to find the time locations at which all musical events in a recording begin. The dataset consists of 85 recordings across 9 different "classes" (e.g. solo drums, polyphonic pitched, etc.). For each sound file, ground truth annotations produced by 3-5 listeners were used for the evaluation. Each algorithm was tested across 10-20 different parameterizations (e.g. thresholds) in order to produce Precision vs. Recall Operating Characteristic (P-ROC) curves. The primary evauluation metric used was the F1-Measure (the equal weighted harmonic mean of precision and recall).  
Line 10: Line 9:
 
====Team ID====
 
====Team ID====
  
−
'''lacoste''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/OD_dixon.pdf Simon Dixon]<br />
+
'''lacoste''' = [https://www.music-ir.org/mirex/abstracts/2007/OD_lacoste.pdf Alexandre Lacoste]<br />
−
'''lee''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/OD_du.pdf Yunfeng Du, Ming Li, Jian Liu]<br />
+
'''lee''' = [https://www.music-ir.org/mirex/abstracts/2007/OD_lee.pdf Wan-Chi Lee, Yu Shiu, C.-C. Jay Kuo]<br />
−
'''roebel''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/OD_roebel.pdf A. R├╢bel]<br />
+
'''roebel''' = [https://www.music-ir.org/mirex/abstracts/2007/OD_roebel.pdf A. Röbel]<br />
−
'''stowell''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AME_BT_OD_TE_brossier.pdf Paul Brossier]<br />
+
'''stowell''' = [https://www.music-ir.org/mirex/abstracts/2007/OD_stowell.pdf Dan Stowell, Mark Plumbley]<br />
−
'''zhou''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AME_BT_OD_TE_brossier.pdf Paul Brossier]<br />
+
'''zhou''' = [https://www.music-ir.org/mirex/abstracts/2007/OD_zhou.pdf Ruohua Zhou, Joshua D. Reiss]<br />
  
 
==Overall Summary Results==
 
==Overall Summary Results==
−
===MIREX 2006 Audio Onset Detection Summary Results - Peak F-measure performance across all parameterizations===
+
===MIREX 2007 Audio Onset Detection Summary Results - Peak F-measure performance across all parameterizations===
  
−
<csv>onset06_sum.csv</csv>  
+
<csv>2007/onset.Total_peak.csv</csv>
  
−
===MIREX 2006 Audio Onset Detection Summary Plot===
+
===MIREX 2007 Audio Onset Detection Summary Plot===
  
−
[[image:onset06_summary.png]]
+
[[image:Total.png]]
  
−
===MIREX 2006 Audio Onset Detection Runtime Data===
+
===MIREX 2007 Audio Onset Detection Runtime Data===
  
−
<csv>onset06_runtime.csv</csv>
+
<csv>2007/onset.runtime.csv</csv>
  
 
==Results by Class==
 
==Results by Class==
−
*[[Audio_Onset_Detection_Results:_Complex]]
+
*[[2007:Audio_Onset_Detection_Results:_Complex]]
−
*[[Audio_Onset_Detection_Results:_Poly_Pitched]]
+
*[[2007:Audio_Onset_Detection_Results:_Poly_Pitched]]
−
*[[Audio_Onset_Detection_Results:_Solo_Bars_and_Bells]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Bars_and_Bells]]
−
*[[Audio_Onset_Detection_Results:_Solo_Brass]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Brass]]
−
*[[Audio_Onset_Detection_Results:_Solo_Drum]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Drum]]
−
*[[Audio_Onset_Detection_Results:_Solo_Plucked_Strings]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Plucked_Strings]]
−
*[[Audio_Onset_Detection_Results:_Solo_Singing_Voice]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Singing_Voice]]
−
*[[Audio_Onset_Detection_Results:_Solo_Sustained_Strings]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Sustained_Strings]]
−
*[[Audio_Onset_Detection_Results:_Solo_Winds]]
+
*[[2007:Audio_Onset_Detection_Results:_Solo_Winds]]
  
 
==Individual Results ==
 
==Individual Results ==
−
* [[Audio_Onset_Detection_Results:_Brossier_-_complex]]
+
* [[2007:Audio_Onset_Detection_Results:_Lacoste]]
−
*[[Audio_Onset_Detection_Results:_Brossier_-_dual]]
+
*[[2007:Audio_Onset_Detection_Results:_Lee_-_Joint_-_0.2]]
−
*[[Audio_Onset_Detection_Results:_Brossier_-_hfc]]
+
*[[2007:Audio_Onset_Detection_Results:_Lee_-_Joint_-_0.3]]
−
*[[Audio_Onset_Detection_Results:_Brossier_-_specdiff]]
+
*[[2007:Audio_Onset_Detection_Results:_Lee_-_Joint_-_0.4]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_cd]]
+
*[[2007:Audio_Onset_Detection_Results:_Lee_-_LP]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_nwpd]]
+
*[[2007:Audio_Onset_Detection_Results:_Roebel_1]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_rcd]]
+
*[[2007:Audio_Onset_Detection_Results:_Roebel_2]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_sf]]
+
*[[2007:Audio_Onset_Detection_Results:_Roebel_3]]
−
*[[Audio_Onset_Detection_Results:_Dixon_-_wpd]]
+
*[[2007:Audio_Onset_Detection_Results:_Roebel_4]]
−
*[[Audio_Onset_Detection_Results:_Du]]
+
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_cd]]
−
*[[Audio_Onset_Detection_Results:_Roebel_1]]
+
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_mkl]]
−
*[[Audio_Onset_Detection_Results:_Roebel_2]]
+
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_pd]]
−
*[[Audio_Onset_Detection_Results:_Roebel_3]]
+
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_pow]]
 +
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_rcd]]
 +
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_som]]
 +
*[[2007:Audio_Onset_Detection_Results:_Stowell_-_wpd]]
 +
*[[2007:Audio_Onset_Detection_Results:_Zhou]]]
 +
 
 +
[[Category: Results]]

Latest revision as of 01:17, 15 December 2011

Introduction

These are the results for the 2007 running of the Audio Onset Detection task set. For background information about this task set please refer to the 2007:Audio Onset Detection page.

The aim of the Audio Onset Detection task is to find the time locations at which all musical events in a recording begin. The dataset consists of 85 recordings across 9 different "classes" (e.g. solo drums, polyphonic pitched, etc.). For each sound file, ground truth annotations produced by 3-5 listeners were used for the evaluation. Each algorithm was tested across 10-20 different parameterizations (e.g. thresholds) in order to produce Precision vs. Recall Operating Characteristic (P-ROC) curves. The primary evauluation metric used was the F1-Measure (the equal weighted harmonic mean of precision and recall).

  • Note: There were a few faulty ground truth annotations in the 2005 and 2006 runs of this task. These have been removed for this year's evaluation. Thanks to Dan Stowell for finding these.

General Legend

Team ID

lacoste = Alexandre Lacoste
lee = Wan-Chi Lee, Yu Shiu, C.-C. Jay Kuo
roebel = A. Röbel
stowell = Dan Stowell, Mark Plumbley
zhou = Ruohua Zhou, Joshua D. Reiss

Overall Summary Results

MIREX 2007 Audio Onset Detection Summary Results - Peak F-measure performance across all parameterizations

Contestant Parameters Class # Files in Class Total Correct Total FP Total FN Total Merged Total Doubled Avg. Correct Avg. FP Avg. FN Avg. Merged Avg. Doubled Avg. Precision Avg. Recall Avg. F-Measure
lacoste 0.48 Total 85 7353 2124 2002 172 255 25.318 7.670 6.597 0.533 0.896 0.758 0.774 0.743
lee_joint_0.2 0.05 Total 85 7381 1841 1974 203 10 25.689 6.335 6.227 0.623 0.031 0.825 0.804 0.800
lee_joint_0.3 0.05 Total 85 7303 1726 2052 200 10 25.450 5.950 6.465 0.613 0.031 0.835 0.799 0.802
lee_joint_0.4 0.05 Total 85 7215 1664 2140 200 10 25.146 5.736 6.769 0.613 0.031 0.841 0.792 0.801
lee_lp 0.01 Total 85 7423 1926 1932 217 28 25.871 6.529 6.044 0.668 0.086 0.820 0.807 0.796
roebel_1 0.06 Total 85 6825 918 2530 182 33 23.761 3.110 8.155 0.581 0.104 0.868 0.773 0.796
roebel_2 0.15 Total 85 6932 1208 2423 169 219 24.029 4.030 7.886 0.542 0.661 0.859 0.782 0.793
roebel_3 0.06 Total 85 6825 918 2530 182 33 23.761 3.110 8.155 0.581 0.104 0.868 0.773 0.796
roebel_4 0.15 Total 85 6932 1208 2423 169 219 24.029 4.030 7.886 0.542 0.661 0.859 0.782 0.793
stowell_cd 0.25 Total 85 7487 1734 1868 192 37 25.772 5.840 6.143 0.605 0.118 0.802 0.797 0.784
stowell_mkl 0.55 Total 85 6970 2126 2385 190 57 24.237 8.069 7.678 0.589 0.235 0.749 0.756 0.717
stowell_pd 0.35 Total 85 5866 4941 3489 125 122 20.928 18.590 10.987 0.427 0.473 0.593 0.665 0.565
stowell_pow 0.15 Total 85 7466 2220 1889 177 59 25.644 7.392 6.272 0.565 0.173 0.780 0.794 0.769
stowell_rcd 0.25 Total 85 7480 2240 1875 177 94 25.751 7.532 6.165 0.556 0.267 0.765 0.800 0.762
stowell_som 0.15 Total 85 7747 2646 1608 183 60 26.684 8.975 5.231 0.590 0.178 0.750 0.828 0.770
stowell_wpd 0.15 Total 85 7640 2788 1715 191 79 26.276 9.618 5.640 0.606 0.246 0.739 0.812 0.753
zhou 1.0 Total 85 7225 1186 2130 189 49 25.169 3.795 6.746 0.613 0.151 0.857 0.782 0.808

download these results as csv

MIREX 2007 Audio Onset Detection Summary Plot

File:Total.png

MIREX 2007 Audio Onset Detection Runtime Data

Contestant Machine Avg. run time per parameter set (sec)
lacoste ALE 3 11.5
lee_joint_0.2 ALE 3 1122
lee_joint_0.3 ALE 3 1123
lee_joint_0.4 ALE 3 1123
lee_lp ALE 3 924
roebel_1 ALE 3 265
roebel_2 ALE 3 445
roebel_3 ALE 3 263
roebel_4 ALE 3 443
stowell_cd MINIMAC 42.5
stowell_mkl MINIMAC 37.2
stowell_pow MINIMAC 31.6
stowell_pd MINIMAC 37.7
stowell_rcd MINIMAC 40.5
stowell_som MINIMAC 32.1
stowell_wpd MINIMAC 38.8
zhou FAST 1399

download these results as csv

Results by Class

Individual Results

Retrieved from "https://www.music-ir.org/mirex/w/index.php?title=2007:Audio_Onset_Detection_Results&oldid=8606"