https://www.music-ir.org/mirex/w/api.php?action=feedcontributions&user=Singh14&feedformat=atom
MIREX Wiki - User contributions [en]
2024-03-28T16:18:57Z
User contributions
MediaWiki 1.31.1
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7786
2010:MIREX2010 Results
2010-08-30T21:32:49Z
<p>Singh14: /* Other Tasks */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset] &nbsp;(DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection] &nbsp;(DIY)<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results] &nbsp;(DIY)<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(DIY)<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] &nbsp;(DIY)<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(DIY)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] &nbsp;(DIY)<br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset] &nbsp;(DIY)<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7785
2010:MIREX2010 Results
2010-08-30T21:32:05Z
<p>Singh14: /* Other Tasks */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] &nbsp;(DIY) *<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset] &nbsp;(DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection] &nbsp;(DIY)<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results] &nbsp;(DIY)<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(DIY)<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] &nbsp;(DIY)<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(DIY)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] &nbsp;(DIY)<br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset] &nbsp;(DIY)<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7784
2010:MIREX2010 Results
2010-08-30T21:31:22Z
<p>Singh14: /* Other Tasks */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset] &nbsp;(DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection] &nbsp;(DIY)<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results] &nbsp;(DIY)<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(DIY)<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] &nbsp;(DIY)<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(DIY)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] &nbsp;(DIY)<br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset] &nbsp;(DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset] &nbsp;(DIY)<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7783
2010:MIREX2010 Results
2010-08-30T21:26:05Z
<p>Singh14: /* Other Tasks */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset] (DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection] (DIY)<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results] (DIY)<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] (DIY)<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] (DIY)<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] (DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation] (DIY)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] (DIY)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation] (DIY)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] (DIY)<br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset] (DIY)<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7782
2010:MIREX2010 Results
2010-08-30T21:22:21Z
<p>Singh14: /* Train-Test Task Set */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ]&nbsp;&nbsp; (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection]<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results]<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] <br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation]<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] <br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset]<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7781
2010:MIREX2010 Results
2010-08-30T21:19:42Z
<p>Singh14: /* Results by Task */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new [http://nema.lis.uiuc.edu/drupal/?q=nema/architecture NEMA MIREX DIY] infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration [https://www.music-ir.org/diy-demo/ video] of the NEMA MIREX DIY system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ] (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ] (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection]<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results]<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] <br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation]<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] <br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset]<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2010:MIREX2010_Results&diff=7780
2010:MIREX2010 Results
2010-08-30T21:09:23Z
<p>Singh14: /* Results by Task */</p>
<hr />
<div>==OVERALL RESULTS POSTERS (First Version: Will need updating as last runs are completed)==<br />
[https://www.music-ir.org/mirex/results/2010/mirex_2010_poster.pdf MIREX 2010 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
This year we ran many MIREX 2010 Tasks using the new NEMA MIREX DIY infrastructure. Task results with "(DIY)" appended are those generated using the NEMA MIREX DIY system. Where appropriate, do explore the various new outputs that help visualize both individual and task-wide comparative performances. A demonstration video of the [https://www.music-ir.org/diy-demo/ NEMA MIREX DIY] system can be found is also available.<br />
<br />
<br />
===Train-Test Task Set===<br />
* [http://nema.lis.uiuc.edu/nema_out/664ccbda-d5b6-48ae-8c47-c27e7c2372fe/results/evaluation/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/d97c8282-883e-4e71-93b7-55283829ad21/results/evaluation/ Audio Latin Genre Classification Results ] (DIY)<br />
* [http://nema.lis.uiuc.edu/nema_out/0e2212ca-2c1a-4c4e-b164-de74974afe43/results/evaluation/ Audio Music Mood Classification Results ] (DIY)<br />
* [[2010:Audio_Mixed_Popular_Genre_Classification_Results | Audio Mixed Popular Genre Classification Results]]<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/mck/ MCK Dataset] (DIY)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/abt/maz/ MAZ Dataset]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ace/ Audio Chord Detection]<br />
* [[2010:Audio_Cover_Song_Identification_Results | Audio Cover Song Identification Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/akd/ Audio Key Detection Results]<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/adc04/ ADC04 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex05/ MIREX05 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/indian08/ INDIAN08 Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_0dB/ MIREX09 0dB Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset]<br />
* [[2010:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/aod/ Audio Onset Detection Results] <br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)]<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask1_report/aff/ Affinity estimation evaluation]<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)]<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/atg/subtask2_report/aff/ Affinity estimation evaluation]<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2010/results/ate/ Audio Tempo Estimation Results] <br />
* [[2010:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex09/ MIREX09 dataset]<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2010/results/struct/mirex10/ MIREX10 dataset]<br />
* [[2010:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] <br />
* [[2010:Query-by-Tapping_Results | Query-by-Tapping Results]]<br />
*[[2010:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2010:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]]<br />
<br />
== Machine Specifications ==<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
* [[2010:Runtime | Runtime]]<br />
<br />
[[Category:Results]]</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Key_Finding_Results&diff=7622
2005:Audio Key Finding Results
2010-08-02T22:34:20Z
<p>Singh14: /* Results */</p>
<hr />
<div>==Introduction==<br />
<br />
===Goal===<br />
The evaluation of key finding algorithms applied to audio sound files<br />
<br />
===Dataset=== <br />
1,252 audio files synthesized from MIDI Note: There is a close relationship (same musical datasets) between this contest and the Symbolic Key Finding contest. Here is a link to the Symbolic Key Finding results.<br />
<br />
Two databases used: Winamp synthesized audio (w) and Timidity with Fusion soundfonts synthesized audio (t). Each database is approximately 3.1 gigabytes for a total of 6.2 gigabytes of audio files.<br />
<br />
The composite score is calculated by averaging the Winamp and Timidity scores. <br />
<br />
==Results==<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Composite Percentage Score<br />
!colspan="2" | Total Score<br />
!colspan="2" | Percentage Score<br />
!colspan="2" | Correct Keys<br />
!colspan="2" | Perfect 5th Errors<br />
!colspan="2" | Relative Major/Minor Errors<br />
!colspan="2" | Parallel Major/Minor Errors<br />
!colspan="2" | Other Errors<br />
!colspan="2" | Runtime (s)<br />
!Machine<br />
|----<br />
|<br />
|<br />
|<br />
|w<br />
|t<br />
|w<br />
|t<br />
|w<br />
|t<br />
|w<br />
|t<br />
|w<br />
|t<br />
|w<br />
|t<br />
|w<br />
|t<br />
|w<br />
|t<br />
|<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/izmirli.pdf Izmirli, Ö. ]<br />
|align="right " | 89.55%<br />
|align="right " | 1188.8<br />
|align="right " | 1122.9<br />
|align="right " | 89.4%<br />
|align="right " | 89.7%<br />
|align="right " | 1086<br />
|align="right " | 1089<br />
|align="right " | 36<br />
|align="right " | 42<br />
|align="right " | 38<br />
|align="right " | 31<br />
|align="right " | 17<br />
|align="right " | 18<br />
|align="right " | 75<br />
|align="right " | 72<br />
|align="right " | 15284<br />
|align="right " | 16354<br />
|align="right " | Y<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/purwins.pdf Purwins &amp; Blankertz ]<br />
|align="right " | 89.00%<br />
|align="right " | 1122.4<br />
|align="right " | 1106.5<br />
|align="right " | 89.6%<br />
|align="right " | 88.4%<br />
|align="right " | 1090<br />
|align="right " | 1060<br />
|align="right " | 44<br />
|align="right " | 72<br />
|align="right " | 24<br />
|align="right " | 21<br />
|align="right " | 16<br />
|align="right " | 21<br />
|align="right " | 78<br />
|align="right " | 78<br />
|align="right " | 45003<br />
|align="right " | 44232<br />
|align="right " | R<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gomez.pdf Gómez, E. (start)]<br />
|align="right " | 86.05%<br />
|align="right " | 1081.9<br />
|align="right " | 1072.9<br />
|align="right " | 86.4%<br />
|align="right " | 85.7%<br />
|align="right " | 1048<br />
|align="right " | 1034<br />
|align="right " | 35<br />
|align="right " | 44<br />
|align="right " | 38<br />
|align="right " | 43<br />
|align="right " | 25<br />
|align="right " | 20<br />
|align="right " | 106<br />
|align="right " | 111<br />
|align="right " | 1560<br />
|align="right " | 1531<br />
|align="right " | B 0<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gomez.pdf Gómez, E. (global)]<br />
|align="right " | 85.90%<br />
|align="right " | 1076.1<br />
|align="right " | 1073.8<br />
|align="right " | 86.0%<br />
|align="right " | 85.8%<br />
|align="right " | 1019<br />
|align="right " | 1015<br />
|align="right " | 69<br />
|align="right " | 73<br />
|align="right " | 62<br />
|align="right " | 59<br />
|align="right " | 20<br />
|align="right " | 23<br />
|align="right " | 82<br />
|align="right " | 82<br />
|align="right " | 2041<br />
|align="right " | 1971<br />
|align="right " | B 0<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/pauws.pdf Pauws, S. ]<br />
|align="right " | 85.00%<br />
|align="right " | 1055.1<br />
|align="right " | 1072.8<br />
|align="right " | 84.3%<br />
|align="right " | 85.7%<br />
|align="right " | 1019<br />
|align="right " | 1034<br />
|align="right " | 20<br />
|align="right " | 23<br />
|align="right " | 67<br />
|align="right " | 69<br />
|align="right " | 30<br />
|align="right " | 33<br />
|align="right " | 116<br />
|align="right " | 93<br />
|align="right " | 503<br />
|align="right " | 507<br />
|align="right " | G<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/zhu.pdf Zhu, Y.]<br />
|align="right " | 83.25%<br />
|align="right " | 1066.2<br />
|align="right " | 1017.7<br />
|align="right " | 85.2%<br />
|align="right " | 81.3%<br />
|align="right " | 1034<br />
|align="right " | 964<br />
|align="right " | 38<br />
|align="right " | 66<br />
|align="right " | 28<br />
|align="right " | 47<br />
|align="right " | 24<br />
|align="right " | 33<br />
|align="right " | 128<br />
|align="right " | 142<br />
|align="right " | 25233<br />
|align="right " | 24039<br />
|align="right " | Y<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/chuan.pdf Chuan &amp; Chew]<br />
|align="right " | 79.10%<br />
|align="right " | 1002.3<br />
|align="right " | 977.3<br />
|align="right " | 80.1%<br />
|align="right " | 78.1%<br />
|align="right " | 937<br />
|align="right " | 905<br />
|align="right " | 83<br />
|align="right " | 95<br />
|align="right " | 66<br />
|align="right " | 68<br />
|align="right " | 20<br />
|align="right " | 22<br />
|align="right " | 146<br />
|align="right " | 162<br />
|align="right " | 3299<br />
|align="right " | 3468<br />
|align="right " | R<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7621
2005:Audio Melody Extraction Results
2010-08-02T22:25:19Z
<p>Singh14: /* McNemars Test Results */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
===Goal===<br />
To extract melodic content from polyphonic audio.<br />
<br />
===Dataset===<br />
25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano<br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
| align="right" |81.8%<br />
| align="right" |17.3%<br />
| align="right" |1.85<br />
| align="right" |68.1%<br />
| align="right" |71.4%<br />
| align="right" |'''71.4%'''<br />
| align="right" |32<br />
| align="right" |R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
| align="right" |90.3%<br />
| align="right" |39.5%<br />
| align="right" |1.56<br />
| align="right" |'''68.6%'''<br />
| align="right" |'''74.1%'''<br />
| align="right" |64.3%<br />
| align="right" |10970<br />
| align="right" |L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
| align="right" |91.6%<br />
| align="right" |42.7%<br />
| align="right" |1.56<br />
| align="right" |67.3%<br />
| align="right" |73.4%<br />
| align="right" |61.1%<br />
| align="right" |5471<br />
| align="right" |B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
| align="right" |68.8%<br />
| align="right" |23.2%<br />
| align="right" |1.22<br />
| align="right" |58.5%<br />
| align="right" |62.0%<br />
| align="right" |61.1%<br />
| align="right" |45618<br />
| align="right" |Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
| align="right" |72.7%<br />
| align="right" |32.4%<br />
| align="right" |1.06<br />
| align="right" |60.1%<br />
| align="right" |67.1%<br />
| align="right" |59.5%<br />
| align="right" |12461<br />
| align="right" |F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
| align="right" |83.4%<br />
| align="right" |55.8%<br />
| align="right" |0.83<br />
| align="right" |62.7%<br />
| align="right" |66.7%<br />
| align="right" |57.8%<br />
| align="right" |44312<br />
| align="right" |G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
| align="right" |99.9% *<br />
| align="right" |99.4% *<br />
| align="right" |0.59 *<br />
| align="right" |65.8%<br />
| align="right" |71.8%<br />
| align="right" |49.9% *<br />
| align="right" |211<br />
| align="right" |F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
| align="right" |96.1% *<br />
| align="right" |93.7% *<br />
| align="right" |0.23 *<br />
| align="right" |59.8%<br />
| align="right" |67.6%<br />
| align="right" |47.9% *<br />
| align="right" |?<br />
| align="right" |G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
| align="right" |99.6% *<br />
| align="right" |96.4% *<br />
| align="right" |0.86 *<br />
| align="right" |59.6%<br />
| align="right" |71.1%<br />
| align="right" |46.4% *<br />
| align="right" |251<br />
| align="right" |G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
| align="right" |99.2% * †<br />
| align="right" |98.8% * †<br />
| align="right" |0.14 * †<br />
| align="right" |3.9% †<br />
| align="right" |8.1% †<br />
| align="right" |3.2% * †<br />
| align="right" |41<br />
| align="right" |B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
| align="right" |70.78%<br />
| align="right" |67.48%<br />
| align="right" |70.83%<br />
| align="right" |80.20%<br />
| align="right" |78.34%<br />
| align="right" |81.48%<br />
| align="right" |67.48%<br />
| align="right" |70.83%<br />
| align="right" |73.61%<br />
| align="right" |32<br />
| align="right" |R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
| align="right" |63.94%<br />
| align="right" |65.99%<br />
| align="right" |69.85%<br />
| align="right" |81.87%<br />
| align="right" |70.22%<br />
| align="right" |73.85%<br />
| align="right" |68.20%<br />
| align="right" |73.70%<br />
| align="right" |67.33%<br />
| align="right" |10970<br />
| align="right" |L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
| align="right" |60.70%<br />
| align="right" |58.16%<br />
| align="right" |61.56%<br />
| align="right" |70.92%<br />
| align="right" |70.23%<br />
| align="right" |74.06%<br />
| align="right" |58.16%<br />
| align="right" |61.56%<br />
| align="right" |63.79%<br />
| align="right" |45618<br />
| align="right" |Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
| align="right" |60.61%<br />
| align="right" |62.47%<br />
| align="right" |67.95%<br />
| align="right" |83.02%<br />
| align="right" |66.18%<br />
| align="right" |71.10%<br />
| align="right" |66.75%<br />
| align="right" |72.86%<br />
| align="right" |65.14%<br />
| align="right" |5471<br />
| align="right" |B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
| align="right" |59.18%<br />
| align="right" |58.06%<br />
| align="right" |63.13%<br />
| align="right" |73.35%<br />
| align="right" |67.53%<br />
| align="right" |72.24%<br />
| align="right" |59.76%<br />
| align="right" |66.73%<br />
| align="right" |63.25%<br />
| align="right" |12461<br />
| align="right" |F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
| align="right" |57.32%<br />
| align="right" |62.23%<br />
| align="right" |66.20%<br />
| align="right" |74.76%<br />
| align="right" |65.42%<br />
| align="right" |69.32%<br />
| align="right" |62.23%<br />
| align="right" |66.20%<br />
| align="right" |60.76%<br />
| align="right" |44312<br />
| align="right" |G<br />
|----<br />
|7<br />
|Goto, M.<br />
| align="right" |49.68%<br />
| align="right" |65.58%<br />
| align="right" |71.47%<br />
| align="right" |77.18%<br />
| align="right" |56.46%<br />
| align="right" |61.99%<br />
| align="right" |65.58%<br />
| align="right" |71.47%<br />
| align="right" |54.89%<br />
| align="right" |211<br />
| align="right" |F<br />
|----<br />
|8<br />
|Vincent, E.<br />
| align="right" |45.98%<br />
| align="right" |59.17%<br />
| align="right" |70.64%<br />
| align="right" |77.61%<br />
| align="right" |51.98%<br />
| align="right" |62.36%<br />
| align="right" |59.17%<br />
| align="right" |70.64%<br />
| align="right" |55.52%<br />
| align="right" |251<br />
| align="right" |G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
| align="right" |3.2%<br />
| align="right" |3.93%<br />
| align="right" |8.06%<br />
| align="right" |76.61%<br />
| align="right" |3.63%<br />
| align="right" |7.27%<br />
| align="right" |3.93%<br />
| align="right" |8.06%<br />
| align="right" |6.43%<br />
| align="right" |41<br />
| align="right" |B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Goto, M.<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Vincent, E.<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Poliner &amp; Ellis<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Paiva, R. (2)<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Paiva, R. (1)<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|<br />
|----<br />
|Marolt, M.<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 2.71%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|<br />
|----<br />
|Dressler, K.<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | 0%<br />
|align="right " | n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7620
2005:Audio Tempo Extraction Results
2010-08-02T22:20:08Z
<p>Singh14: /* Results */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Tempo Extraction task. <br />
<br />
===Goal===<br />
The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
===Dataset=== <br />
140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/alonso.pdf Alonso, David, &amp; Richard]<br />
|align="right " | 0.689 (0.231)<br />
|align="right " | 95.00%<br />
|align="right " | 55.71%<br />
|align="right " | 25.00%<br />
|align="right " | 5.00%<br />
|align="right " | 0.239<br />
|align="right " | 2875<br />
|align="right " | G<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (1)]<br />
|align="right " | 0.675 (0.273)<br />
|align="right " | 90.71%<br />
|align="right " | 59.29%<br />
|align="right " | 32.14%<br />
|align="right " | 7.14%<br />
|align="right " | 0.222<br />
|align="right " | 1160<br />
|align="right " | F<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (2)]<br />
|align="right " | 0.675 (0.272)<br />
|align="right " | 90.71%<br />
|align="right " | 59.29%<br />
|align="right " | 32.86%<br />
|align="right " | 6.43%<br />
|align="right " | 0.222<br />
|align="right " | 2621<br />
|align="right " | F<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (1)]<br />
|align="right " | 0.670 (0.252)<br />
|align="right " | 92.14%<br />
|align="right " | 56.43%<br />
|align="right " | 40.71%<br />
|align="right " | 7.86%<br />
|align="right " | 0.311<br />
|align="right " | 3303<br />
|align="right " | G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/peeters.pdf Peeters, G.]<br />
|align="right " | 0.656 (0.223)<br />
|align="right " | 95.71%<br />
|align="right " | 47.86%<br />
|align="right " | 27.86%<br />
|align="right " | 4.29%<br />
|align="right " | 0.258<br />
|align="right " | 2159<br />
|align="right " | R<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (2)]<br />
|align="right " | 0.649 (0.253)<br />
|align="right " | 92.14%<br />
|align="right " | 51.43%<br />
|align="right " | 37.14%<br />
|align="right " | 5.71%<br />
|align="right " | 0.305<br />
|align="right " | 2050<br />
|align="right " | G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (4)]<br />
|align="right " | 0.645 (0.294)<br />
|align="right " | 87.14%<br />
|align="right " | 55.71%<br />
|align="right " | 48.57%<br />
|align="right " | 10.71%<br />
|align="right " | 0.313<br />
|align="right " | 1357<br />
|align="right " | G<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/eck.pdf Eck, D.]<br />
|align="right " | 0.644 (0.300)<br />
|align="right " | 86.43%<br />
|align="right " | 53.57%<br />
|align="right " | 37.14%<br />
|align="right " | 5.71%<br />
|align="right " | 0.230<br />
|align="right " | 1665<br />
|align="right " | Y<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Davies &amp; Brossier]<br />
|align="right " | 0.628 (0.284)<br />
|align="right " | 86.43%<br />
|align="right " | 48.57%<br />
|align="right " | 26.43%<br />
|align="right " | 4.29%<br />
|align="right " | 0.224<br />
|align="right " | 1005<br />
|align="right " | R<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (3)]<br />
|align="right " | 0.607 (0.287)<br />
|align="right " | 87.14%<br />
|align="right " | 47.14%<br />
|align="right " | 36.43%<br />
|align="right " | 6.43%<br />
|align="right " | 0.294<br />
|align="right " | 1388<br />
|align="right " | R<br />
|----<br />
|11<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sethares.pdf Sethares, W.]<br />
|align="right " | 0.597 (0.252)<br />
|align="right " | 90.71%<br />
|align="right " | 37.86%<br />
|align="right " | 30.71%<br />
|align="right " | 0.71%<br />
|align="right " | 0.239<br />
|align="right " | 70975<br />
|align="right " | Y<br />
|----<br />
|12<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Brossier, P.]<br />
|align="right " | 0.583 (0.333)<br />
|align="right " | 80.71%<br />
|align="right " | 51.43%<br />
|align="right " | 28.57%<br />
|align="right " | 2.14%<br />
|align="right " | 0.223<br />
|align="right " | 180<br />
|align="right " | B 0<br />
|----<br />
|13<br />
|[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.]<br />
|align="right " | 0.538 (0.359)<br />
|align="right " | 71.43%<br />
|align="right " | 50.71%<br />
|align="right " | 28.57%<br />
|align="right " | 3.57%<br />
|align="right " | 0.295<br />
|align="right " | 7173<br />
|align="right " | B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|align="right " | n/a<br />
|align="right " | <br />
|align="right " | <br />
|align="right " | <br />
|align="right " | <br />
|align="right " | <br />
|align="right " | <br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|align="right " | 17.44%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|align="right " | 50.00%<br />
|align="right " | 19.58%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|align="right " | 12.15%<br />
|align="right " | 50.00%<br />
|align="right " | 4.61%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|align="right " | 50.00%<br />
|align="right " | 23.54%<br />
|align="right " | 55.31%<br />
|align="right " | 20.05%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|align="right " | 2.88%<br />
|align="right " | 26.64%<br />
|align="right " | 6.76%<br />
|align="right " | 31.36%<br />
|align="right " | 5.41%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|align="right " | 2.88%<br />
|align="right " | 26.64%<br />
|align="right " | 6.31%<br />
|align="right " | 30.89%<br />
|align="right " | 5.86%<br />
|align="right " | 75.00%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|align="right " | 0.99%<br />
|align="right " | 0.02%<br />
|align="right " | 0.27%<br />
|align="right " | 0.02%<br />
|align="right " | 1.24%<br />
|align="right " | 0.01%<br />
|align="right " | 0.01%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|align="right " | 34.70%<br />
|align="right " | 7.19%<br />
|align="right " | 29.83%<br />
|align="right " | 6.32%<br />
|align="right " | 30.73%<br />
|align="right " | 2.40%<br />
|align="right " | 2.40%<br />
|align="right " | 2.97%<br />
|align="right " | n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|align="right " | 32.58%<br />
|align="right " | 38.04%<br />
|align="right " | 38.54%<br />
|align="right " | 32.20%<br />
|align="right " | 38.54%<br />
|align="right " | 12.15%<br />
|align="right " | 12.15%<br />
|align="right " | 0.41%<br />
|align="right " | 18.32%<br />
|align="right " | n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|align="right " | 38.77%<br />
|align="right " | 7.17%<br />
|align="right " | 33.59%<br />
|align="right " | 6.31%<br />
|align="right " | 33.89%<br />
|align="right " | 2.67%<br />
|align="right " | 2.67%<br />
|align="right " | 1.38%<br />
|align="right " | 50.00%<br />
|align="right " | 21.35%<br />
|align="right " | n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|align="right " | 18.02%<br />
|align="right " | 56.12%<br />
|align="right " | 23.99%<br />
|align="right " | 50.00%<br />
|align="right " | 23.99%<br />
|align="right " | 26.64%<br />
|align="right " | 26.64%<br />
|align="right " | 0.04%<br />
|align="right " | 7.62%<br />
|align="right " | 39.39%<br />
|align="right " | 6.07%<br />
|align="right " | n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7619
2005:Symbolic Genre Classification Results
2010-08-02T22:14:31Z
<p>Singh14: /* 9 Classes */</p>
<hr />
<div>==Introduction==<br />
===Goal===<br />
To classify MIDI recordings into genre categories.<br />
<br />
===Dataset===<br />
Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold cross validated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|align="right " | 77.17%<br />
|align="right " | 65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|align="right " | 72.08%<br />
|align="right " | 58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|align="right " | 67.57%<br />
|align="right " | 55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|align="right " | 67.14%<br />
|align="right " | 53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|align="right " | 37.76%<br />
|align="right " | 26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|align="right " | 64.33% <br />
|align="right " | 1.04<br />
|align="right " | 46.11%<br />
|align="right " | 1.51<br />
|align="right " | 3 days<br />
|align="right " | R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|align="right " | 62.60%<br />
|align="right " | 0.26<br />
|align="right " | 45.05%<br />
|align="right " | 0.55<br />
|align="right " | N/A<br />
|align="right " | N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|align="right " | 57.61%<br />
|align="right " | 1.14<br />
|align="right " | 40.95%<br />
|align="right " | 1.35<br />
|align="right " | N/A<br />
|align="right " | N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|align="right " | 54.91%<br />
|align="right " | 0.66<br />
|align="right " | 39.79%<br />
|align="right " | 0.87<br />
|align="right " | 15,948<br />
|align="right " | G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|align="right " | 24.84%<br />
|align="right " | 1.40<br />
|align="right " | 15.26%<br />
|align="right " | 1.13<br />
|align="right " | 821<br />
|align="right " | L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|align="right " | 90.00% <br />
|align="right " | 0.60<br />
|align="right " | 84.44%<br />
|align="right " | 1.41<br />
|align="right " | 18,375<br />
|align="right " | R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|align="right " | 81.56%<br />
|align="right " | 0.76<br />
|align="right " | 72.00%<br />
|align="right " | 0.88<br />
|align="right " | N/A<br />
|align="right " | N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|align="right " | 80.22%<br />
|align="right " | 1.47<br />
|align="right " | 72.00%<br />
|align="right " | 2.31<br />
|align="right " | 3,777<br />
|align="right " | G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|align="right " | 76.67%<br />
|align="right " | 1.11<br />
|align="right " | 65.33%<br />
|align="right " | 1.65<br />
|align="right " | N/A<br />
|align="right " | N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|align="right " | 50.67%<br />
|align="right " | 1.26<br />
|align="right " | 37.78%<br />
|align="right " | 2.30<br />
|align="right " | 197<br />
|align="right " | L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7618
2005:Symbolic Genre Classification Results
2010-08-02T22:12:55Z
<p>Singh14: /* 38 Classes */</p>
<hr />
<div>==Introduction==<br />
===Goal===<br />
To classify MIDI recordings into genre categories.<br />
<br />
===Dataset===<br />
Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold cross validated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|align="right " | 77.17%<br />
|align="right " | 65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|align="right " | 72.08%<br />
|align="right " | 58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|align="right " | 67.57%<br />
|align="right " | 55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|align="right " | 67.14%<br />
|align="right " | 53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|align="right " | 37.76%<br />
|align="right " | 26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|align="right " | 64.33% <br />
|align="right " | 1.04<br />
|align="right " | 46.11%<br />
|align="right " | 1.51<br />
|align="right " | 3 days<br />
|align="right " | R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|align="right " | 62.60%<br />
|align="right " | 0.26<br />
|align="right " | 45.05%<br />
|align="right " | 0.55<br />
|align="right " | N/A<br />
|align="right " | N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|align="right " | 57.61%<br />
|align="right " | 1.14<br />
|align="right " | 40.95%<br />
|align="right " | 1.35<br />
|align="right " | N/A<br />
|align="right " | N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|align="right " | 54.91%<br />
|align="right " | 0.66<br />
|align="right " | 39.79%<br />
|align="right " | 0.87<br />
|align="right " | 15,948<br />
|align="right " | G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|align="right " | 24.84%<br />
|align="right " | 1.40<br />
|align="right " | 15.26%<br />
|align="right " | 1.13<br />
|align="right " | 821<br />
|align="right " | L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7617
2005:Symbolic Genre Classification Results
2010-08-02T22:11:32Z
<p>Singh14: /* Overall */</p>
<hr />
<div>==Introduction==<br />
===Goal===<br />
To classify MIDI recordings into genre categories.<br />
<br />
===Dataset===<br />
Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold cross validated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|align="right " | 77.17%<br />
|align="right " | 65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|align="right " | 72.08%<br />
|align="right " | 58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|align="right " | 67.57%<br />
|align="right " | 55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|align="right " | 67.14%<br />
|align="right " | 53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|align="right " | 37.76%<br />
|align="right " | 26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7616
2005:Symbolic Genre Classification Results
2010-08-02T22:10:08Z
<p>Singh14: /* Introduction */</p>
<hr />
<div>==Introduction==<br />
===Goal===<br />
To classify MIDI recordings into genre categories.<br />
<br />
===Dataset===<br />
Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold cross validated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|77.17%<br />
|65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|72.08%<br />
|58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|67.57%<br />
|55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|67.14%<br />
|53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|37.76%<br />
|26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Melodic_Similarity_Results&diff=7615
2005:Symbolic Melodic Similarity Results
2010-08-02T22:07:20Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
===Goal===<br />
To retrieve the most similar incipits from the RISM A/II collection, given one of the incipits as a query.<br />
<br />
===Dataset===<br />
558 MusicXML files of the incipits from the RISM collection. 11 incipits of this collection are chosen as queries. These MusicXML files were converted to MIDI files. <br />
<br />
==Result==<br />
The scores reported here are averaged over the 11 queries. <br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average Dynamic Recall <br />
!Normalized Recall at Group Boundaries<br />
!Average precision (non-interpolated)<br />
!Precision at N documents (N is number of relevant document) <br />
!Input Data Format<br />
!Runtime (s)<br />
!Machine<br />
!Detailed Evaluation Result<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/grachten.pdf Grachten, Arcos &amp; Mántaras]<br />
|align="right " | 65.98%<br />
|align="right " | 55.24%<br />
|align="right " | 51.72%<br />
|align="right " | 44.33%<br />
|align="center " | MIDI<br />
|align="right " | 80.174 *<br />
|align="right " | B0<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/GAM_eval.txt GAM_eval.txt]<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/orio.pdf Orio, N.]<br />
|align="right " | 64.96%<br />
|align="right " | 53.35%<br />
|align="right " | 42.96%<br />
|align="right " | 39.86%<br />
|align="center" | XML<br />
|align="right " | 24.610<br />
|align="right " | B4<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/O_eval.txt O_eval.txt]<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/suyoto.pdf Suyoto &amp; Uitdenbogerd]<br />
|align="right " | 64.18%<br />
|align="right " | 51.79%<br />
|align="right " | 40.42%<br />
|align="right " | 41.72%<br />
|align="center " | MIDI<br />
|align="right " | 48.133<br />
|align="right " | B3<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/SU_eval.txt SU_eval.txt]<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/typke.pdf Typke, Wiering &amp; Veltkamp]<br />
|align="right " | 57.09%<br />
|align="right " | 48.17%<br />
|align="right " | 35.64%<br />
|align="right " | 33.46%<br />
|align="center " | MIDI<br />
|align="right " | 51240<br />
|align="right " | B4<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/TWV_eval.txt TWV_eval.txt]<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mikkila.pdf Lemström, Mikkilä, Mäkinen &amp; Ukkonen (P3)]<br />
|align="right " | 55.82%<br />
|align="right " | 46.56%<br />
|align="right " | 41.40%<br />
|align="right " | 39.18%<br />
|align="center " | MIDI<br />
|align="right " | 10.007 *<br />
|align="right " | B0<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/LMMU_P3_eval.txt LMMU_P3_eval.txt]<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mikkila.pdf Lemström, Mikkilä, Mäkinen &amp; Ukkonen (DP)]<br />
|align="right " | 54.27%<br />
|align="right " | 47.26%<br />
|align="right " | 39.91%<br />
|align="right " | 36.20%<br />
|align="center " | MIDI<br />
|align="right " | 10.106 *<br />
|align="right " | B0<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/LMMU_DP_eval.txt LMMU_DP_eval.txt]<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/frieler.pdf Frieler &amp; Müllensiefen]<br />
|align="right " | 51.81%<br />
|align="right " | 45.10%<br />
|align="right " | 33.93%<br />
|align="right " | 33.71%<br />
|align="center " | MIDI<br />
|align="right " | 54.593<br />
|align="right " | B4<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/FM_eval.txt FM_eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Key_Finding_Results&diff=7614
2005:Symbolic Key Finding Results
2010-08-02T22:02:48Z
<p>Singh14: /* Goal */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Symbolic Key Finding task set.<br />
<br />
===Goal===<br />
The evaluation of key finding algorithms applied to MIDI files. Note: There is a close relationship (same musical datasets) between this contest and the Audio Key Finding contest. Here is a link to the Audio Key Finding results.<br />
<br />
===Dataset===<br />
1,252 MIDI files, 3.62 Megabytes <br />
<br />
==Result==<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Total Score<br />
!Percentage Score<br />
!Correct Keys<br />
!Perfect 5th Errors<br />
!Relative Major/Minor Errors<br />
!Parallel Major/Minor Errors<br />
!Other Errors<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/temperley.pdf Temperley, D.]<br />
|align="right " | 1143.8<br />
|align="right " | 91.4%<br />
|align="right " | 1127<br />
|align="right " | 15<br />
|align="right " | 27<br />
|align="right " | 6<br />
|align="right " | 77<br />
|align="right " | 91<br />
|align="right " | B 0<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/zhu2.pdf Zhu, Y.]<br />
|align="right " | 1075.2<br />
|align="right " | 85.9%<br />
|align="right " | 1041<br />
|align="right " | 35<br />
|align="right " | 47<br />
|align="right " | 13<br />
|align="right " | 116<br />
|align="right " | ? (See note)<br />
|align="right " | R<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/rizo.pdf Rizo &amp; Iñesta]<br />
|align="right " | 982.4<br />
|align="right " | 78.5%<br />
|align="right " | 913<br />
|align="right " | 81<br />
|align="right " | 87<br />
|align="right " | 14<br />
|align="right " | 157<br />
|align="right " | 631<br />
|align="right " | B 0<br />
|----<br />
|4<br />
|Ehmann, A.<br />
|align="right " | 947.4<br />
|align="right " | 75.7%<br />
|align="right " | 851<br />
|align="right " | 160<br />
|align="right " | 44<br />
|align="right " | 16<br />
|align="right " | 181<br />
|align="right " | 5670<br />
|align="right " | G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mardirossian.pdf Mardirossian &amp; Chew]<br />
|align="right " | 934.0<br />
|align="right " | 74.6%<br />
|align="right " | 799<br />
|align="right " | 210<br />
|align="right " | 80<br />
|align="right " | 30<br />
|align="right " | 133<br />
|align="right " | 471<br />
|align="right " | B 0<br />
|----<br />
|}<br />
<br />
'''Note:''' Runtime undetermined due to system hang.</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Key_Finding_Results&diff=7613
2005:Symbolic Key Finding Results
2010-08-02T22:02:31Z
<p>Singh14: /* Result */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Symbolic Key Finding task set.<br />
<br />
===Goal===<br />
The evaluation of key finding algorithms applied to MIDI files. Note: There is a close relationship (same musical datasets) between this contest and the Audio Key Finding contest. Here is a link to the Audio Key Finding results.<br />
<br />
===Dataset===<br />
1,252 MIDI files, 3.62 Megabytes <br />
<br />
==Result==<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Total Score<br />
!Percentage Score<br />
!Correct Keys<br />
!Perfect 5th Errors<br />
!Relative Major/Minor Errors<br />
!Parallel Major/Minor Errors<br />
!Other Errors<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/temperley.pdf Temperley, D.]<br />
|align="right " | 1143.8<br />
|align="right " | 91.4%<br />
|align="right " | 1127<br />
|align="right " | 15<br />
|align="right " | 27<br />
|align="right " | 6<br />
|align="right " | 77<br />
|align="right " | 91<br />
|align="right " | B 0<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/zhu2.pdf Zhu, Y.]<br />
|align="right " | 1075.2<br />
|align="right " | 85.9%<br />
|align="right " | 1041<br />
|align="right " | 35<br />
|align="right " | 47<br />
|align="right " | 13<br />
|align="right " | 116<br />
|align="right " | ? (See note)<br />
|align="right " | R<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/rizo.pdf Rizo &amp; Iñesta]<br />
|align="right " | 982.4<br />
|align="right " | 78.5%<br />
|align="right " | 913<br />
|align="right " | 81<br />
|align="right " | 87<br />
|align="right " | 14<br />
|align="right " | 157<br />
|align="right " | 631<br />
|align="right " | B 0<br />
|----<br />
|4<br />
|Ehmann, A.<br />
|align="right " | 947.4<br />
|align="right " | 75.7%<br />
|align="right " | 851<br />
|align="right " | 160<br />
|align="right " | 44<br />
|align="right " | 16<br />
|align="right " | 181<br />
|align="right " | 5670<br />
|align="right " | G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mardirossian.pdf Mardirossian &amp; Chew]<br />
|align="right " | 934.0<br />
|align="right " | 74.6%<br />
|align="right " | 799<br />
|align="right " | 210<br />
|align="right " | 80<br />
|align="right " | 30<br />
|align="right " | 133<br />
|align="right " | 471<br />
|align="right " | B 0<br />
|----<br />
|}<br />
<br />
'''Note:''' Runtime undetermined due to system hang.</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Key_Finding_Results&diff=7612
2005:Symbolic Key Finding Results
2010-08-02T22:01:37Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Symbolic Key Finding task set.<br />
<br />
===Goal===<br />
The evaluation of key finding algorithms applied to MIDI files. Note: There is a close relationship (same musical datasets) between this contest and the Audio Key Finding contest. Here is a link to the Audio Key Finding results.<br />
<br />
===Dataset===<br />
1,252 MIDI files, 3.62 Megabytes <br />
<br />
==Result==<br />
<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Total Score<br />
!Percentage Score<br />
!Correct Keys<br />
!Perfect 5th Errors<br />
!Relative Major/Minor Errors<br />
!Parallel Major/Minor Errors<br />
!Other Errors<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/temperley.pdf Temperley, D.]<br />
|align="right " | 1143.8<br />
|align="right " | 91.4%<br />
|align="right " | 1127<br />
|align="right " | 15<br />
|align="right " | 27<br />
|align="right " | 6<br />
|align="right " | 77<br />
|align="right " | 91<br />
|align="right " | B 0<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/zhu2.pdf Zhu, Y.]<br />
|align="right " | 1075.2<br />
|align="right " | 85.9%<br />
|align="right " | 1041<br />
|align="right " | 35<br />
|align="right " | 47<br />
|align="right " | 13<br />
|align="right " | 116<br />
|align="right " | ? (See note)<br />
|align="right " | R<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/rizo.pdf Rizo &amp; Iñesta]<br />
|align="right " | 982.4<br />
|align="right " | 78.5%<br />
|align="right " | 913<br />
|align="right " | 81<br />
|align="right " | 87<br />
|align="right " | 14<br />
|align="right " | 157<br />
|align="right " | 631<br />
|align="right " | B 0<br />
|----<br />
|4<br />
|Ehmann, A.<br />
|align="right " | 947.4<br />
|align="right " | 75.7%<br />
|align="right " | 851<br />
|align="right " | 160<br />
|align="right " | 44<br />
|align="right " | 16<br />
|align="right " | 181<br />
|align="right " | 5670<br />
|align="right " | G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mardirossian.pdf Mardirossian &amp; Chew]<br />
|align="right " | 934.0<br />
|align="right " | 74.6%<br />
|align="right " | 799<br />
|align="right " | 210<br />
|align="right " | 80<br />
|align="right " | 30<br />
|align="right " | 133<br />
|align="right " | 471<br />
|align="right " | B 0<br />
|----<br />
|}<br />
<br />
'''Note:''' Runtime undetermined due to system hang.</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7607
2005:Audio Melody Extraction Results
2010-08-02T17:26:29Z
<p>Singh14: /* Original statistics results */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
===Goal===<br />
To extract melodic content from polyphonic audio.<br />
<br />
===Dataset===<br />
25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano<br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
| align="right" |81.8%<br />
| align="right" |17.3%<br />
| align="right" |1.85<br />
| align="right" |68.1%<br />
| align="right" |71.4%<br />
| align="right" |'''71.4%'''<br />
| align="right" |32<br />
| align="right" |R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
| align="right" |90.3%<br />
| align="right" |39.5%<br />
| align="right" |1.56<br />
| align="right" |'''68.6%'''<br />
| align="right" |'''74.1%'''<br />
| align="right" |64.3%<br />
| align="right" |10970<br />
| align="right" |L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
| align="right" |91.6%<br />
| align="right" |42.7%<br />
| align="right" |1.56<br />
| align="right" |67.3%<br />
| align="right" |73.4%<br />
| align="right" |61.1%<br />
| align="right" |5471<br />
| align="right" |B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
| align="right" |68.8%<br />
| align="right" |23.2%<br />
| align="right" |1.22<br />
| align="right" |58.5%<br />
| align="right" |62.0%<br />
| align="right" |61.1%<br />
| align="right" |45618<br />
| align="right" |Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
| align="right" |72.7%<br />
| align="right" |32.4%<br />
| align="right" |1.06<br />
| align="right" |60.1%<br />
| align="right" |67.1%<br />
| align="right" |59.5%<br />
| align="right" |12461<br />
| align="right" |F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
| align="right" |83.4%<br />
| align="right" |55.8%<br />
| align="right" |0.83<br />
| align="right" |62.7%<br />
| align="right" |66.7%<br />
| align="right" |57.8%<br />
| align="right" |44312<br />
| align="right" |G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
| align="right" |99.9% *<br />
| align="right" |99.4% *<br />
| align="right" |0.59 *<br />
| align="right" |65.8%<br />
| align="right" |71.8%<br />
| align="right" |49.9% *<br />
| align="right" |211<br />
| align="right" |F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
| align="right" |96.1% *<br />
| align="right" |93.7% *<br />
| align="right" |0.23 *<br />
| align="right" |59.8%<br />
| align="right" |67.6%<br />
| align="right" |47.9% *<br />
| align="right" |?<br />
| align="right" |G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
| align="right" |99.6% *<br />
| align="right" |96.4% *<br />
| align="right" |0.86 *<br />
| align="right" |59.6%<br />
| align="right" |71.1%<br />
| align="right" |46.4% *<br />
| align="right" |251<br />
| align="right" |G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
| align="right" |99.2% * †<br />
| align="right" |98.8% * †<br />
| align="right" |0.14 * †<br />
| align="right" |3.9% †<br />
| align="right" |8.1% †<br />
| align="right" |3.2% * †<br />
| align="right" |41<br />
| align="right" |B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
| align="right" |70.78%<br />
| align="right" |67.48%<br />
| align="right" |70.83%<br />
| align="right" |80.20%<br />
| align="right" |78.34%<br />
| align="right" |81.48%<br />
| align="right" |67.48%<br />
| align="right" |70.83%<br />
| align="right" |73.61%<br />
| align="right" |32<br />
| align="right" |R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
| align="right" |63.94%<br />
| align="right" |65.99%<br />
| align="right" |69.85%<br />
| align="right" |81.87%<br />
| align="right" |70.22%<br />
| align="right" |73.85%<br />
| align="right" |68.20%<br />
| align="right" |73.70%<br />
| align="right" |67.33%<br />
| align="right" |10970<br />
| align="right" |L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
| align="right" |60.70%<br />
| align="right" |58.16%<br />
| align="right" |61.56%<br />
| align="right" |70.92%<br />
| align="right" |70.23%<br />
| align="right" |74.06%<br />
| align="right" |58.16%<br />
| align="right" |61.56%<br />
| align="right" |63.79%<br />
| align="right" |45618<br />
| align="right" |Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
| align="right" |60.61%<br />
| align="right" |62.47%<br />
| align="right" |67.95%<br />
| align="right" |83.02%<br />
| align="right" |66.18%<br />
| align="right" |71.10%<br />
| align="right" |66.75%<br />
| align="right" |72.86%<br />
| align="right" |65.14%<br />
| align="right" |5471<br />
| align="right" |B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
| align="right" |59.18%<br />
| align="right" |58.06%<br />
| align="right" |63.13%<br />
| align="right" |73.35%<br />
| align="right" |67.53%<br />
| align="right" |72.24%<br />
| align="right" |59.76%<br />
| align="right" |66.73%<br />
| align="right" |63.25%<br />
| align="right" |12461<br />
| align="right" |F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
| align="right" |57.32%<br />
| align="right" |62.23%<br />
| align="right" |66.20%<br />
| align="right" |74.76%<br />
| align="right" |65.42%<br />
| align="right" |69.32%<br />
| align="right" |62.23%<br />
| align="right" |66.20%<br />
| align="right" |60.76%<br />
| align="right" |44312<br />
| align="right" |G<br />
|----<br />
|7<br />
|Goto, M.<br />
| align="right" |49.68%<br />
| align="right" |65.58%<br />
| align="right" |71.47%<br />
| align="right" |77.18%<br />
| align="right" |56.46%<br />
| align="right" |61.99%<br />
| align="right" |65.58%<br />
| align="right" |71.47%<br />
| align="right" |54.89%<br />
| align="right" |211<br />
| align="right" |F<br />
|----<br />
|8<br />
|Vincent, E.<br />
| align="right" |45.98%<br />
| align="right" |59.17%<br />
| align="right" |70.64%<br />
| align="right" |77.61%<br />
| align="right" |51.98%<br />
| align="right" |62.36%<br />
| align="right" |59.17%<br />
| align="right" |70.64%<br />
| align="right" |55.52%<br />
| align="right" |251<br />
| align="right" |G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
| align="right" |3.2%<br />
| align="right" |3.93%<br />
| align="right" |8.06%<br />
| align="right" |76.61%<br />
| align="right" |3.63%<br />
| align="right" |7.27%<br />
| align="right" |3.93%<br />
| align="right" |8.06%<br />
| align="right" |6.43%<br />
| align="right" |41<br />
| align="right" |B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7597
2005:Audio Melody Extraction Results
2010-08-02T17:04:15Z
<p>Singh14: /* Results */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
===Goal===<br />
To extract melodic content from polyphonic audio.<br />
<br />
===Dataset===<br />
25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano<br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
| align="right" |81.8%<br />
| align="right" |17.3%<br />
| align="right" |1.85<br />
| align="right" |68.1%<br />
| align="right" |71.4%<br />
| align="right" |'''71.4%'''<br />
| align="right" |32<br />
| align="right" |R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
| align="right" |90.3%<br />
| align="right" |39.5%<br />
| align="right" |1.56<br />
| align="right" |'''68.6%'''<br />
| align="right" |'''74.1%'''<br />
| align="right" |64.3%<br />
| align="right" |10970<br />
| align="right" |L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
| align="right" |91.6%<br />
| align="right" |42.7%<br />
| align="right" |1.56<br />
| align="right" |67.3%<br />
| align="right" |73.4%<br />
| align="right" |61.1%<br />
| align="right" |5471<br />
| align="right" |B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
| align="right" |68.8%<br />
| align="right" |23.2%<br />
| align="right" |1.22<br />
| align="right" |58.5%<br />
| align="right" |62.0%<br />
| align="right" |61.1%<br />
| align="right" |45618<br />
| align="right" |Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
| align="right" |72.7%<br />
| align="right" |32.4%<br />
| align="right" |1.06<br />
| align="right" |60.1%<br />
| align="right" |67.1%<br />
| align="right" |59.5%<br />
| align="right" |12461<br />
| align="right" |F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
| align="right" |83.4%<br />
| align="right" |55.8%<br />
| align="right" |0.83<br />
| align="right" |62.7%<br />
| align="right" |66.7%<br />
| align="right" |57.8%<br />
| align="right" |44312<br />
| align="right" |G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
| align="right" |99.9% *<br />
| align="right" |99.4% *<br />
| align="right" |0.59 *<br />
| align="right" |65.8%<br />
| align="right" |71.8%<br />
| align="right" |49.9% *<br />
| align="right" |211<br />
| align="right" |F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
| align="right" |96.1% *<br />
| align="right" |93.7% *<br />
| align="right" |0.23 *<br />
| align="right" |59.8%<br />
| align="right" |67.6%<br />
| align="right" |47.9% *<br />
| align="right" |?<br />
| align="right" |G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
| align="right" |99.6% *<br />
| align="right" |96.4% *<br />
| align="right" |0.86 *<br />
| align="right" |59.6%<br />
| align="right" |71.1%<br />
| align="right" |46.4% *<br />
| align="right" |251<br />
| align="right" |G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
| align="right" |99.2% * †<br />
| align="right" |98.8% * †<br />
| align="right" |0.14 * †<br />
| align="right" |3.9% †<br />
| align="right" |8.1% †<br />
| align="right" |3.2% * †<br />
| align="right" |41<br />
| align="right" |B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
|70.78%<br />
|67.48%<br />
|70.83%<br />
|80.20%<br />
|78.34%<br />
|81.48%<br />
|67.48%<br />
|70.83%<br />
|73.61%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
|63.94%<br />
|65.99%<br />
|69.85%<br />
|81.87%<br />
|70.22%<br />
|73.85%<br />
|68.20%<br />
|73.70%<br />
|67.33%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
|60.70%<br />
|58.16%<br />
|61.56%<br />
|70.92%<br />
|70.23%<br />
|74.06%<br />
|58.16%<br />
|61.56%<br />
|63.79%<br />
|45618<br />
|Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
|60.61%<br />
|62.47%<br />
|67.95%<br />
|83.02%<br />
|66.18%<br />
|71.10%<br />
|66.75%<br />
|72.86%<br />
|65.14%<br />
|5471<br />
|B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
|59.18%<br />
|58.06%<br />
|63.13%<br />
|73.35%<br />
|67.53%<br />
|72.24%<br />
|59.76%<br />
|66.73%<br />
|63.25%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
|57.32%<br />
|62.23%<br />
|66.20%<br />
|74.76%<br />
|65.42%<br />
|69.32%<br />
|62.23%<br />
|66.20%<br />
|60.76%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|Goto, M.<br />
|49.68%<br />
|65.58%<br />
|71.47%<br />
|77.18%<br />
|56.46%<br />
|61.99%<br />
|65.58%<br />
|71.47%<br />
|54.89%<br />
|211<br />
|F<br />
|----<br />
|8<br />
|Vincent, E.<br />
|45.98%<br />
|59.17%<br />
|70.64%<br />
|77.61%<br />
|51.98%<br />
|62.36%<br />
|59.17%<br />
|70.64%<br />
|55.52%<br />
|251<br />
|G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
|3.2%<br />
|3.93%<br />
|8.06%<br />
|76.61%<br />
|3.63%<br />
|7.27%<br />
|3.93%<br />
|8.06%<br />
|6.43%<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7596
2005:Audio Melody Extraction Results
2010-08-02T16:49:59Z
<p>Singh14: /* Introduction */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
===Goal===<br />
To extract melodic content from polyphonic audio.<br />
<br />
===Dataset===<br />
25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano<br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
|81.8%<br />
|17.3%<br />
|1.85<br />
|68.1%<br />
|71.4%<br />
!71.4%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
|90.3%<br />
|39.5%<br />
|1.56<br />
!68.6%<br />
!74.1%<br />
|64.3%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
|91.6%<br />
|42.7%<br />
|1.56<br />
|67.3%<br />
|73.4%<br />
|61.1%<br />
|5471<br />
|B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
|68.8%<br />
|23.2%<br />
|1.22<br />
|58.5%<br />
|62.0%<br />
|61.1%<br />
|45618<br />
|Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
|72.7%<br />
|32.4%<br />
|1.06<br />
|60.1%<br />
|67.1%<br />
|59.5%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
|83.4%<br />
|55.8%<br />
|0.83<br />
|62.7%<br />
|66.7%<br />
|57.8%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
|99.9% *<br />
|99.4% *<br />
|0.59 *<br />
|65.8%<br />
|71.8%<br />
|49.9% *<br />
|211<br />
|F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
|96.1% *<br />
|93.7% *<br />
|0.23 *<br />
|59.8%<br />
|67.6%<br />
|47.9% *<br />
|?<br />
|G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
|99.6% *<br />
|96.4% *<br />
|0.86 *<br />
|59.6%<br />
|71.1%<br />
|46.4% *<br />
|251<br />
|G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
|99.2% * †<br />
|98.8% * †<br />
|0.14 * †<br />
|3.9% †<br />
|8.1% †<br />
|3.2% * †<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
|70.78%<br />
|67.48%<br />
|70.83%<br />
|80.20%<br />
|78.34%<br />
|81.48%<br />
|67.48%<br />
|70.83%<br />
|73.61%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
|63.94%<br />
|65.99%<br />
|69.85%<br />
|81.87%<br />
|70.22%<br />
|73.85%<br />
|68.20%<br />
|73.70%<br />
|67.33%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
|60.70%<br />
|58.16%<br />
|61.56%<br />
|70.92%<br />
|70.23%<br />
|74.06%<br />
|58.16%<br />
|61.56%<br />
|63.79%<br />
|45618<br />
|Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
|60.61%<br />
|62.47%<br />
|67.95%<br />
|83.02%<br />
|66.18%<br />
|71.10%<br />
|66.75%<br />
|72.86%<br />
|65.14%<br />
|5471<br />
|B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
|59.18%<br />
|58.06%<br />
|63.13%<br />
|73.35%<br />
|67.53%<br />
|72.24%<br />
|59.76%<br />
|66.73%<br />
|63.25%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
|57.32%<br />
|62.23%<br />
|66.20%<br />
|74.76%<br />
|65.42%<br />
|69.32%<br />
|62.23%<br />
|66.20%<br />
|60.76%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|Goto, M.<br />
|49.68%<br />
|65.58%<br />
|71.47%<br />
|77.18%<br />
|56.46%<br />
|61.99%<br />
|65.58%<br />
|71.47%<br />
|54.89%<br />
|211<br />
|F<br />
|----<br />
|8<br />
|Vincent, E.<br />
|45.98%<br />
|59.17%<br />
|70.64%<br />
|77.61%<br />
|51.98%<br />
|62.36%<br />
|59.17%<br />
|70.64%<br />
|55.52%<br />
|251<br />
|G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
|3.2%<br />
|3.93%<br />
|8.06%<br />
|76.61%<br />
|3.63%<br />
|7.27%<br />
|3.93%<br />
|8.06%<br />
|6.43%<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7595
2005:Audio Melody Extraction Results
2010-08-02T16:49:45Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
===Goal===<br />
To extract melodic content from polyphonic audio.<br />
<br />
====Dataset===<br />
25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano<br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
|81.8%<br />
|17.3%<br />
|1.85<br />
|68.1%<br />
|71.4%<br />
!71.4%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
|90.3%<br />
|39.5%<br />
|1.56<br />
!68.6%<br />
!74.1%<br />
|64.3%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
|91.6%<br />
|42.7%<br />
|1.56<br />
|67.3%<br />
|73.4%<br />
|61.1%<br />
|5471<br />
|B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
|68.8%<br />
|23.2%<br />
|1.22<br />
|58.5%<br />
|62.0%<br />
|61.1%<br />
|45618<br />
|Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
|72.7%<br />
|32.4%<br />
|1.06<br />
|60.1%<br />
|67.1%<br />
|59.5%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
|83.4%<br />
|55.8%<br />
|0.83<br />
|62.7%<br />
|66.7%<br />
|57.8%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
|99.9% *<br />
|99.4% *<br />
|0.59 *<br />
|65.8%<br />
|71.8%<br />
|49.9% *<br />
|211<br />
|F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
|96.1% *<br />
|93.7% *<br />
|0.23 *<br />
|59.8%<br />
|67.6%<br />
|47.9% *<br />
|?<br />
|G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
|99.6% *<br />
|96.4% *<br />
|0.86 *<br />
|59.6%<br />
|71.1%<br />
|46.4% *<br />
|251<br />
|G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
|99.2% * †<br />
|98.8% * †<br />
|0.14 * †<br />
|3.9% †<br />
|8.1% †<br />
|3.2% * †<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
|70.78%<br />
|67.48%<br />
|70.83%<br />
|80.20%<br />
|78.34%<br />
|81.48%<br />
|67.48%<br />
|70.83%<br />
|73.61%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
|63.94%<br />
|65.99%<br />
|69.85%<br />
|81.87%<br />
|70.22%<br />
|73.85%<br />
|68.20%<br />
|73.70%<br />
|67.33%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
|60.70%<br />
|58.16%<br />
|61.56%<br />
|70.92%<br />
|70.23%<br />
|74.06%<br />
|58.16%<br />
|61.56%<br />
|63.79%<br />
|45618<br />
|Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
|60.61%<br />
|62.47%<br />
|67.95%<br />
|83.02%<br />
|66.18%<br />
|71.10%<br />
|66.75%<br />
|72.86%<br />
|65.14%<br />
|5471<br />
|B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
|59.18%<br />
|58.06%<br />
|63.13%<br />
|73.35%<br />
|67.53%<br />
|72.24%<br />
|59.76%<br />
|66.73%<br />
|63.25%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
|57.32%<br />
|62.23%<br />
|66.20%<br />
|74.76%<br />
|65.42%<br />
|69.32%<br />
|62.23%<br />
|66.20%<br />
|60.76%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|Goto, M.<br />
|49.68%<br />
|65.58%<br />
|71.47%<br />
|77.18%<br />
|56.46%<br />
|61.99%<br />
|65.58%<br />
|71.47%<br />
|54.89%<br />
|211<br />
|F<br />
|----<br />
|8<br />
|Vincent, E.<br />
|45.98%<br />
|59.17%<br />
|70.64%<br />
|77.61%<br />
|51.98%<br />
|62.36%<br />
|59.17%<br />
|70.64%<br />
|55.52%<br />
|251<br />
|G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
|3.2%<br />
|3.93%<br />
|8.06%<br />
|76.61%<br />
|3.63%<br />
|7.27%<br />
|3.93%<br />
|8.06%<br />
|6.43%<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Genre_Classification_Results&diff=7594
2005:Audio Genre Classification Results
2010-08-02T16:41:23Z
<p>Singh14: /* USPOP Dataset */</p>
<hr />
<div>==Introduction==<br />
<br />
===Goal===<br />
To classify polyphonic music audio (in PCM format) into genre categories. <br />
<br />
===Dataset===<br />
Two sets of data were used: Magnatune and USPOP. The Magnatune dataset has a hierarchical genre taxonomy, while the USPOP categories are at a single level. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table: <br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files <br />
|-<br />
| Magnatune || 34.3 GB || 1005 || 510 <br />
|-<br />
| USPOP || 28.4 GB || 940 || 474 <br />
|-<br />
|}<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Hierarchical Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande & Eck (2)] || 82.34%<br />
|-<br />
| 2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande & Eck (1)] || 81.77%<br />
|-<br />
| 3 || [https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 78.81%<br />
|- <br />
| 4 || [https://www.music-ir.org/mirex/abstracts/2005/west.pdf West, K.] || 75.29%<br />
|- <br />
| 5 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (SSD+RH)] || 75.27% <br />
|-<br />
| 6 || [https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 75.14% <br />
|-<br />
| 7 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (RP+SSD)] || 74.78% <br />
|-<br />
| 8 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (RP+SSD+RH)] || 74.58% <br />
|-<br />
| 9 || [https://www.music-ir.org/mirex/abstracts/2005/scaringella.pdf Scaringella, N.] || 73.11% <br />
|-<br />
| 10 || [https://www.music-ir.org/mirex/abstracts/2005/ahrendt.pdf Ahrendt, P.] || 71.55% <br />
|-<br />
| 11 || [https://www.music-ir.org/mirex/abstracts/2005/burred.pdf Burred, J.] || 62.63% <br />
|-<br />
| 12 || [https://www.music-ir.org/mirex/abstracts/2005/soares.pdf Soares, V.] || 60.98% <br />
|-<br />
| 13 || [https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 60.72% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | Magnatune Dataset <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Hierarchical Classification Accuracy !! Normalized Hierarchical Classification Accuracy !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files <br />
|-<br />
| 1 || Bergstra, Casagrande & Eck (2) || 77.75% || 73.04% || 75.10% || 69.49% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_2_MTeval.txt BCE_2_MTeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande & Eck (1) || 77.25% || 72.13% || 74.71% || 68.73% || 23400 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 3 || Mandel & Ellis ||71.96%|| 69.63%|| 67.65% ||63.99%|| 8729 ||R ||[https://www.music-ir.org/mirex/results/2005/audio-genre/ME_MTeval.txt ME_MTeval.txt] <br />
|-<br />
| 4 ||West, K.|| 71.67%|| 68.33% ||68.43%||63.87%||43327 ||B4|| [https://www.music-ir.org/mirex/results/2005/audio-genre/W_MTeval.txt W_MTeval.txt] <br />
|-<br />
| 5 || Lidy & Rauber (RP+SSD) || 71.08% || 70.90%|| 67.65%||66.85%||6372||B1||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD_MTeval.txt LR_RP+SSD_MTeval.txt] <br />
|-<br />
| 6 || Lidy & Rauber (RP+SSD+RH) || 70.88% ||70.52% ||67.25% ||66.27% ||6372 ||B1 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD+RH_MTeval.txt LR_RP+SSD+RH_MTeval.txt] <br />
|-<br />
| 7 || Lidy & Rauber (SSD+RH) ||70.78% ||69.31% ||67.65% ||65.54% || 6372 || B1 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_SSD+RH_MTeval.txt LR_SSD+RH_MTeval.txt]<br />
|-<br />
| 8 || Scaringella, N. || 70.47% || 72.30%|| 66.14%|| 67.12%|| 22740|| G|| [https://www.music-ir.org/mirex/results/2005/audio-genre/SN_MTeval.txt SN_MTeval.txt]<br />
|-<br />
| 9 || Pampalk, E. || 69.90% || 70.91% || 66.47% || 66.26% || 3312 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/P_MTeval.txt P_MTeval.txt]<br />
|-<br />
| 10 || Ahrendt, P. || 64.61% || 61.40% || 60.98% || 57.15% || 4920 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-genre/A_MTeval.txt A_MTeval.txt]<br />
|-<br />
| 11 || Burred, J. || 59.22% || 61.96% || 54.12% || 55.68% || 12483 || B2 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/B_MTeval.txt B_MTeval.txt]<br />
|-<br />
| 12 || Tzanetakis, G. || 58.14% || 53.47% || 55.49% || 50.39% || 1312 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/T_MTeval.txt T_MTeval.txt]<br />
|-<br />
| 13 || Soares, V. || 55.29% || 60.73% || 49.41% || 53.54% || 23880 || Y ||[https://www.music-ir.org/mirex/results/2005/audio-genre/SV_MTeval.txt SV_MTeval.txt] <br />
|-<br />
| 14 || Li, M. || TO * || -- || -- || -- || -- || -- || -- <br />
|-<br />
| 15 || Chen & Gao || DNC * || -- || -- || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset<br />
|-style="background: yellow;"<br />
|----<br />
!Rank<br />
!Participant<br />
!Raw Classification Accuracy<br />
!Normalized Raw Classification Accuracy<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|Bergstra, Casagrande &amp; Eck (2)<br />
|86.92%<br />
|82.91%<br />
|<br />
|<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_2_USeval.txt BCE_2_USeval.txt]<br />
|----<br />
|2<br />
|Bergstra, Casagrande &amp; Eck (1)<br />
|86.29%<br />
|82.50%<br />
|23400<br />
|B0<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_1_USeval.txt BCE_1_USeval.txt]<br />
|----<br />
|3<br />
|Mandel &amp; Ellis<br />
|85.65%<br />
|76.91%<br />
|7856<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/ME_USeval.txt ME_USeval.txt]<br />
|----<br />
|4<br />
|Pampalk, E.<br />
|80.38%<br />
|78.74%<br />
|3090<br />
|B0<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/P_USeval.txt P_USeval.txt]<br />
|----<br />
|5<br />
|Lidy &amp; Rauber (SSD+RH)<br />
|79.75%<br />
|75.45%<br />
|5164<br />
|B1<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_SSD+RH_USeval.txt LR_SSD+RH_USeval.txt]<br />
|----<br />
|6<br />
|West, K.<br />
|78.90%<br />
|74.67%<br />
|18557<br />
|B4<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/W_USeval.txt W_USeval.txt]<br />
|----<br />
|7<br />
|Lidy &amp; Rauber (RP+SSD)<br />
|78.48%<br />
|77.62%<br />
|5164<br />
|B1<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD_USeval.txt LR_RP+SSD_USeval.txt]<br />
|----<br />
|8<br />
|Ahrendt, P.<br />
|78.48%<br />
|73.23%<br />
|9702<br />
|B1<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/A_USeval.txt A_USeval.txt]<br />
|----<br />
|9<br />
|Lidy &amp; Rauber (RP+SSD+RH)<br />
|78.27%<br />
|76.84%<br />
|5194<br />
|B1<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD+RH_USeval.txt LR_RP+SSD+RH_USeval.txt]<br />
|----<br />
|10<br />
|Scaringella, N.<br />
|75.74%<br />
|77.67%<br />
|24606<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/SN_USeval.txt SN_USeval.txt]<br />
|----<br />
|11<br />
|Soares, V.<br />
|66.67%<br />
|67.28%<br />
|14369<br />
|Y<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/SV_USeval.txt SV_USeval.txt]<br />
|----<br />
|12<br />
|Burred, J.<br />
|66.03%<br />
|72.50%<br />
|9233<br />
|B2<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/B_USeval.txt B_USeval.txt]<br />
|----<br />
|13<br />
|Tzanetakis, G.<br />
|63.29%<br />
|50.19%<br />
|1320<br />
|B0<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/T_USeval.txt T_USeval.txt]<br />
|----<br />
|14<br />
|Chen &amp; Gao<br />
|22.93%<br />
|17.96%<br />
|N/A<br />
|Y<br />
|[https://www.music-ir.org/mirex/results/2005/audio-genre/CG_USeval.txt CG_USeval.txt]<br />
|----<br />
|15<br />
|Li, M.<br />
|TO *<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Genre_Classification_Results&diff=7593
2005:Audio Genre Classification Results
2010-08-02T16:36:15Z
<p>Singh14: /* Magnatune Dataset */</p>
<hr />
<div>==Introduction==<br />
<br />
===Goal===<br />
To classify polyphonic music audio (in PCM format) into genre categories. <br />
<br />
===Dataset===<br />
Two sets of data were used: Magnatune and USPOP. The Magnatune dataset has a hierarchical genre taxonomy, while the USPOP categories are at a single level. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table: <br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files <br />
|-<br />
| Magnatune || 34.3 GB || 1005 || 510 <br />
|-<br />
| USPOP || 28.4 GB || 940 || 474 <br />
|-<br />
|}<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Hierarchical Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande & Eck (2)] || 82.34%<br />
|-<br />
| 2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande & Eck (1)] || 81.77%<br />
|-<br />
| 3 || [https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 78.81%<br />
|- <br />
| 4 || [https://www.music-ir.org/mirex/abstracts/2005/west.pdf West, K.] || 75.29%<br />
|- <br />
| 5 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (SSD+RH)] || 75.27% <br />
|-<br />
| 6 || [https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 75.14% <br />
|-<br />
| 7 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (RP+SSD)] || 74.78% <br />
|-<br />
| 8 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (RP+SSD+RH)] || 74.58% <br />
|-<br />
| 9 || [https://www.music-ir.org/mirex/abstracts/2005/scaringella.pdf Scaringella, N.] || 73.11% <br />
|-<br />
| 10 || [https://www.music-ir.org/mirex/abstracts/2005/ahrendt.pdf Ahrendt, P.] || 71.55% <br />
|-<br />
| 11 || [https://www.music-ir.org/mirex/abstracts/2005/burred.pdf Burred, J.] || 62.63% <br />
|-<br />
| 12 || [https://www.music-ir.org/mirex/abstracts/2005/soares.pdf Soares, V.] || 60.98% <br />
|-<br />
| 13 || [https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 60.72% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | Magnatune Dataset <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Hierarchical Classification Accuracy !! Normalized Hierarchical Classification Accuracy !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files <br />
|-<br />
| 1 || Bergstra, Casagrande & Eck (2) || 77.75% || 73.04% || 75.10% || 69.49% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_2_MTeval.txt BCE_2_MTeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande & Eck (1) || 77.25% || 72.13% || 74.71% || 68.73% || 23400 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 3 || Mandel & Ellis ||71.96%|| 69.63%|| 67.65% ||63.99%|| 8729 ||R ||[https://www.music-ir.org/mirex/results/2005/audio-genre/ME_MTeval.txt ME_MTeval.txt] <br />
|-<br />
| 4 ||West, K.|| 71.67%|| 68.33% ||68.43%||63.87%||43327 ||B4|| [https://www.music-ir.org/mirex/results/2005/audio-genre/W_MTeval.txt W_MTeval.txt] <br />
|-<br />
| 5 || Lidy & Rauber (RP+SSD) || 71.08% || 70.90%|| 67.65%||66.85%||6372||B1||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD_MTeval.txt LR_RP+SSD_MTeval.txt] <br />
|-<br />
| 6 || Lidy & Rauber (RP+SSD+RH) || 70.88% ||70.52% ||67.25% ||66.27% ||6372 ||B1 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD+RH_MTeval.txt LR_RP+SSD+RH_MTeval.txt] <br />
|-<br />
| 7 || Lidy & Rauber (SSD+RH) ||70.78% ||69.31% ||67.65% ||65.54% || 6372 || B1 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_SSD+RH_MTeval.txt LR_SSD+RH_MTeval.txt]<br />
|-<br />
| 8 || Scaringella, N. || 70.47% || 72.30%|| 66.14%|| 67.12%|| 22740|| G|| [https://www.music-ir.org/mirex/results/2005/audio-genre/SN_MTeval.txt SN_MTeval.txt]<br />
|-<br />
| 9 || Pampalk, E. || 69.90% || 70.91% || 66.47% || 66.26% || 3312 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/P_MTeval.txt P_MTeval.txt]<br />
|-<br />
| 10 || Ahrendt, P. || 64.61% || 61.40% || 60.98% || 57.15% || 4920 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-genre/A_MTeval.txt A_MTeval.txt]<br />
|-<br />
| 11 || Burred, J. || 59.22% || 61.96% || 54.12% || 55.68% || 12483 || B2 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/B_MTeval.txt B_MTeval.txt]<br />
|-<br />
| 12 || Tzanetakis, G. || 58.14% || 53.47% || 55.49% || 50.39% || 1312 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/T_MTeval.txt T_MTeval.txt]<br />
|-<br />
| 13 || Soares, V. || 55.29% || 60.73% || 49.41% || 53.54% || 23880 || Y ||[https://www.music-ir.org/mirex/results/2005/audio-genre/SV_MTeval.txt SV_MTeval.txt] <br />
|-<br />
| 14 || Li, M. || TO * || -- || -- || -- || -- || -- || -- <br />
|-<br />
| 15 || Chen & Gao || DNC * || -- || -- || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset<br />
|-style="background: yellow;"<br />
|----<br />
!Rank<br />
!Participant<br />
!Raw Classification Accuracy<br />
!Normalized Raw Classification Accuracy<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|Bergstra, Casagrande &amp; Eck (2)<br />
|86.92%<br />
|82.91%<br />
|<br />
|<br />
|BCE_2_USeval.txt<br />
|----<br />
|2<br />
|Bergstra, Casagrande &amp; Eck (1)<br />
|86.29%<br />
|82.50%<br />
|23400<br />
|B0<br />
|BCE_1_USeval.txt<br />
|----<br />
|3<br />
|Mandel &amp; Ellis<br />
|85.65%<br />
|76.91%<br />
|7856<br />
|R<br />
|ME_USeval.txt<br />
|----<br />
|4<br />
|Pampalk, E.<br />
|80.38%<br />
|78.74%<br />
|3090<br />
|B0<br />
|P_USeval.txt<br />
|----<br />
|5<br />
|Lidy &amp; Rauber (SSD+RH)<br />
|79.75%<br />
|75.45%<br />
|5164<br />
|B1<br />
|LR_SSD+RH_USeval.txt<br />
|----<br />
|6<br />
|West, K.<br />
|78.90%<br />
|74.67%<br />
|18557<br />
|B4<br />
|W_USeval.txt<br />
|----<br />
|7<br />
|Lidy &amp; Rauber (RP+SSD)<br />
|78.48%<br />
|77.62%<br />
|5164<br />
|B1<br />
|LR_RP+SSD_USeval.txt<br />
|----<br />
|8<br />
|Ahrendt, P.<br />
|78.48%<br />
|73.23%<br />
|9702<br />
|B1<br />
|A_USeval.txt<br />
|----<br />
|9<br />
|Lidy &amp; Rauber (RP+SSD+RH)<br />
|78.27%<br />
|76.84%<br />
|5194<br />
|B1<br />
|LR_RP+SSD+RH_USeval.txt<br />
|----<br />
|10<br />
|Scaringella, N.<br />
|75.74%<br />
|77.67%<br />
|24606<br />
|G<br />
|SN_USeval.txt<br />
|----<br />
|11<br />
|Soares, V.<br />
|66.67%<br />
|67.28%<br />
|14369<br />
|Y<br />
|SV_USeval.txt<br />
|----<br />
|12<br />
|Burred, J.<br />
|66.03%<br />
|72.50%<br />
|9233<br />
|B2<br />
|B_USeval.txt<br />
|----<br />
|13<br />
|Tzanetakis, G.<br />
|63.29%<br />
|50.19%<br />
|1320<br />
|B0<br />
|T_USeval.txt<br />
|----<br />
|14<br />
|Chen &amp; Gao<br />
|22.93%<br />
|17.96%<br />
|N/A<br />
|Y<br />
|CG_USeval.txt<br />
|----<br />
|15<br />
|Li, M.<br />
|TO *<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Genre_Classification_Results&diff=7592
2005:Audio Genre Classification Results
2010-08-02T16:35:33Z
<p>Singh14: /* Magnatune Dataset */</p>
<hr />
<div>==Introduction==<br />
<br />
===Goal===<br />
To classify polyphonic music audio (in PCM format) into genre categories. <br />
<br />
===Dataset===<br />
Two sets of data were used: Magnatune and USPOP. The Magnatune dataset has a hierarchical genre taxonomy, while the USPOP categories are at a single level. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table: <br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files <br />
|-<br />
| Magnatune || 34.3 GB || 1005 || 510 <br />
|-<br />
| USPOP || 28.4 GB || 940 || 474 <br />
|-<br />
|}<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Hierarchical Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande & Eck (2)] || 82.34%<br />
|-<br />
| 2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande & Eck (1)] || 81.77%<br />
|-<br />
| 3 || [https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 78.81%<br />
|- <br />
| 4 || [https://www.music-ir.org/mirex/abstracts/2005/west.pdf West, K.] || 75.29%<br />
|- <br />
| 5 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (SSD+RH)] || 75.27% <br />
|-<br />
| 6 || [https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 75.14% <br />
|-<br />
| 7 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (RP+SSD)] || 74.78% <br />
|-<br />
| 8 || [https://www.music-ir.org/mirex/abstracts/2005/lidy.pdf Lidy & Rauber (RP+SSD+RH)] || 74.58% <br />
|-<br />
| 9 || [https://www.music-ir.org/mirex/abstracts/2005/scaringella.pdf Scaringella, N.] || 73.11% <br />
|-<br />
| 10 || [https://www.music-ir.org/mirex/abstracts/2005/ahrendt.pdf Ahrendt, P.] || 71.55% <br />
|-<br />
| 11 || [https://www.music-ir.org/mirex/abstracts/2005/burred.pdf Burred, J.] || 62.63% <br />
|-<br />
| 12 || [https://www.music-ir.org/mirex/abstracts/2005/soares.pdf Soares, V.] || 60.98% <br />
|-<br />
| 13 || [https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 60.72% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | Magnatune Dataset <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Hierarchical Classification Accuracy !! Normalized Hierarchical Classification Accuracy !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files <br />
|-<br />
| 1 || Bergstra, Casagrande & Eck (2) || 77.75% || 73.04% || 75.10% || 69.49% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_2_MTeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande & Eck (1) || 77.25% || 72.13% || 74.71% || 68.73% || 23400 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 3 || Mandel & Ellis ||71.96%|| 69.63%|| 67.65% ||63.99%|| 8729 ||R ||[https://www.music-ir.org/mirex/results/2005/audio-genre/ME_MTeval.txt ME_MTeval.txt] <br />
|-<br />
| 4 ||West, K.|| 71.67%|| 68.33% ||68.43%||63.87%||43327 ||B4|| [https://www.music-ir.org/mirex/results/2005/audio-genre/W_MTeval.txt W_MTeval.txt] <br />
|-<br />
| 5 || Lidy & Rauber (RP+SSD) || 71.08% || 70.90%|| 67.65%||66.85%||6372||B1||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD_MTeval.txt LR_RP+SSD_MTeval.txt] <br />
|-<br />
| 6 || Lidy & Rauber (RP+SSD+RH) || 70.88% ||70.52% ||67.25% ||66.27% ||6372 ||B1 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_RP+SSD+RH_MTeval.txt LR_RP+SSD+RH_MTeval.txt] <br />
|-<br />
| 7 || Lidy & Rauber (SSD+RH) ||70.78% ||69.31% ||67.65% ||65.54% || 6372 || B1 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/LR_SSD+RH_MTeval.txt LR_SSD+RH_MTeval.txt]<br />
|-<br />
| 8 || Scaringella, N. || 70.47% || 72.30%|| 66.14%|| 67.12%|| 22740|| G|| [https://www.music-ir.org/mirex/results/2005/audio-genre/SN_MTeval.txt SN_MTeval.txt]<br />
|-<br />
| 9 || Pampalk, E. || 69.90% || 70.91% || 66.47% || 66.26% || 3312 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/P_MTeval.txt P_MTeval.txt]<br />
|-<br />
| 10 || Ahrendt, P. || 64.61% || 61.40% || 60.98% || 57.15% || 4920 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-genre/A_MTeval.txt A_MTeval.txt]<br />
|-<br />
| 11 || Burred, J. || 59.22% || 61.96% || 54.12% || 55.68% || 12483 || B2 ||[https://www.music-ir.org/mirex/results/2005/audio-genre/B_MTeval.txt B_MTeval.txt]<br />
|-<br />
| 12 || Tzanetakis, G. || 58.14% || 53.47% || 55.49% || 50.39% || 1312 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-genre/T_MTeval.txt T_MTeval.txt]<br />
|-<br />
| 13 || Soares, V. || 55.29% || 60.73% || 49.41% || 53.54% || 23880 || Y ||[https://www.music-ir.org/mirex/results/2005/audio-genre/SV_MTeval.txt SV_MTeval.txt] <br />
|-<br />
| 14 || Li, M. || TO * || -- || -- || -- || -- || -- || -- <br />
|-<br />
| 15 || Chen & Gao || DNC * || -- || -- || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset<br />
|-style="background: yellow;"<br />
|----<br />
!Rank<br />
!Participant<br />
!Raw Classification Accuracy<br />
!Normalized Raw Classification Accuracy<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|Bergstra, Casagrande &amp; Eck (2)<br />
|86.92%<br />
|82.91%<br />
|<br />
|<br />
|BCE_2_USeval.txt<br />
|----<br />
|2<br />
|Bergstra, Casagrande &amp; Eck (1)<br />
|86.29%<br />
|82.50%<br />
|23400<br />
|B0<br />
|BCE_1_USeval.txt<br />
|----<br />
|3<br />
|Mandel &amp; Ellis<br />
|85.65%<br />
|76.91%<br />
|7856<br />
|R<br />
|ME_USeval.txt<br />
|----<br />
|4<br />
|Pampalk, E.<br />
|80.38%<br />
|78.74%<br />
|3090<br />
|B0<br />
|P_USeval.txt<br />
|----<br />
|5<br />
|Lidy &amp; Rauber (SSD+RH)<br />
|79.75%<br />
|75.45%<br />
|5164<br />
|B1<br />
|LR_SSD+RH_USeval.txt<br />
|----<br />
|6<br />
|West, K.<br />
|78.90%<br />
|74.67%<br />
|18557<br />
|B4<br />
|W_USeval.txt<br />
|----<br />
|7<br />
|Lidy &amp; Rauber (RP+SSD)<br />
|78.48%<br />
|77.62%<br />
|5164<br />
|B1<br />
|LR_RP+SSD_USeval.txt<br />
|----<br />
|8<br />
|Ahrendt, P.<br />
|78.48%<br />
|73.23%<br />
|9702<br />
|B1<br />
|A_USeval.txt<br />
|----<br />
|9<br />
|Lidy &amp; Rauber (RP+SSD+RH)<br />
|78.27%<br />
|76.84%<br />
|5194<br />
|B1<br />
|LR_RP+SSD+RH_USeval.txt<br />
|----<br />
|10<br />
|Scaringella, N.<br />
|75.74%<br />
|77.67%<br />
|24606<br />
|G<br />
|SN_USeval.txt<br />
|----<br />
|11<br />
|Soares, V.<br />
|66.67%<br />
|67.28%<br />
|14369<br />
|Y<br />
|SV_USeval.txt<br />
|----<br />
|12<br />
|Burred, J.<br />
|66.03%<br />
|72.50%<br />
|9233<br />
|B2<br />
|B_USeval.txt<br />
|----<br />
|13<br />
|Tzanetakis, G.<br />
|63.29%<br />
|50.19%<br />
|1320<br />
|B0<br />
|T_USeval.txt<br />
|----<br />
|14<br />
|Chen &amp; Gao<br />
|22.93%<br />
|17.96%<br />
|N/A<br />
|Y<br />
|CG_USeval.txt<br />
|----<br />
|15<br />
|Li, M.<br />
|TO *<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Drum_Detection_Results&diff=7590
2005:Audio Drum Detection Results
2010-08-02T16:26:42Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
===Goal===<br />
To detect the occurrences of drum events in polyphonic audio. <br />
<br />
===Dataset===<br />
At least 50 files of both live and sequenced music, with many genres encompassed and various degrees of drum audio contained in the files. Three collections of music were used: Christian Dittmar (CD), Koen Tanghe (KT) and Masataka Goto (MG). Participants were evaluated against music from each individual collection, and then the three collection scores are averaged to produce a composite score.<br />
<br />
==Results==<br />
<br />
===Overall Collections===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="11" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Total Average Classification F-measure !! Total Overall Onset Precision !! Total Overall Onset Recall !! Total Overall Onset F-measure !! BD Average F-measure !! HH Average F-measure !! SD Average F-measure !! Runtime (s) !! Machine <br />
|-<br />
| 1 || [https://www.music-ir.org/mirex/abstracts/2005/yoshii.pdf Yoshii, Goto, & Okuno] || 0.670 || 64.92% || 67.02% || 0.659 || 0.728 || 0.574 || 0.702 || 8534 || B 0 <br />
|-<br />
| 2 || [https://www.music-ir.org/mirex/abstracts/2005/tanghe.pdf Tanghe, Degroeve, & De Baets 3] || 0.611 || 63.30% || 71.19% || 0.670 || 0.688 || 0.601 || 0.555 || 1337 || Y <br />
|-<br />
| 3 || [https://www.music-ir.org/mirex/abstracts/2005/tanghe.pdf Tanghe, Degroeve, & De Baets 4] || 0.609 || 62.57% || 71.09% || 0.666 || 0.686 || 0.590 || 0.562 || 1342 || Y <br />
|-<br />
| 4 || [https://www.music-ir.org/mirex/abstracts/2005/tanghe.pdf Tanghe, Degroeve, & De Baets 1] || 0.599 || 60.02% || 72.45% || 0.657 || 0.677 || 0.588 || 0.542 || 1350 || Y <br />
|-<br />
| 5 || [https://www.music-ir.org/mirex/abstracts/2005/dittmar.pdf Dittmar, C.] || 0.588 || 65.68% || 63.38% || 0.645 || 0.606 || 0.585 || 0.581 || 673 || R <br />
|-<br />
| 6 || [https://www.music-ir.org/mirex/abstracts/2005/paulus.pdf Paulus, J.] || 0.499 || 59.61% || 64.86% || 0.621 || 0.527 || 0.587 || 0.430 || 1137 || L <br />
|-<br />
| 7 || [https://www.music-ir.org/mirex/abstracts/2005/gillet.pdf Gillet & Richard 2] || 0.443 || 77.09% || 40.63% || 0.532 || 0.598 || 0.334 || 0.428 || 21248 || F <br />
|-<br />
| 8 || [https://www.music-ir.org/mirex/abstracts/2005/gillet.pdf Gillet & Richard 1] || 0.391 || 69.84% || 37.98% || 0.492 || 0.533 || 0.343 || 0.317 || 21997 || F <br />
|-<br />
|}<br />
<br />
===Christan Dittmar Collection===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | CHRISTIAN DITTMAR COLLECTION <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Total Average Classification F-measure !! Total Overall Onset Precision !! Total Overall Onset Recall !! Total Overall Onset F-measure !! BD Average F-measure !! HH Average F-measure !! SD Average F-measure <br />
|-<br />
|1 || Dittmar, C. || 0.753 || 77.73% || 72.56% || 0.751 || 0.783 || 0.696 || 0.790 <br />
|-<br />
|2 || Yoshii, Goto, & Okuno || 0.690 || 64.25% || 62.75% || 0.660 || 0.714 || 0.533 || 0.811 <br />
|-<br />
|3 || Tanghe, Degroeve, & De Baets 3 || 0.595 || 61.85% || 64.85% || 0.633 || 0.685 || 0.568 || 0.548 <br />
|-<br />
|4 || Tanghe, Degroeve, & De Baets 4 || 0.589 || 62.45% || 64.22% || 0.628 || 0.668 || 0.555 || 0.559 <br />
|-<br />
|5 || Tanghe, Degroeve, & De Baets 1 || 0.580 || 57.78% || 65.94% || 0.616 || 0.669 || 0.553 || 0.533 <br />
|-<br />
|6 || Paulus, J. || 0.440 || 55.82% || 54.36% || 0.551 || 0.430 || 0.497 || 0.424 <br />
|-<br />
|7 || Gillet & Richard 2 || 0.401 || 77.22% || 30.16% || 0.434 || 0.658 || 0.156 || 0.364 <br />
|-<br />
|8 || Gillet & Richard 1 || 0.339 || 66.33% || 30.57% || 0.418 || 0.466 || 0.279 || 0.265 <br />
|-<br />
|}<br />
<br />
===Koen Tanghe Collection===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | KOEN TANGHE COLLECTION<br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Total Average Classification F-measure !! Total Overall Onset Precision !! Total Overall Onset Recall !! Total Overall Onset F-measure !! BD Average F-measure !! HH Average F-measure !! SD Average F-measure<br />
|-<br />
|1 || Yoshii, Goto, & Okuno || 0.617 || 53.06% || 66.30% || 0.589 || 0.686 || 0.481 || 0.652 <br />
|-<br />
|2 || Tanghe, Degroeve, & De Baets 3 || 0.546 || 52.67% || 68.21% || 0.594 || 0.613 || 0.485 || 0.525 <br />
|-<br />
|3 || Tanghe, Degroeve, & De Baets 4 || 0.541 || 51.90% || 67.98% || 0.589 || 0.612 || 0.479 || 0.523 <br />
|-<br />
|4 || Tanghe, Degroeve, & De Baets 1 || 0.533 || 47.48% || 69.24% || 0.577 || 0.602 || 0.467 || 0.511 <br />
|-<br />
|5 || Dittmar, C. || 0.494 || 51.20% || 51.51% || 0.514 || 0.509 || 0.418 || 0.535 <br />
|-<br />
|6 || Paulus, J. || 0.425 || 55.82% || 55.10% || 0.555 || 0.444 || 0.489 || 0.412 <br />
|-<br />
|7 || Gillet & Richard 2 || 0.273 || 66.68% || 28.22% || 0.397 || 0.393 || 0.178 || 0.269 <br />
|-<br />
|8 || Gillet & Richard 1 || 0.259 || 58.48% || 26.11% || 0.361 || 0.375 || 0.196|| 0.210 <br />
|-<br />
|}<br />
<br />
===Masataka Goto Collection===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | MASATAKA GOTO COLLECTION (50 songs from RWC Music Database: RWC-MDB-P-2001) <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Total Average Classification F-measure !! Total Overall Onset Precision !! Total Overall Onset Recall !! Total Overall Onset F-measure !! BD Average F-measure !! HH Average F-measure !! SD Average F-measure <br />
|-<br />
|1 || Yoshii, Goto, & Okuno || 0.716 || 76.13% || 69.16% || 0.725 || 0.776 || 0.661 || 0.710 <br />
|-<br />
|2 || Tanghe, Degroeve, & De Baets 4 || 0.685 || 72.32% || 75.83% || 0.740 || 0.766 || 0.691 || 0.599 <br />
|-<br />
|3 || Tanghe, Degroeve, & De Baets 3 || 0.683 || 72.74% || 75.65% || 0.742 || 0.763 || 0.701 || 0.585 <br />
|-<br />
|4 || Tanghe, Degroeve, & De Baets 1 || 0.673 || 69.86% || 77.12% || 0.733 || 0.753 || 0.693 || 0.574 <br />
|-<br />
|5 || Gillet & Richard 2 || 0.630 || 81.60% || 53.00% || 0.643 || 0.774 || 0.517 || 0.599 <br />
|-<br />
|6 || Dittmar, C. || 0.617 || 71.37% || 67.78% || 0.695 || 0.631 || 0.675 || 0.544 <br />
|-<br />
|7 || Paulus, J. || 0.597 || 62.90% || 75.47% || 0.686 || 0.648 || 0.695 || 0.449 <br />
|-<br />
|8 || Gillet & Richard 1 || 0.544 || 76.11% || 48.80% || 0.595 || 0.715 || 0.479 || 0.436 <br />
|-<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Artist_Identification_Results&diff=7589
2005:Audio Artist Identification Results
2010-08-02T16:23:56Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Artist Identification task. <br />
<br />
===Goal===<br />
To identify artist from music audio (in PCM format).<br />
<br />
===Datasets===<br />
Two sets of data were used: Magnatune and USPOP. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files<br />
|-<br />
! Magnatune<br />
| 35.2 GB || 1158 || 642<br />
|-<br />
! USPOP<br />
| 37.3 GB || 1158 || 653<br />
|}<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Raw Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 ||[https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 72.45% <br />
|-<br />
|2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande, & Eck (1)] || 68.57% <br />
|-<br />
| 3 ||[https://www.music-ir.org/mirex/abstracts/2005/bergstra Bergstra, Casagrande, & Eck (2)] || 66.71%<br />
|- <br />
| 4 ||[https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 61.28%<br />
|- <br />
| 5 || West & Lamere || 47.24%<br />
|-<br />
| 6 ||[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 42.05%<br />
|- <br />
| 7 ||[https://www.music-ir.org/mirex/abstracts/2005/logan.pdf Logan, B] || 25.95% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | Magnatune Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Bergstra, Casagrande, & Eck (1) || 77.26% || 79.64% || 24 hours || B0 ||[https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 2 || Mandel & Ellis || 76.60% || 76.62% || 11073 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_MTeval.txt ME_MTeval.txt]<br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 74.45% || 74.51% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_MTeval.txt BCE_2_MTeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 66.36% || 66.48% || 4272 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_MTeval.txt P_MTeval.txt] <br />
|-<br />
| 5 || Tzanetakis, G. || 55.45% || 55.59% || 2632 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_MTeval.txt T_MTeval.txt] <br />
|-<br />
| 6 || West & Lamere || 53.43% || 53.48% || 27480 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_MTeval.txt WL_MTeval.txt]<br />
|-<br />
| 7 || Logan, B || 37.07% || 37.10% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_MTeval.txt L_MTeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Mandel & Ellis || 68.30% || 67.96% || 10240 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_USeval.txt ME_USeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande, & Eck (1) || 59.88% || 60.90% || 24 Hours || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_USeval.txt ME_USeval.txt] <br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 58.96% || 58.96% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_USeval.txt BCE_2_USeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 56.20% || 56.03% || 4321 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_USeval.txt P_USeval.txt] <br />
|-<br />
| 5 || West & Lamere || 41.04% || 41.00% || 26871 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_USeval.txt WL_USeval.txt]<br />
|-<br />
| 6 || Tzanetakis, G. || 28.64% || 28.48% || 2443 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_USeval.txt T_USeval.txt]<br />
|-<br />
| 7 || Logan, B. || 14.83% || 14.76% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_USeval.txt L_USeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br><br />
'''Note:''' <br />
DNC: did not complete ( error in execution).<br />
TO: timed out (did not complete within 24 hours).</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Artist_Identification_Results&diff=7588
2005:Audio Artist Identification Results
2010-08-02T16:23:41Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Artist Identification task. <br />
<br />
===Goal===<br />
To identify artist from music audio (in PCM format).<br />
<br />
===Datasets===<br />
Two sets of data were used: Magnatune and USPOP. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files<br />
|-<br />
! Magnatune<br />
| 35.2 GB || 1158 || 642<br />
|-<br />
! USPOP<br />
| 37.3 GB || 1158 || 653<br />
|}<br />
<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Raw Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 ||[https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 72.45% <br />
|-<br />
|2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande, & Eck (1)] || 68.57% <br />
|-<br />
| 3 ||[https://www.music-ir.org/mirex/abstracts/2005/bergstra Bergstra, Casagrande, & Eck (2)] || 66.71%<br />
|- <br />
| 4 ||[https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 61.28%<br />
|- <br />
| 5 || West & Lamere || 47.24%<br />
|-<br />
| 6 ||[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 42.05%<br />
|- <br />
| 7 ||[https://www.music-ir.org/mirex/abstracts/2005/logan.pdf Logan, B] || 25.95% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | Magnatune Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Bergstra, Casagrande, & Eck (1) || 77.26% || 79.64% || 24 hours || B0 ||[https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 2 || Mandel & Ellis || 76.60% || 76.62% || 11073 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_MTeval.txt ME_MTeval.txt]<br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 74.45% || 74.51% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_MTeval.txt BCE_2_MTeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 66.36% || 66.48% || 4272 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_MTeval.txt P_MTeval.txt] <br />
|-<br />
| 5 || Tzanetakis, G. || 55.45% || 55.59% || 2632 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_MTeval.txt T_MTeval.txt] <br />
|-<br />
| 6 || West & Lamere || 53.43% || 53.48% || 27480 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_MTeval.txt WL_MTeval.txt]<br />
|-<br />
| 7 || Logan, B || 37.07% || 37.10% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_MTeval.txt L_MTeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Mandel & Ellis || 68.30% || 67.96% || 10240 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_USeval.txt ME_USeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande, & Eck (1) || 59.88% || 60.90% || 24 Hours || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_USeval.txt ME_USeval.txt] <br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 58.96% || 58.96% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_USeval.txt BCE_2_USeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 56.20% || 56.03% || 4321 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_USeval.txt P_USeval.txt] <br />
|-<br />
| 5 || West & Lamere || 41.04% || 41.00% || 26871 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_USeval.txt WL_USeval.txt]<br />
|-<br />
| 6 || Tzanetakis, G. || 28.64% || 28.48% || 2443 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_USeval.txt T_USeval.txt]<br />
|-<br />
| 7 || Logan, B. || 14.83% || 14.76% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_USeval.txt L_USeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br><br />
'''Note:''' <br />
DNC: did not complete ( error in execution).<br />
TO: timed out (did not complete within 24 hours).</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7587
2005:Audio Tempo Extraction Results
2010-08-02T16:21:59Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Tempo Extraction task. <br />
<br />
===Goal===<br />
The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
===Dataset=== <br />
140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/alonso.pdf Alonso, David, &amp; Richard]<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (1)]<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (2)]<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (1)]<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/peeters.pdf Peeters, G.]<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (2)]<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (4)]<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/eck.pdf Eck, D.]<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Davies &amp; Brossier]<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (3)]<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sethares.pdf Sethares, W.]<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Brossier, P.]<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.]<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Artist_Identification_Results&diff=7586
2005:Audio Artist Identification Results
2010-08-02T16:21:11Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Artist Identification task. <br />
<br />
===Goal===<br />
To identify artist from music audio (in PCM format).<br />
<br />
===Datasets===<br />
Two sets of data were used: Magnatune and USPOP. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files<br />
|-<br />
! Magnatune<br />
| 35.2 GB || 1158 || 642<br />
|-<br />
! USPOP<br />
| 37.3 GB || 1158 || 653<br />
|}<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Raw Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 ||[https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 72.45% <br />
|-<br />
|2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande, & Eck (1)] || 68.57% <br />
|-<br />
| 3 ||[https://www.music-ir.org/mirex/abstracts/2005/bergstra Bergstra, Casagrande, & Eck (2)] || 66.71%<br />
|- <br />
| 4 ||[https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 61.28%<br />
|- <br />
| 5 || West & Lamere || 47.24%<br />
|-<br />
| 6 ||[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 42.05%<br />
|- <br />
| 7 ||[https://www.music-ir.org/mirex/abstracts/2005/logan.pdf Logan, B] || 25.95% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | Magnatune Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Bergstra, Casagrande, & Eck (1) || 77.26% || 79.64% || 24 hours || B0 ||[https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 2 || Mandel & Ellis || 76.60% || 76.62% || 11073 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_MTeval.txt ME_MTeval.txt]<br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 74.45% || 74.51% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_MTeval.txt BCE_2_MTeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 66.36% || 66.48% || 4272 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_MTeval.txt P_MTeval.txt] <br />
|-<br />
| 5 || Tzanetakis, G. || 55.45% || 55.59% || 2632 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_MTeval.txt T_MTeval.txt] <br />
|-<br />
| 6 || West & Lamere || 53.43% || 53.48% || 27480 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_MTeval.txt WL_MTeval.txt]<br />
|-<br />
| 7 || Logan, B || 37.07% || 37.10% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_MTeval.txt L_MTeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Mandel & Ellis || 68.30% || 67.96% || 10240 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_USeval.txt ME_USeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande, & Eck (1) || 59.88% || 60.90% || 24 Hours || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_USeval.txt ME_USeval.txt] <br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 58.96% || 58.96% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_USeval.txt BCE_2_USeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 56.20% || 56.03% || 4321 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_USeval.txt P_USeval.txt] <br />
|-<br />
| 5 || West & Lamere || 41.04% || 41.00% || 26871 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_USeval.txt WL_USeval.txt]<br />
|-<br />
| 6 || Tzanetakis, G. || 28.64% || 28.48% || 2443 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_USeval.txt T_USeval.txt]<br />
|-<br />
| 7 || Logan, B. || 14.83% || 14.76% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_USeval.txt L_USeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br><br />
'''Note:''' <br />
DNC: did not complete ( error in execution).<br />
TO: timed out (did not complete within 24 hours).</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7585
2005:Audio Tempo Extraction Results
2010-08-02T16:19:48Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
===Goal:=== The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
===Dataset:=== 140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/alonso.pdf Alonso, David, &amp; Richard]<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (1)]<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (2)]<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (1)]<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/peeters.pdf Peeters, G.]<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (2)]<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (4)]<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/eck.pdf Eck, D.]<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Davies &amp; Brossier]<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (3)]<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sethares.pdf Sethares, W.]<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Brossier, P.]<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.]<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7584
2005:Audio Tempo Extraction Results
2010-08-02T16:18:01Z
<p>Singh14: /* McNemar's Test Results */</p>
<hr />
<div>==Introduction==<br />
'''Goal:''' The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
'''Dataset:''' 140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/alonso.pdf Alonso, David, &amp; Richard]<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (1)]<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (2)]<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (1)]<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/peeters.pdf Peeters, G.]<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (2)]<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (4)]<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/eck.pdf Eck, D.]<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Davies &amp; Brossier]<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (3)]<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sethares.pdf Sethares, W.]<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Brossier, P.]<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.]<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2006:QBSH:_Query-by-Singing/Humming_Results&diff=7571
2006:QBSH: Query-by-Singing/Humming Results
2010-07-31T21:06:50Z
<p>Singh14: /* Team ID */</p>
<hr />
<div>[[Category: Results]]<br />
==Introduction==<br />
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[QBSH: Query-by-Singing/Humming]] page.<br />
QBSH task consists of two subtasks. The first subtask is known as '''Known-Item Retrieval'''. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries. <br />
<br />
The second subtask is called '''Queries as Variations'''. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries. <br />
<br />
===General Legend===<br />
====Team ID==== <br />
'''AU''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_uitdenbogerd.pdf Alexandra Uitdenbogerd]<br /><br />
'''CS1''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-ear]<br /> <br />
'''CS2''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-midi]<br /> <br />
'''CS3''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-warp]<br /> <br />
'''FH''' = Pascal Ferraro and Pierre Hanna<br /><br />
'''NM''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_mikkila.pdf Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen]<br /><br />
'''RJ''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_jang.pdf Jyh-Shing Roger Jang and Nien-Jung Lee]<br /><br />
'''RL''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_tararira.pdf Ernesto López and Martín Rocamora]<br /><br />
'''RT''' = [https://www.music-ir.org/mirex/abstracts/2006/SMS_QBSH_typke.pdf Rainer Typke, Frans Wiering and Remco C. Veltkamp]<br /> <br />
'''XW''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_wu.pdf Xiao Wu and Ming Li]<br /><br />
<br />
===Calculating Summary Measures===<br />
MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.<br />
<br />
==Overall Summary Results==<br />
<br />
<csv>2006/qbsh_overall_sum.csv</csv><br />
<br />
===QBSH Task 1 Runtime Data===<br />
<br />
<csv>2006/qbsh06_task1_runtime.csv</csv><br />
<br />
===QBSH Task 2 Runtime Data===<br />
<br />
<csv>2006/qbsh06_task2_runtime.csv</csv><br />
<br />
==Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)==<br />
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);<br />
<csv>2006/qush_friedman.csv</csv><br />
<csv>2006/QBSH_sum.csv</csv><br />
<br />
==Task I: "Known-Item Searching" Summary Results==<br />
MMR data summarized by the 48 ground truth song groups.<br />
<csv>2006/qbsh_task1_sum.csv</csv><br />
<br />
==Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)==<br />
<br />
The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); <br />
<csv>2006/qbsh2_friedman.csv</csv><br />
<csv>2006/qbsh2_sum.csv</csv><br />
<br />
==Task II: "Queries as Variations" Summary Results==<br />
Mean precision data summarized over the 48 ground truth song groups. <br />
<csv>2006/qbsh06_task2_sum.csv</csv><br />
<br />
==Raw Scores==<br />
The raw data are located on the [[2006:Query-by-Singing/Humming Raw Data]] page.</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2006:QBSH:_Query-by-Singing/Humming_Results&diff=7570
2006:QBSH: Query-by-Singing/Humming Results
2010-07-31T21:03:32Z
<p>Singh14: /* Team ID */</p>
<hr />
<div>[[Category: Results]]<br />
==Introduction==<br />
These are the results for the 2006 running of the QBSH: Query-by-Singing Humming task set. For background information about this task set please refer to the [[QBSH: Query-by-Singing/Humming]] page.<br />
QBSH task consists of two subtasks. The first subtask is known as '''Known-Item Retrieval'''. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries. <br />
<br />
The second subtask is called '''Queries as Variations'''. In this subtask, systems take an input from the query database which consists of 2797 sung queries + 48 ground truth files and return a list of 20 songs from the test database which consists of 48 ground truth MIDIs + 2000 Essen MIDI noise files + 2797 sung queries. The precision based on the number of songs within the same ground truth class of the query is calculated over the top 20 returns for each of the 2845 queries. <br />
<br />
===General Legend===<br />
====Team ID==== <br />
'''AU''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_uitdenbogerd.pdf Alexandra Uitdenbogerd]<br /><br />
'''CS1''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-ear]<br /> <br />
'''CS2''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-midi]<br /> <br />
'''CS3''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_sailer.pdf Christian Sailer-warp]<br /> <br />
'''FH''' = Pascal Ferraro and Pierre Hanna<br /><br />
'''NM''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_mikkila.pdf Kjell Lemström, Niko Mikkilä, Veli Mäkinen and Esko Ukkonen]<br /><br />
'''RJ''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_jang.pdf Jyh-Shing Roger Jang and Nien-Jung Lee]<br /><br />
'''RL''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_tararira.pdf Ernesto Lopez and Martín Rocamora]<br /><br />
'''RT''' = [https://www.music-ir.org/mirex/abstracts/2006/SMS_QBSH_typke.pdf Rainer Typke, Frans Wiering and Remco C. Veltkamp]<br /> <br />
'''XW''' = [https://www.music-ir.org/mirex/abstracts/2006/QBSH_wu.pdf Xiao Wu and Ming Li]<br /><br />
<br />
===Calculating Summary Measures===<br />
MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each ground truth group as well as overall.<br />
<br />
==Overall Summary Results==<br />
<br />
<csv>2006/qbsh_overall_sum.csv</csv><br />
<br />
===QBSH Task 1 Runtime Data===<br />
<br />
<csv>2006/qbsh06_task1_runtime.csv</csv><br />
<br />
===QBSH Task 2 Runtime Data===<br />
<br />
<csv>2006/qbsh06_task2_runtime.csv</csv><br />
<br />
==Task I: "Known-Item Searching" Friedman Test with Multiple Comparisons Results (p=0.05)==<br />
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);<br />
<csv>2006/qush_friedman.csv</csv><br />
<csv>2006/QBSH_sum.csv</csv><br />
<br />
==Task I: "Known-Item Searching" Summary Results==<br />
MMR data summarized by the 48 ground truth song groups.<br />
<csv>2006/qbsh_task1_sum.csv</csv><br />
<br />
==Task II: "Queries as Variations" Friedman Test with Multiple Comparisons Results (p=0.05)==<br />
<br />
The Friedman test was run in MATLAB against the QBSH Task 2 mean precision data over the 48 ground truth song groups.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); <br />
<csv>2006/qbsh2_friedman.csv</csv><br />
<csv>2006/qbsh2_sum.csv</csv><br />
<br />
==Task II: "Queries as Variations" Summary Results==<br />
Mean precision data summarized over the 48 ground truth song groups. <br />
<csv>2006/qbsh06_task2_sum.csv</csv><br />
<br />
==Raw Scores==<br />
The raw data are located on the [[2006:Query-by-Singing/Humming Raw Data]] page.</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Artist_Identification_Results&diff=7569
2005:Audio Artist Identification Results
2010-07-31T21:00:01Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
These are the results for the 2008 running of the Audio Artist Identification task. <br />
<br />
===Goal===<br />
To identify artist from music audio (in PCM format).<br />
<br />
===Datasets===<br />
Two sets of data were used: Magnatune and USPOP. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow;"<br />
! Dataset !! Size (@ 44.1 KHz) !! Number of Training Files !! Number of Testing Files<br />
|-<br />
! Magnatune<br />
| 35.2 GB || 1158 || 642<br />
|-<br />
! USPOP<br />
| 37.3 GB || 1158 || 653<br />
|}<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="3" | OVERALL <br />
|-style="background: yellow;"<br />
! Rank !! Participant !! Mean of Magnatune Raw Classification Accuracy <br> and USPOP Raw Classification Accuracy <br />
|-<br />
| 1 ||[https://www.music-ir.org/mirex/abstracts/2005/mandel.pdf Mandel & Ellis] || 72.45% <br />
|-<br />
|2 || [https://www.music-ir.org/mirex/abstracts/2005/bergstra.pdf Bergstra, Casagrande, & Eck (1)] || 68.57% <br />
|-<br />
| 3 ||[https://www.music-ir.org/mirex/abstracts/2005/bergstra Bergstra, Casagrande, & Eck (2)] || 66.71%<br />
|- <br />
| 4 ||[https://www.music-ir.org/mirex/abstracts/2005/pampalk.pdf Pampalk, E.] || 61.28%<br />
|- <br />
| 5 || West & Lamere || 47.24%<br />
|-<br />
| 6 ||[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.] || 42.05%<br />
|- <br />
| 7 ||[https://www.music-ir.org/mirex/abstracts/2005/logan.pdf Logan, B] || 25.95% <br />
|-<br />
|}<br />
<br />
===Magnatune Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | Magnatune Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Bergstra, Casagrande, & Eck (1) || 77.26% || 79.64% || 24 hours || B0 ||[https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_MTeval.txt BCE_1_MTeval.txt] <br />
|-<br />
| 2 || Mandel & Ellis || 76.60% || 76.62% || 11073 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_MTeval.txt ME_MTeval.txt]<br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 74.45% || 74.51% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_MTeval.txt BCE_2_MTeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 66.36% || 66.48% || 4272 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_MTeval.txt P_MTeval.txt] <br />
|-<br />
| 5 || Tzanetakis, G. || 55.45% || 55.59% || 2632 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_MTeval.txt T_MTeval.txt] <br />
|-<br />
| 6 || West & Lamere || 53.43% || 53.48% || 27480 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_MTeval.txt WL_MTeval.txt]<br />
|-<br />
| 7 || Logan, B || 37.07% || 37.10% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_MTeval.txt L_MTeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br />
===USPOP Dataset===<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="7" | USPOP Dataset <br />
|-style="background: yellow;" <br />
! Rank !! Participant !! Raw Classification Accuracy !! Normalized Raw Classification Accuracy !! Runtime (s) !! Machine !! Confusion Matrix Files<br />
|- <br />
| 1 || Mandel & Ellis || 68.30% || 67.96% || 10240 || R || [https://www.music-ir.org/mirex/results/2005/audio-artist/ME_USeval.txt ME_USeval.txt] <br />
|-<br />
| 2 || Bergstra, Casagrande, & Eck (1) || 59.88% || 60.90% || 24 Hours || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_1_USeval.txt ME_USeval.txt] <br />
|-<br />
| 3 || Bergstra, Casagrande, & Eck (2) || 58.96% || 58.96% || -- || -- || [https://www.music-ir.org/mirex/results/2005/audio-artist/BCE_2_USeval.txt BCE_2_USeval.txt]<br />
|-<br />
| 4 || Pampalk, E. || 56.20% || 56.03% || 4321 || B1 || [https://www.music-ir.org/mirex/results/2005/audio-artist/P_USeval.txt P_USeval.txt] <br />
|-<br />
| 5 || West & Lamere || 41.04% || 41.00% || 26871 || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/WL_USeval.txt WL_USeval.txt]<br />
|-<br />
| 6 || Tzanetakis, G. || 28.64% || 28.48% || 2443 || B0 || [https://www.music-ir.org/mirex/results/2005/audio-artist/T_USeval.txt T_USeval.txt]<br />
|-<br />
| 7 || Logan, B. || 14.83% || 14.76% || N/A || B3 || [https://www.music-ir.org/mirex/results/2005/audio-artist/L_USeval.txt L_USeval.txt]<br />
|-<br />
| 8 || Lidy & Rauber (SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD) || TO * || -- || -- || -- || -- <br />
|-<br />
| 8 || Lidy & Rauber (RP+SSD+RH) || TO * || -- || -- || -- || -- <br />
|-<br />
|}<br />
<br><br />
'''Note:''' <br />
DNC: did not complete ( error in execution).<br />
TO: timed out (did not complete within 24 hours).</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7568
2005:Audio Melody Extraction Results
2010-07-31T20:58:51Z
<p>Singh14: /* Introduction */</p>
<hr />
<div>==Introduction==<br />
These are the results for the 2005 running of the Audio Melody Extraction task set.<br />
<br />
''' Goal:''' To extract melodic content from polyphonic audio.<br />
<br />
'''Dataset:''' 25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano<br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
|81.8%<br />
|17.3%<br />
|1.85<br />
|68.1%<br />
|71.4%<br />
!71.4%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
|90.3%<br />
|39.5%<br />
|1.56<br />
!68.6%<br />
!74.1%<br />
|64.3%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
|91.6%<br />
|42.7%<br />
|1.56<br />
|67.3%<br />
|73.4%<br />
|61.1%<br />
|5471<br />
|B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
|68.8%<br />
|23.2%<br />
|1.22<br />
|58.5%<br />
|62.0%<br />
|61.1%<br />
|45618<br />
|Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
|72.7%<br />
|32.4%<br />
|1.06<br />
|60.1%<br />
|67.1%<br />
|59.5%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
|83.4%<br />
|55.8%<br />
|0.83<br />
|62.7%<br />
|66.7%<br />
|57.8%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
|99.9% *<br />
|99.4% *<br />
|0.59 *<br />
|65.8%<br />
|71.8%<br />
|49.9% *<br />
|211<br />
|F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
|96.1% *<br />
|93.7% *<br />
|0.23 *<br />
|59.8%<br />
|67.6%<br />
|47.9% *<br />
|?<br />
|G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
|99.6% *<br />
|96.4% *<br />
|0.86 *<br />
|59.6%<br />
|71.1%<br />
|46.4% *<br />
|251<br />
|G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
|99.2% * †<br />
|98.8% * †<br />
|0.14 * †<br />
|3.9% †<br />
|8.1% †<br />
|3.2% * †<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
|70.78%<br />
|67.48%<br />
|70.83%<br />
|80.20%<br />
|78.34%<br />
|81.48%<br />
|67.48%<br />
|70.83%<br />
|73.61%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
|63.94%<br />
|65.99%<br />
|69.85%<br />
|81.87%<br />
|70.22%<br />
|73.85%<br />
|68.20%<br />
|73.70%<br />
|67.33%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
|60.70%<br />
|58.16%<br />
|61.56%<br />
|70.92%<br />
|70.23%<br />
|74.06%<br />
|58.16%<br />
|61.56%<br />
|63.79%<br />
|45618<br />
|Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
|60.61%<br />
|62.47%<br />
|67.95%<br />
|83.02%<br />
|66.18%<br />
|71.10%<br />
|66.75%<br />
|72.86%<br />
|65.14%<br />
|5471<br />
|B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
|59.18%<br />
|58.06%<br />
|63.13%<br />
|73.35%<br />
|67.53%<br />
|72.24%<br />
|59.76%<br />
|66.73%<br />
|63.25%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
|57.32%<br />
|62.23%<br />
|66.20%<br />
|74.76%<br />
|65.42%<br />
|69.32%<br />
|62.23%<br />
|66.20%<br />
|60.76%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|Goto, M.<br />
|49.68%<br />
|65.58%<br />
|71.47%<br />
|77.18%<br />
|56.46%<br />
|61.99%<br />
|65.58%<br />
|71.47%<br />
|54.89%<br />
|211<br />
|F<br />
|----<br />
|8<br />
|Vincent, E.<br />
|45.98%<br />
|59.17%<br />
|70.64%<br />
|77.61%<br />
|51.98%<br />
|62.36%<br />
|59.17%<br />
|70.64%<br />
|55.52%<br />
|251<br />
|G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
|3.2%<br />
|3.93%<br />
|8.06%<br />
|76.61%<br />
|3.63%<br />
|7.27%<br />
|3.93%<br />
|8.06%<br />
|6.43%<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Key_Finding_Results&diff=7567
2005:Symbolic Key Finding Results
2010-07-31T20:52:40Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
'''Goal:''' The evaluation of key finding algorithms applied to MIDI files. Note: There is a close relationship (same musical datasets) between this contest and the Audio Key Finding contest. Here is a link to the Audio Key Finding results.<br />
<br />
'''Dataset:''' 1,252 MIDI files, 3.62 Megabytes <br />
<br />
==Result==<br />
<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Total Score<br />
!Percentage Score<br />
!Correct Keys<br />
!Perfect 5th Errors<br />
!Relative Major/Minor Errors<br />
!Parallel Major/Minor Errors<br />
!Other Errors<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/temperley.pdf Temperley, D.]<br />
|1143.8<br />
|91.4%<br />
|1127<br />
|15<br />
|27<br />
|6<br />
|77<br />
|91<br />
|B 0<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/zhu2.pdf Zhu, Y.]<br />
|1075.2<br />
|85.9%<br />
|1041<br />
|35<br />
|47<br />
|13<br />
|116<br />
|? (See note)<br />
|R<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/rizo.pdf Rizo &amp; Iñesta]<br />
|982.4<br />
|78.5%<br />
|913<br />
|81<br />
|87<br />
|14<br />
|157<br />
|631<br />
|B 0<br />
|----<br />
|4<br />
|Ehmann, A.<br />
|947.4<br />
|75.7%<br />
|851<br />
|160<br />
|44<br />
|16<br />
|181<br />
|5670<br />
|G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mardirossian.pdf Mardirossian &amp; Chew]<br />
|934.0<br />
|74.6%<br />
|799<br />
|210<br />
|80<br />
|30<br />
|133<br />
|471<br />
|B 0<br />
|----<br />
|}<br />
<br />
'''Note:''' Runtime undetermined due to system hang.</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Melodic_Similarity_Results&diff=7566
2005:Symbolic Melodic Similarity Results
2010-07-31T20:48:27Z
<p>Singh14: /* Result */</p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To retrieve the most similar incipits from the RISM A/II collection, given one of the incipits as a query.<br />
<br />
'''Dataset:''' 558 MusicXML files of the incipits from the RISM collection. 11 incipits of this collection are chosen as queries. These MusicXML files were converted to MIDI files. <br />
<br />
==Result==<br />
The scores reported here are averaged over the 11 queries. <br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average Dynamic Recall <br />
!Normalized Recall at Group Boundaries<br />
!Average precision (non-interpolated)<br />
!Precision at N documents (N is number of relevant document) <br />
!Input Data Format<br />
!Runtime (s)<br />
!Machine<br />
!Detailed Evaluation Result<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/grachten.pdf Grachten, Arcos &amp; Mántaras]<br />
|65.98%<br />
|55.24%<br />
|51.72%<br />
|44.33%<br />
|MIDI<br />
|80.174 *<br />
|B0<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/GAM_eval.txt GAM_eval.txt]<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/orio.pdf Orio, N.]<br />
|64.96%<br />
|53.35%<br />
|42.96%<br />
|39.86%<br />
|XML<br />
|24.610<br />
|B4<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/O_eval.txt O_eval.txt]<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/suyoto.pdf Suyoto &amp; Uitdenbogerd]<br />
|64.18%<br />
|51.79%<br />
|40.42%<br />
|41.72%<br />
|MIDI<br />
|48.133<br />
|B3<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/SU_eval.txt SU_eval.txt]<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/typke.pdf Typke, Wiering &amp; Veltkamp]<br />
|57.09%<br />
|48.17%<br />
|35.64%<br />
|33.46%<br />
|MIDI<br />
|51240<br />
|B4<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/TWV_eval.txt TWV_eval.txt]<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mikkila.pdf Lemström, Mikkilä, Mäkinen &amp; Ukkonen (P3)]<br />
|55.82%<br />
|46.56%<br />
|41.40%<br />
|39.18%<br />
|MIDI<br />
|10.007 *<br />
|B0<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/LMMU_P3_eval.txt LMMU_P3_eval.txt]<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mikkila.pdf Lemström, Mikkilä, Mäkinen &amp; Ukkonen (DP)]<br />
|54.27%<br />
|47.26%<br />
|39.91%<br />
|36.20%<br />
|MIDI<br />
|10.106 *<br />
|B0<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/LMMU_DP_eval.txt LMMU_DP_eval.txt]<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/frieler.pdf Frieler &amp; Müllensiefen]<br />
|51.81%<br />
|45.10%<br />
|33.93%<br />
|33.71%<br />
|MIDI<br />
|54.593<br />
|B4<br />
|[https://www.music-ir.org/mirex/results/2005/sym-melody/FM_eval.txt FM_eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Melodic_Similarity_Results&diff=7565
2005:Symbolic Melodic Similarity Results
2010-07-31T20:45:50Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To retrieve the most similar incipits from the RISM A/II collection, given one of the incipits as a query.<br />
<br />
'''Dataset:''' 558 MusicXML files of the incipits from the RISM collection. 11 incipits of this collection are chosen as queries. These MusicXML files were converted to MIDI files. <br />
<br />
==Result==<br />
The scores reported here are averaged over the 11 queries. <br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average Dynamic Recall <br />
!Normalized Recall at Group Boundaries<br />
!Average precision (non-interpolated)<br />
!Precision at N documents (N is number of relevant document) <br />
!Input Data Format<br />
!Runtime (s)<br />
!Machine<br />
!Detailed Evaluation Result<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/grachten.pdf Grachten, Arcos &amp; Mántaras]<br />
|65.98%<br />
|55.24%<br />
|51.72%<br />
|44.33%<br />
|MIDI<br />
|80.174 *<br />
|B0<br />
|GAM_eval.txt<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/orio.pdf Orio, N.]<br />
|64.96%<br />
|53.35%<br />
|42.96%<br />
|39.86%<br />
|XML<br />
|24.610<br />
|B4<br />
|O_eval.txt<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/suyoto.pdf Suyoto &amp; Uitdenbogerd]<br />
|64.18%<br />
|51.79%<br />
|40.42%<br />
|41.72%<br />
|MIDI<br />
|48.133<br />
|B3<br />
|SU_eval.txt<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/typke.pdf Typke, Wiering &amp; Veltkamp]<br />
|57.09%<br />
|48.17%<br />
|35.64%<br />
|33.46%<br />
|MIDI<br />
|51240<br />
|B4<br />
|TWV_eval.txt<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mikkila.pdf Lemström, Mikkilä, Mäkinen &amp; Ukkonen (P3)]<br />
|55.82%<br />
|46.56%<br />
|41.40%<br />
|39.18%<br />
|MIDI<br />
|10.007 *<br />
|B0<br />
|LMMU_P3_eval.txt<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mikkila.pdf Lemström, Mikkilä, Mäkinen &amp; Ukkonen (DP)]<br />
|54.27%<br />
|47.26%<br />
|39.91%<br />
|36.20%<br />
|MIDI<br />
|10.106 *<br />
|B0<br />
|LMMU_DP_eval.txt<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/frieler.pdf Frieler &amp; Müllensiefen]<br />
|51.81%<br />
|45.10%<br />
|33.93%<br />
|33.71%<br />
|MIDI<br />
|54.593<br />
|B4<br />
|FM_eval.txt<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7564
2005:Symbolic Genre Classification Results
2010-07-31T16:36:08Z
<p>Singh14: /* 9 Classes */</p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To classify MIDI recordings into genre categories.<br />
<br />
'''Dataset:''' Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold crossvalidated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|77.17%<br />
|65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|72.08%<br />
|58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|67.57%<br />
|55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|67.14%<br />
|53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|37.76%<br />
|26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7563
2005:Symbolic Genre Classification Results
2010-07-31T16:35:17Z
<p>Singh14: /* 38 Classes */</p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To classify MIDI recordings into genre categories.<br />
<br />
'''Dataset:''' Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold crossvalidated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|77.17%<br />
|65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|72.08%<br />
|58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|67.57%<br />
|55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|67.14%<br />
|53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|37.76%<br />
|26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7562
2005:Symbolic Genre Classification Results
2010-07-31T16:34:32Z
<p>Singh14: /* 38 Classes */</p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To classify MIDI recordings into genre categories.<br />
<br />
'''Dataset:''' Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold crossvalidated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|77.17%<br />
|65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|72.08%<br />
|58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|67.57%<br />
|55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|67.14%<br />
|53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|37.76%<br />
|26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|[https://www.music-ir.org/mirex/results/2005/sym-genre/MF_38eval.txt MF_38eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7561
2005:Symbolic Genre Classification Results
2010-07-31T16:33:26Z
<p>Singh14: /* Results */</p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To classify MIDI recordings into genre categories.<br />
<br />
'''Dataset:''' Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold crossvalidated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/mckay.pdf McKay &amp; Fujinaga]<br />
|77.17%<br />
|65.28%<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (NB)]<br />
|72.08%<br />
|58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|67.57%<br />
|55.90%<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/basili.pdf Basili, Serafini, &amp; Stellato (J48)]<br />
|67.14%<br />
|53.14%<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ponce.pdf Ponce de Leon &amp; Inesta]<br />
|37.76%<br />
|26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/MF_38eval.txt<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_NB_38eval.txt BST_NB_38eval.txt]<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_J48_38eval.txt BST_J48_38eval.txt]<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/L_38eval.txt L_38eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/PI_38eval.txt PI_38eval.txt]<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/MF_9eval.txt MF_9eval.txt]<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_NB_9eval.txt BST_NB_9eval.txt]<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/L_9eval.txt L_9eval.txt]<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/BST_J48_9eval.txt BST_J48_9eval.txt]<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sym-genre/PI_9eval.txt PI_9eval.txt]<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Symbolic_Genre_Classification_Results&diff=7560
2005:Symbolic Genre Classification Results
2010-07-31T16:27:44Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To classify MIDI recordings into genre categories.<br />
<br />
'''Dataset:''' Two sets of genre categories were used, one consisting of 38 categories and one consisting of 9 categories. Each category was represented by 25 MIDI files.Thus, the 38 genre test contained 950 MIDI files and the 9 genre test contained 225 MIDI files.Test runs were 3-fold crossvalidated with each algorithm tested using identical training and testing data splits.<br />
<br />
==Results==<br />
<br />
===Overall===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="4" | OVERALL <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Mean Hierarchical Classification Accuracy <br />
!Mean Raw Classification Accuracy<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|77.17%<br />
|65.28%<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|72.08%<br />
|58.53%<br />
|----<br />
|3<br />
|Li, M.<br />
|67.57%<br />
|55.90%<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|67.14%<br />
|53.14%<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|37.76%<br />
|26.52%<br />
|----<br />
|}<br />
<br />
===38 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 38 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|64.33% <br />
|1.04<br />
|46.11%<br />
|1.51<br />
|3 days<br />
|R<br />
|MF_38eval.txt<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|62.60%<br />
|0.26<br />
|45.05%<br />
|0.55<br />
|N/A<br />
|N/A<br />
|BST_NB_38eval.txt<br />
|----<br />
|3<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|57.61%<br />
|1.14<br />
|40.95%<br />
|1.35<br />
|N/A<br />
|N/A<br />
|BST_J48_38eval.txt<br />
|----<br />
|4<br />
|Li, M.<br />
|54.91%<br />
|0.66<br />
|39.79%<br />
|0.87<br />
|15,948<br />
|G<br />
|L_38eval.txt<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta<br />
|24.84%<br />
|1.40<br />
|15.26%<br />
|1.13<br />
|821<br />
|L<br />
|PI_38eval.txt<br />
|----<br />
|}<br />
<br />
===9 Classes===<br />
{| border="1"<br />
|- style="background: yellow; text-align: center;"<br />
! colspan="9" | 9 Classes <br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Hierarchical Classification Accuracy<br />
!Hierarchical Classification Accuracy Std<br />
!Raw Classification Accuracy<br />
!Raw Classification Accuracy Std<br />
!Runtime (s)<br />
!Machine<br />
!Confusion Matrix Files<br />
|----<br />
|1<br />
|McKay &amp; Fujinaga<br />
|90.00% <br />
|0.60<br />
|84.44%<br />
|1.41<br />
|18,375<br />
|R<br />
|MF_9eval.txt<br />
|----<br />
|2<br />
|Basili, Serafini, &amp; Stellato (NB)<br />
|81.56%<br />
|0.76<br />
|72.00%<br />
|0.88<br />
|N/A<br />
|N/A<br />
|BST_NB_9eval.txt<br />
|----<br />
|3<br />
|Li, M. <br />
|80.22%<br />
|1.47<br />
|72.00%<br />
|2.31<br />
|3,777<br />
|G<br />
|L_9eval.txt<br />
|----<br />
|4<br />
|Basili, Serafini, &amp; Stellato (J48)<br />
|76.67%<br />
|1.11<br />
|65.33%<br />
|1.65<br />
|N/A<br />
|N/A<br />
|BST_J48_9eval.txt<br />
|----<br />
|5<br />
|Ponce de Leon &amp; Inesta <br />
|50.67%<br />
|1.26<br />
|37.78%<br />
|2.30<br />
|197<br />
|L<br />
|PI_9eval.txt<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7559
2005:Audio Tempo Extraction Results
2010-07-31T16:25:15Z
<p>Singh14: /* Results */</p>
<hr />
<div>==Introduction==<br />
'''Goal:''' The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
'''Dataset:''' 140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/alonso.pdf Alonso, David, &amp; Richard]<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (1)]<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (2)]<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (1)]<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/peeters.pdf Peeters, G.]<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (2)]<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (4)]<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/eck.pdf Eck, D.]<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Davies &amp; Brossier]<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (3)]<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|[https://www.music-ir.org/mirex/abstracts/2005/sethares.pdf Sethares, W.]<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|[https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Brossier, P.]<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|[https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.]<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7558
2005:Audio Tempo Extraction Results
2010-07-31T16:23:48Z
<p>Singh14: /* Results */</p>
<hr />
<div>==Introduction==<br />
'''Goal:''' The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
'''Dataset:''' 140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/alonso.pdf Alonso, David, &amp; Richard]<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (1)]<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|https://www.music-ir.org/mirex/abstracts/2005/uhle.pdf Uhle, C. (2)]<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (1)]<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|https://www.music-ir.org/mirex/abstracts/2005/peeters.pdf Peeters, G.]<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (2)]<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (4)]<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|https://www.music-ir.org/mirex/abstracts/2005/eck.pdf Eck, D.]<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Davies &amp; Brossier]<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|https://www.music-ir.org/mirex/abstracts/2005/gouyon.pdf Gouyon &amp; Dixon (3)]<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|https://www.music-ir.org/mirex/abstracts/2005/sethares.pdf Sethares, W.]<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|https://www.music-ir.org/mirex/abstracts/2005/davies.pdf Brossier, P.]<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|https://www.music-ir.org/mirex/abstracts/2005/tzanetakis.pdf Tzanetakis, G.]<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7557
2005:Audio Tempo Extraction Results
2010-07-31T16:17:48Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
'''Goal:''' The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
'''Dataset:''' 140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Alonso, David, &amp; Richard<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|Uhle, C. (1)<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|Uhle, C. (2)<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|Gouyon &amp; Dixon (1)<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|Peeters, G.<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|Gouyon &amp; Dixon (2)<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|Gouyon &amp; Dixon (4)<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|Eck, D.<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|Davies &amp; Brossier<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|Gouyon &amp; Dixon (3)<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|Sethares, W.<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|Brossier, P.<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|Tzanetakis, G.<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Tempo_Extraction_Results&diff=7556
2005:Audio Tempo Extraction Results
2010-07-31T16:17:16Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
<br />
'''Goal:''' The comparison and evaluation of current methods for the extraction of tempo from musical audio<br />
<br />
'''Dataset:''' 140 wav files, 354 Megabytes<br />
<br />
==Results==<br />
<br><br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Rank<br />
!Participant<br />
!Score (std. deviation)<br />
!At Least One Tempo Correct<br />
!Both Tempos Correct<br />
!At Least One Phase Correct<br />
!Both Phases Correct<br />
!Mean Absolute Difference of Scored Saliences<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Alonso, David, &amp; Richard<br />
|0.689 (0.231)<br />
|95.00%<br />
|55.71%<br />
|25.00%<br />
|5.00%<br />
|0.239<br />
|2875<br />
|G<br />
|----<br />
|2<br />
|Uhle, C. (1)<br />
|0.675 (0.273)<br />
|90.71%<br />
|59.29%<br />
|32.14%<br />
|7.14%<br />
|0.222<br />
|1160<br />
|F<br />
|----<br />
|3<br />
|Uhle, C. (2)<br />
|0.675 (0.272)<br />
|90.71%<br />
|59.29%<br />
|32.86%<br />
|6.43%<br />
|0.222<br />
|2621<br />
|F<br />
|----<br />
|4<br />
|Gouyon &amp; Dixon (1)<br />
|0.670 (0.252)<br />
|92.14%<br />
|56.43%<br />
|40.71%<br />
|7.86%<br />
|0.311<br />
|3303<br />
|G<br />
|----<br />
|5<br />
|Peeters, G.<br />
|0.656 (0.223)<br />
|95.71%<br />
|47.86%<br />
|27.86%<br />
|4.29%<br />
|0.258<br />
|2159<br />
|R<br />
|----<br />
|6<br />
|Gouyon &amp; Dixon (2)<br />
|0.649 (0.253)<br />
|92.14%<br />
|51.43%<br />
|37.14%<br />
|5.71%<br />
|0.305<br />
|2050<br />
|G<br />
|----<br />
|7<br />
|Gouyon &amp; Dixon (4)<br />
|0.645 (0.294)<br />
|87.14%<br />
|55.71%<br />
|48.57%<br />
|10.71%<br />
|0.313<br />
|1357<br />
|G<br />
|----<br />
|8<br />
|Eck, D.<br />
|0.644 (0.300)<br />
|86.43%<br />
|53.57%<br />
|37.14%<br />
|5.71%<br />
|0.230<br />
|1665<br />
|Y<br />
|----<br />
|9<br />
|Davies &amp; Brossier<br />
|0.628 (0.284)<br />
|86.43%<br />
|48.57%<br />
|26.43%<br />
|4.29%<br />
|0.224<br />
|1005<br />
|R<br />
|----<br />
|10<br />
|Gouyon &amp; Dixon (3)<br />
|0.607 (0.287)<br />
|87.14%<br />
|47.14%<br />
|36.43%<br />
|6.43%<br />
|0.294<br />
|1388<br />
|R<br />
|----<br />
|11<br />
|Sethares, W.<br />
|0.597 (0.252)<br />
|90.71%<br />
|37.86%<br />
|30.71%<br />
|0.71%<br />
|0.239<br />
|70975<br />
|Y<br />
|----<br />
|12<br />
|Brossier, P.<br />
|0.583 (0.333)<br />
|80.71%<br />
|51.43%<br />
|28.57%<br />
|2.14%<br />
|0.223<br />
|180<br />
|B 0<br />
|----<br />
|13<br />
|Tzanetakis, G.<br />
|0.538 (0.359)<br />
|71.43%<br />
|50.71%<br />
|28.57%<br />
|3.57%<br />
|0.295<br />
|7173<br />
|B 0<br />
|----<br />
|}<br />
<br />
===McNemar's Test Results===<br />
<br />
Statistical probability that algorithms have same error function: Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|- style="background: yellow; text-align: center;"<br />
!Tzanetakis, G.<br />
!Gouyon &amp; Dixon (3)<br />
!Gouyon &amp; Dixon (1)<br />
!Gouyon &amp; Dixon (0)<br />
!Brossier, P.<br />
!Uhle, C. (1)<br />
!Uhle, C. (0)<br />
!Sethares, B<br />
!Peeters, G<br />
!Eck, D.<br />
!Davies, M.<br />
!Alonso, David, &amp; Richard<br />
|----<br />
|Tzanetakis, G.<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (3)<br />
|17.44%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (1)<br />
|50.00%<br />
|19.58%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Gouyon &amp; Dixon (0)<br />
|12.15%<br />
|50.00%<br />
|4.61%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Brossier, P.<br />
|50.00%<br />
|23.54%<br />
|55.31%<br />
|20.05%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (1)<br />
|2.88%<br />
|26.64%<br />
|6.76%<br />
|31.36%<br />
|5.41%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Uhle, C. (0)<br />
|2.88%<br />
|26.64%<br />
|6.31%<br />
|30.89%<br />
|5.86%<br />
|75.00%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Sethares, B.<br />
|0.99%<br />
|0.02%<br />
|0.27%<br />
|0.02%<br />
|1.24%<br />
|0.01%<br />
|0.01%<br />
|n/a<br />
|<br />
|<br />
|<br />
|<br />
|----<br />
|Peeters, G.<br />
|34.70%<br />
|7.19%<br />
|29.83%<br />
|6.32%<br />
|30.73%<br />
|2.40%<br />
|2.40%<br />
|2.97%<br />
|n/a<br />
|<br />
|<br />
|<br />
|----<br />
|Eck, D.<br />
|32.58%<br />
|38.04%<br />
|38.54%<br />
|32.20%<br />
|38.54%<br />
|12.15%<br />
|12.15%<br />
|0.41%<br />
|18.32%<br />
|n/a<br />
|<br />
|<br />
|----<br />
|Davies, M.<br />
|38.77%<br />
|7.17%<br />
|33.59%<br />
|6.31%<br />
|33.89%<br />
|2.67%<br />
|2.67%<br />
|1.38%<br />
|50.00%<br />
|21.35%<br />
|n/a<br />
|<br />
|----<br />
|Alsonso, David, &amp; Richard<br />
|18.02%<br />
|56.12%<br />
|23.99%<br />
|50.00%<br />
|23.99%<br />
|26.64%<br />
|26.64%<br />
|0.04%<br />
|7.62%<br />
|39.39%<br />
|6.07%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7555
2005:Audio Melody Extraction Results
2010-07-31T16:11:04Z
<p>Singh14: </p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To extract melodic content from polyphonic audio.<br />
<br />
'''Dataset:''' 25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano <br />
<br />
==Results==<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
|81.8%<br />
|17.3%<br />
|1.85<br />
|68.1%<br />
|71.4%<br />
!71.4%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
|90.3%<br />
|39.5%<br />
|1.56<br />
!68.6%<br />
!74.1%<br />
|64.3%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
|91.6%<br />
|42.7%<br />
|1.56<br />
|67.3%<br />
|73.4%<br />
|61.1%<br />
|5471<br />
|B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
|68.8%<br />
|23.2%<br />
|1.22<br />
|58.5%<br />
|62.0%<br />
|61.1%<br />
|45618<br />
|Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
|72.7%<br />
|32.4%<br />
|1.06<br />
|60.1%<br />
|67.1%<br />
|59.5%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
|83.4%<br />
|55.8%<br />
|0.83<br />
|62.7%<br />
|66.7%<br />
|57.8%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
|99.9% *<br />
|99.4% *<br />
|0.59 *<br />
|65.8%<br />
|71.8%<br />
|49.9% *<br />
|211<br />
|F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
|96.1% *<br />
|93.7% *<br />
|0.23 *<br />
|59.8%<br />
|67.6%<br />
|47.9% *<br />
|?<br />
|G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
|99.6% *<br />
|96.4% *<br />
|0.86 *<br />
|59.6%<br />
|71.1%<br />
|46.4% *<br />
|251<br />
|G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
|99.2% * †<br />
|98.8% * †<br />
|0.14 * †<br />
|3.9% †<br />
|8.1% †<br />
|3.2% * †<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
===Explanation of Statistics===<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
===Statistical Significance===<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
===Raw data===<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
===Original statistics results===<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
|70.78%<br />
|67.48%<br />
|70.83%<br />
|80.20%<br />
|78.34%<br />
|81.48%<br />
|67.48%<br />
|70.83%<br />
|73.61%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
|63.94%<br />
|65.99%<br />
|69.85%<br />
|81.87%<br />
|70.22%<br />
|73.85%<br />
|68.20%<br />
|73.70%<br />
|67.33%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
|60.70%<br />
|58.16%<br />
|61.56%<br />
|70.92%<br />
|70.23%<br />
|74.06%<br />
|58.16%<br />
|61.56%<br />
|63.79%<br />
|45618<br />
|Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
|60.61%<br />
|62.47%<br />
|67.95%<br />
|83.02%<br />
|66.18%<br />
|71.10%<br />
|66.75%<br />
|72.86%<br />
|65.14%<br />
|5471<br />
|B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
|59.18%<br />
|58.06%<br />
|63.13%<br />
|73.35%<br />
|67.53%<br />
|72.24%<br />
|59.76%<br />
|66.73%<br />
|63.25%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
|57.32%<br />
|62.23%<br />
|66.20%<br />
|74.76%<br />
|65.42%<br />
|69.32%<br />
|62.23%<br />
|66.20%<br />
|60.76%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|Goto, M.<br />
|49.68%<br />
|65.58%<br />
|71.47%<br />
|77.18%<br />
|56.46%<br />
|61.99%<br />
|65.58%<br />
|71.47%<br />
|54.89%<br />
|211<br />
|F<br />
|----<br />
|8<br />
|Vincent, E.<br />
|45.98%<br />
|59.17%<br />
|70.64%<br />
|77.61%<br />
|51.98%<br />
|62.36%<br />
|59.17%<br />
|70.64%<br />
|55.52%<br />
|251<br />
|G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
|3.2%<br />
|3.93%<br />
|8.06%<br />
|76.61%<br />
|3.63%<br />
|7.27%<br />
|3.93%<br />
|8.06%<br />
|6.43%<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
<br />
<br />
===McNemars Test Results===<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14
https://www.music-ir.org/mirex/w/index.php?title=2005:Audio_Melody_Extraction_Results&diff=7554
2005:Audio Melody Extraction Results
2010-07-31T16:07:38Z
<p>Singh14: /* McNemars Test Results */</p>
<hr />
<div>==Introduction==<br />
''' Goal:''' To extract melodic content from polyphonic audio.<br />
<br />
'''Dataset:''' 25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
! Rank<br />
!Participant<br />
!A. Voicing Detection<br />
!B. Voicing False Alarm<br />
!C. Voicing d-prime<br />
!D. Raw Pitch Accuracy<br />
!E. Raw Chroma Accuracy<br />
!F. Overall Accuracy<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|[https://www.music-ir.org/mirex/abstracts/2005/dressler.pdf Dressler, K.]<br />
|81.8%<br />
|17.3%<br />
|1.85<br />
|68.1%<br />
|71.4%<br />
!71.4%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|[https://www.music-ir.org/mirex/abstracts/2005/ryynanen.pdf Ryynänen &amp; Klapuri]<br />
|90.3%<br />
|39.5%<br />
|1.56<br />
!68.6%<br />
!74.1%<br />
|64.3%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/poliner.pdf Poliner &amp; Ellis]<br />
|91.6%<br />
|42.7%<br />
|1.56<br />
|67.3%<br />
|73.4%<br />
|61.1%<br />
|5471<br />
|B 0<br />
|----<br />
|3<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 2]<br />
|68.8%<br />
|23.2%<br />
|1.22<br />
|58.5%<br />
|62.0%<br />
|61.1%<br />
|45618<br />
|Y<br />
|----<br />
|5<br />
|[https://www.music-ir.org/mirex/abstracts/2005/marolt.pdf Marolt, M.]<br />
|72.7%<br />
|32.4%<br />
|1.06<br />
|60.1%<br />
|67.1%<br />
|59.5%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|[https://www.music-ir.org/mirex/abstracts/2005/paiva.pdf Paiva, R. 1]<br />
|83.4%<br />
|55.8%<br />
|0.83<br />
|62.7%<br />
|66.7%<br />
|57.8%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|[https://www.music-ir.org/mirex/abstracts/2005/goto.pdf Goto, M.]<br />
|99.9% *<br />
|99.4% *<br />
|0.59 *<br />
|65.8%<br />
|71.8%<br />
|49.9% *<br />
|211<br />
|F<br />
|----<br />
|8<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 1]<br />
|96.1% *<br />
|93.7% *<br />
|0.23 *<br />
|59.8%<br />
|67.6%<br />
|47.9% *<br />
|?<br />
|G<br />
|----<br />
|9<br />
|[https://www.music-ir.org/mirex/abstracts/2005/vincent.pdf Vincent &amp; Plumbley 2]<br />
|99.6% *<br />
|96.4% *<br />
|0.86 *<br />
|59.6%<br />
|71.1%<br />
|46.4% *<br />
|251<br />
|G<br />
|----<br />
|10<br />
|[https://www.music-ir.org/mirex/abstracts/2005/brossier.pdf Brossier, P.]<br />
|99.2% * †<br />
|98.8% * †<br />
|0.14 * †<br />
|3.9% †<br />
|8.1% †<br />
|3.2% * †<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br><br />
<small><br />
'''Notes:'''<br />
Bold numbers are the best in each column<br />
* Goto, Vincent, and Brossier did not perform voiced/unvoiced detection, so the starred results cannot be meaningfully compared to other systems. The Voicing rates are not 100% because for a few files these systems reported small numbers of no-voicing frames because the pitch tracking terminated early, and the remainder of the time was padded with zeros.<br />
† Scores for Brossier are artificially low due to an unresolved algorithmic issue.<br />
</small><br />
<br />
[[File:2006_melody.jpg]]<br />
<br />
==Explanation of Statistics==<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are: <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!unvx <br />
!vx <br />
!sum<br />
|-<br />
|Ground unvoiced || TN || FP || GU<br />
|-<br />
|Truth voiced || FN || TP || GV<br />
|-<br />
|Sum || DU || DV || TO<br />
|-<br />
|}<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. Statistic A, Voicing Detection is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
<br />
Statistic B, Voicing False Alarm, is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
<br />
Statistic C, Voicing d-prime, is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pooled the frames from all excerpts to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts gave some misleading results.<br />
<br />
Now we move on to the actual pitch detection. Statistic D, Raw Pitch Accuracy is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
<br />
Similarly, Statistic E, Raw Chroma Accuracy, is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
<br />
Finally, Statistic F, Overall Accuracy, combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculated the performance for each of the 25 excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts. <br />
<br />
==Statistical Significance==<br />
<br />
In a comparative evaluation of this kind, a criticial question is whether an observed difference in the performance of two systems is statistically significant, or whether two systems with equal underlying error rates would show this level of difference due only to random variations in successive measurements. One typical approach is to adopt a binomial model, where each system is assumed to have a fixed probability of making an error on each trial, which is being estimated by the proportion of errors made on the test set. Under the binomial model, the variance of any count is proportional to the size of the count, thus the standard error declines as the square-root of the number of (independent) trials.<br />
<br />
A more precise approach is McNemar's test, which considers only trials in which two systems make different predictions (since no information on their difference is available from trials in which they report the same outcome). Some results from McNemar's test are presented below.<br />
<br />
The problem with these significance tests is that they rely on the trials being independent. Our counts are based on 10 ms frames, which are likely to be highly dependent on their neighbors. One way to ensure trials which are more independent would be to space them more widely in time, for instance by recording the result from a 10 ms frame only every 250 ms. (This number comes from the idea that an average musical note might last around 250 ms, so this level of subsampling would tend to pick samples from different notes, which are closer to independent trials.) If we actually measured results only on these subselected samples, we would expect to see the same overall error rates, but the total count of trials would be many fewer.<br />
<br />
Very roughly speaking, the 25 test signals of around 30 s each might contain around 2000 independent trials. Under this number of trials, at an accuracy of 60-70%, systems whose performance differs by less than about 2.5% would not be statistically different at the 5% level (using a simplistic, one-tailed binomial test). Thus, in the above table, the top two systems are significantly different from the others, but the next three are in a statistical dead heat. <br />
<br />
==Raw data==<br />
<br />
For flexibility in calculating other results, we are including the raw values that we used to caclulate the statistics above. Because the data are broken down by track, it's a lot of data. Thus we are making it available as an Microsoft Excel Workbook, with one sheet for each entry.<br />
<br />
==Original statistics results==<br />
<br />
Below are the original statistics reported for the task, for historical purposes. The important numbers are the same as the table above. <br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!Rank<br />
!Participant<br />
!Average correctly transcribed voiced and unvoiced portions<br />
!Averaged correctly transcribed voiced instants<br />
!Averaged correctly transcribed voiced instants mapped to one octave<br />
!Average estimation of temporal boundaries of the melodic segments<br />
!Average F-measure<br />
!Average F-measure mapped to one octave<br />
!Average correctly transcribed instants (ignoring segmentation errors)<br />
!Average correctly transcribed instants mapped to one octave (ignoring segmentation errors)<br />
!Average correctly transcribed voiced and unvoiced portions mapped to one Octave<br />
!Runtime (s)<br />
!Machine<br />
|----<br />
|1<br />
|Dressler, K.<br />
|70.78%<br />
|67.48%<br />
|70.83%<br />
|80.20%<br />
|78.34%<br />
|81.48%<br />
|67.48%<br />
|70.83%<br />
|73.61%<br />
|32<br />
|R<br />
|----<br />
|2<br />
|Ryynänen &amp; Klapuri<br />
|63.94%<br />
|65.99%<br />
|69.85%<br />
|81.87%<br />
|70.22%<br />
|73.85%<br />
|68.20%<br />
|73.70%<br />
|67.33%<br />
|10970<br />
|L<br />
|----<br />
|3<br />
|Paiva, R. 2<br />
|60.70%<br />
|58.16%<br />
|61.56%<br />
|70.92%<br />
|70.23%<br />
|74.06%<br />
|58.16%<br />
|61.56%<br />
|63.79%<br />
|45618<br />
|Y<br />
|----<br />
|4<br />
|Poliner &amp; Ellis<br />
|60.61%<br />
|62.47%<br />
|67.95%<br />
|83.02%<br />
|66.18%<br />
|71.10%<br />
|66.75%<br />
|72.86%<br />
|65.14%<br />
|5471<br />
|B 0<br />
|----<br />
|5<br />
|Marolt, M.<br />
|59.18%<br />
|58.06%<br />
|63.13%<br />
|73.35%<br />
|67.53%<br />
|72.24%<br />
|59.76%<br />
|66.73%<br />
|63.25%<br />
|12461<br />
|F<br />
|----<br />
|6<br />
|Paiva, R. 1<br />
|57.32%<br />
|62.23%<br />
|66.20%<br />
|74.76%<br />
|65.42%<br />
|69.32%<br />
|62.23%<br />
|66.20%<br />
|60.76%<br />
|44312<br />
|G<br />
|----<br />
|7<br />
|Goto, M.<br />
|49.68%<br />
|65.58%<br />
|71.47%<br />
|77.18%<br />
|56.46%<br />
|61.99%<br />
|65.58%<br />
|71.47%<br />
|54.89%<br />
|211<br />
|F<br />
|----<br />
|8<br />
|Vincent, E.<br />
|45.98%<br />
|59.17%<br />
|70.64%<br />
|77.61%<br />
|51.98%<br />
|62.36%<br />
|59.17%<br />
|70.64%<br />
|55.52%<br />
|251<br />
|G<br />
|----<br />
|9<br />
|(See note) Brossier, P.<br />
|3.2%<br />
|3.93%<br />
|8.06%<br />
|76.61%<br />
|3.63%<br />
|7.27%<br />
|3.93%<br />
|8.06%<br />
|6.43%<br />
|41<br />
|B 0<br />
|----<br />
|}<br />
<br />
The statistics are:<br />
<br />
1. Average Correctly transcribed voiced and unvoiced portions<br />
= (TPC + TN) / TO (i.e. proportion of frames with voicing and pitch right)<br />
<br />
2 . Averaged correctly transcribed voiced instants<br />
= TPC / GV (proportion of truly-voiced frames with vx, pitch right)<br />
<br />
3. Averaged correctly transcribed voiced instants mapped to one octave<br />
= TPCch / GV (statistic 2 without octave errors)<br />
<br />
4. Average estimation of temporal boundaries of the melodic segments<br />
= (TP + TN) / TO (overall voicing detection frame accuracy) <br />
<br />
5. Average F-measure<br />
Kris defined PRECIS = (TPC + TN)/(TP + TN + FP)<br />
and RECALL = (TPC + TN)/(TP + TN + FN), thus <br />
= 2*PRECIS*RECALL/(PRECIS + RECALL) (a hybrid of pitch and voicing)<br />
<br />
6. Average F-measure mapped to one octave<br />
[as above with TPC replaced by TPCch]<br />
<br />
7. Average Correctly transcribed instants (ignoring segmentation errors)<br />
= (TPC + FNC) / GV (pitch accuracy on pitched frames ignoring voicing)<br />
<br />
8. Average Correctly transcribed instants mapped to one octave (ignoring<br />
segmentation errors)<br />
= (TPCch + FNCch) / GV (statistic 7 without octave errors)<br />
<br />
9. Average Correctly transcribed voiced and unvoiced portions mapped to<br />
one Octave<br />
= (TPCch + TN) / TO (statistic 1 without octave errors)<br />
<br />
<br />
<br />
==McNemars Test Results==<br />
<br />
Statistical probability that algorithms have same error function. Note: Results of less than 5% indicate significant differences, results of 1% or less indicate highly significant differences.<br />
<br />
{| border="1" cellspacing="0"<br />
|-style="background: yellow;"<br />
!<br />
!Ryynänen &amp; Klapuri<br />
!Goto, M.<br />
!Brossier, P.<br />
!Vincent, E.<br />
!Poliner &amp; Ellis<br />
!Paiva, R. (2)<br />
!Paiva, R. (1)<br />
!Marolt, M.<br />
!Dressler, K.<br />
|----<br />
|Ryynänen &amp; Klapuri<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Goto, M.<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Brossier, P.<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Vincent, E.<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Poliner &amp; Ellis<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (2)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|--<br />
|----<br />
|Paiva, R. (1)<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|--<br />
|--<br />
|----<br />
|Marolt, M.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|2.71%<br />
|0%<br />
|n/a<br />
|--<br />
|----<br />
|Dressler, K.<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|0%<br />
|n/a<br />
|----<br />
|}</div>
Singh14