Difference between revisions of "2016:Audio Chord Estimation Results"

From MIREX Wiki
(Correct external link formatting)
(Detailed Results)
 
(2 intermediate revisions by the same user not shown)
Line 45: Line 45:
 
=====Billboard 2012=====
 
=====Billboard 2012=====
 
<csv>2016/ace/2016-Billboard2012.csv</csv>
 
<csv>2016/ace/2016-Billboard2012.csv</csv>
=====Bllboard 2013=====
+
=====Billboard 2013=====
 
<csv>2016/ace/2016-Billboard2013.csv</csv>
 
<csv>2016/ace/2016-Billboard2013.csv</csv>
 
=====JayChou 2015=====
 
=====JayChou 2015=====
Line 63: Line 63:
 
===Detailed Results===
 
===Detailed Results===
  
In progress<!--More details about the performance of the algorithms, including per-song performance and supplementary statistics, are available from these archives:
+
More details about the performance of the algorithms, including per-song performance, confusion matrices and supplementary statistics, are available in this [https://music-ir.org/mirex/results/2016/ace/detailled-results-2016.zip zip-file].
 
 
* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Results.zip Isophonics2009Results.zip]
 
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Results.zip Billboard2012Results.zip]
 
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Results.zip Billboard2013Results.zip]
 
* [https://music-ir.org/mirex/results/2015/ace/JayChou2015Results.zip JayChou2015Results.zip]-->
 
  
 
===Algorithmic Output===
 
===Algorithmic Output===
  
In progress <!--The raw output of the algorithms are available in the archives below. This can be used to experiment with alternative evaluation measures and statistics.
+
The raw output of the algorithms are available on [https://github.com/ismir-mirex/ace-output/tree/master/2016 GitHub]. They can be used to experiment with alternative evaluation measures and statistics.
 
 
* [https://music-ir.org/mirex/results/2015/ace/Isophonics2009Output.zip Isophonics2009Output.zip]
 
* [https://music-ir.org/mirex/results/2015/ace/Billboard2012Output.zip Billboard2012Output.zip]
 
* [https://music-ir.org/mirex/results/2015/ace/Billboard2013Output.zip Billboard2013Output.zip]-->
 

Latest revision as of 07:29, 30 August 2016

Introduction

This page contains the results of the 2016 edition of the MIREX automatic chord estimation tasks. This edition was the fourth one since the reorganization of the evaluation procedure in 2013. The results can therefore be directly compared to those of the last three years. Chord labels are evaluated according to five different chord vocabularies and the segmentation is also assessed. Additional information about the used measures can be found on the page of the 2013 edition.

What’s new?

Software

All software used for the evaluation has been made open-source. The evaluation framework is described by Pauwels and Peeters (2013). The corresponding binaries and code repository can be found on GitHub and the used measures are available as presets. The raw algorithmic output provided below makes it possible to calculate the additional measures from the paper (separate results for tetrads, etc.), in addition to those presented below. More help can be found in the readme.

The statistical comparison between the different submissions is explained in Burgoyne et al. (2014). The software is available at BitBucket. It uses the detailed results provided below as input.

Submissions

Abstract Contributors
CM1 (Chordino) PDF Chris Cannam, Matthias Mauch
DK1-DK4 PDF Junqi Deng, Yu-Kwong Kwok
FK2, FK4 PDF Filip Korzeniowski
KO1 (shineChords) PDF Maksim Khadkevich, Maurizio Omologo

Results

Summary

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

Isophonics 2009
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 78.56 75.41 72.48 54.67 52.26 85.90 87.17 86.09
DK1 79.21 76.19 74.00 66.02 64.15 85.71 82.62 91.23
DK2 77.84 74.49 71.93 61.61 59.47 85.82 82.72 91.28
DK3 80.03 77.55 74.79 68.40 65.88 85.81 82.50 91.53
DK4 76.05 72.96 71.41 62.77 61.44 78.19 87.97 72.43
FK2 86.09 85.53 82.24 74.42 71.54 87.76 85.79 90.73
FK4 82.28 80.93 78.03 70.91 68.26 85.62 82.40 90.89
KO1 82.93 82.19 79.61 76.04 73.43 87.69 85.66 91.24

download these results as csv

Billboard 2012
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 74.15 72.22 70.21 55.35 53.40 83.64 85.31 83.39
DK1 75.28 73.57 71.87 59.98 58.53 83.35 80.26 88.52
DK2 73.77 71.69 69.86 58.66 57.00 83.57 80.40 88.70
DK3 75.92 74.75 72.69 53.42 51.67 83.39 79.97 88.92
DK4 72.59 70.85 69.78 56.29 55.36 76.13 87.72 70.05
FK2 85.64 85.38 82.55 60.70 58.38 87.62 86.09 90.13
FK4 79.23 78.62 76.20 56.53 54.51 85.09 81.98 89.94
KO1 77.45 75.58 73.51 57.68 55.82 84.16 82.80 87.44

download these results as csv

Billboard 2013
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 71.16 67.28 65.20 48.99 47.17 81.54 83.11 82.63
DK1 72.06 68.69 67.26 54.54 53.29 80.82 77.58 88.06
DK2 70.18 66.54 64.66 52.97 51.41 80.85 77.68 88.02
DK3 72.39 68.53 66.55 48.99 47.28 80.76 77.26 88.30
DK4 69.56 65.83 64.78 51.81 50.93 74.55 86.31 69.18
FK2 80.07 77.89 75.42 55.41 53.22 82.94 82.43 86.80
FK4 74.66 71.85 69.44 51.93 49.80 80.61 77.19 88.70
KO1 75.36 71.39 69.43 53.57 51.78 81.63 79.61 87.75

download these results as csv

JayChou 2015
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 72.75 72.08 65.48 54.39 48.98 86.60 86.89 86.91
DK1 74.70 73.87 70.33 54.98 52.25 86.76 82.78 91.79
DK2 72.19 72.55 69.10 54.09 51.46 87.09 83.35 91.75
DK3 75.01 74.75 63.56 49.27 40.24 86.76 82.54 92.08
DK4 71.51 69.03 65.93 50.07 47.45 78.11 87.87 70.56
FK2 79.51 78.66 68.15 50.69 42.34 86.81 85.43 88.56
FK4 76.13 75.44 64.36 49.69 40.74 84.55 81.22 88.95
KO1 78.73 77.69 66.87 54.16 44.55 88.46 87.12 90.11

download these results as csv

RobbieWilliams 2016
Algorithm MirexRoot MirexMajMin MirexMajMinBass MirexSevenths MirexSeventhsBass MeanSeg UnderSeg OverSeg
CM1 81.90 78.25 76.05 57.92 55.90 87.96 88.96 87.45
DK1 81.50 77.77 76.10 68.88 67.34 87.03 83.22 92.11
DK2 79.01 75.97 73.57 65.26 62.98 87.20 83.40 92.23
DK3 81.85 78.56 76.16 74.71 72.55 86.98 82.95 92.34
DK4 78.92 75.15 73.66 66.72 65.34 81.82 88.44 76.88
FK2 88.53 87.23 84.19 82.57 79.88 90.04 88.62 91.88
FK4 83.37 80.96 78.42 77.04 74.76 87.22 84.50 91.02
KO1 83.55 80.33 78.16 73.54 71.39 88.04 85.39 91.68

download these results as csv

Comparative Statistics

In progress

Detailed Results

More details about the performance of the algorithms, including per-song performance, confusion matrices and supplementary statistics, are available in this zip-file.

Algorithmic Output

The raw output of the algorithms are available on GitHub. They can be used to experiment with alternative evaluation measures and statistics.