Difference between revisions of "2017:Audio Key Detection Results"

From MIREX Wiki
(Created page with "Coming soon.")
 
(First addition of AKD results)
Line 1: Line 1:
Coming soon.
+
==Introduction==
 +
 
 +
This page contains the results of the 2017 edition of the MIREX automatic key detection estimation task.
 +
 
 +
==What’s new?==
 +
* The NEMA system was retired this year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
 +
* New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the [http://isophonics.net/content/reference-annotations Isophonics set] and the [http://ispg.deib.polimi.it/mir-software.html Robbie Williams set], but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
 +
* New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their [https://arxiv.org/abs/1706.02921 2017 EUSIPCO paper]). The annotations are [http://www.cp.jku.at/people/korzeniowski/bb.zip freely available]
 +
 
 +
==Submissions==
 +
 
 +
{| class="wikitable"
 +
!
 +
! Abstract
 +
! Contributors
 +
|-
 +
| BD1, BD2
 +
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/BD1.pdf PDF]
 +
| Gilberto Bernardes, Matthew Davies
 +
|-
 +
| CN1
 +
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/CN1.pdf PDF]
 +
| Chris Cannam, Katy Noland
 +
|-
 +
| FK1
 +
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/FK1.pdf PDF]
 +
| Filip Korzeniowski
 +
|-
 +
| HS1-HS3
 +
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/HS1.pdf PDF]
 +
| Hendrik Schreiber
 +
|-
 +
| PRGR5
 +
| style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2017/PRGR5.pdf PDF]
 +
| Adam Pluta, Marcin Gawrysz
 +
|}
 +
 
 +
==Results==
 +
 
 +
===Summary===
 +
 
 +
All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).
 +
 
 +
=====MIREX2005Key=====
 +
<csv>2017/akd/2017-MIREX2005Key.csv</csv>
 +
=====GiantStepsKey=====
 +
<csv>2017/akd/2017-GiantStepsKey.csv</csv>
 +
=====PresegmentedKeyIsophonics=====
 +
<csv>2017/akd/2017-PresegmentedKeyIsophonics.csv</csv>
 +
=====PresegmentedKeyRobbieWilliams=====
 +
<csv>2017/akd/2017-PresegmentedKeyRobbieWilliams.csv</csv>
 +
=====Billboard2012Key=====
 +
<csv>2017/akd/2017-Billboard2012Key.csv</csv>
 +
 
 +
The submission PRGR5 is currently not able to complete the task without crashing, but hopefully this can still be remedied. These tables will be updated as soon as this is the case.
 +
 
 +
==Note==
 +
This page will be further updated with more detailed info and extended results (extra statistics, per-file results, confusion matrices) once I get back to a country where the wifi is better and Google's services aren't blocked (which includes the captcha's for this bloody wiki). That will be around November 10th. --Johan

Revision as of 22:09, 1 November 2017

Introduction

This page contains the results of the 2017 edition of the MIREX automatic key detection estimation task.

What’s new?

  • The NEMA system was retired this year, since a bug has been found in the calculation of the results. Keys with tonics related by a fifth and the same mode (a.k.a. adjacent keys) are supposed to get a score of 0.5, but only ascending fifths (going from ground-truth to estimation) were counted, not descending ones. It has been brought to my attention that the description of the measure on the wiki has been ambiguous for years, and probably the NEMA implementer got confused by this. However, the intention has always been to count ascending and descending fifth (or fourth) relationships between the tonics (in my humble opinion).
  • New datasets: "PresegmentedKeyIsophonics" and "PresegmentedKeyRobbieWilliams" use the local key annotations for the Isophonics set and the Robbie Williams set, but have been split into separate files according to the local key annotations. The segments annotated with major and minor modes have been retained and were presented to the submissions. Therefore their results are slightly optimistic in the sense that the segments are guaranteed to contain just a single key, which is not the case for real-world songs. Keep also in mind that some files are strongly correlated (different segments or even repeated chorusses of the same song). Any statistical analysis of the results (e.g. pairwise significance tests) that relies on independence between files is consequently invalid.
  • New dataset: "Billboard2012Key" is the subset of the Billboard2012 chord dataset for which it was possible to derive the key automatically from the chord annotations (using the procedure outlined by Korzeniowski & Widmer in their 2017 EUSIPCO paper). The annotations are freely available

Submissions

Abstract Contributors
BD1, BD2 PDF Gilberto Bernardes, Matthew Davies
CN1 PDF Chris Cannam, Katy Noland
FK1 PDF Filip Korzeniowski
HS1-HS3 PDF Hendrik Schreiber
PRGR5 PDF Adam Pluta, Marcin Gawrysz

Results

Summary

All figures can be interpreted as percentages and range from 0 (worst) to 100 (best).

MIREX2005Key
Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
BD1 84.84 75.80 15.42 3.75 1.04 0.16 3.83
BD2 58.44 42.97 12.30 30.27 1.20 0.00 13.26
CN1 88.07 82.75 8.31 2.72 1.76 0.08 4.39
FK1 80.19 70.77 15.73 3.43 2.64 0.08 7.35
HS1 80.25 68.69 18.61 6.71 1.20 0.00 4.79
HS2 65.85 50.32 13.82 27.40 2.00 0.16 6.31
HS3 66.96 52.16 12.06 28.19 1.60 0.16 5.83
PRGR5 []

download these results as csv

GiantStepsKey
Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
BD1 59.62 46.03 19.87 9.44 4.14 1.66 18.87
BD2 71.84 65.40 7.45 5.96 4.64 3.15 13.41
CN1 50.53 39.74 11.92 13.24 4.30 2.65 28.15
FK1 74.11 67.38 7.45 7.12 4.30 3.15 10.60
HS1 69.35 62.09 8.44 7.28 4.30 3.31 14.57
HS2 27.02 9.11 2.81 45.53 14.24 0.00 28.31
HS3 46.64 33.44 4.47 32.12 6.62 1.66 21.69
PRGR5 []

download these results as csv

PresegmentedKeyIsophonics
Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
BD1 75.06 65.98 13.56 5.06 3.91 2.30 9.20
BD2 46.69 32.87 3.45 30.80 14.25 1.38 17.24
CN1 64.99 55.40 13.56 4.60 7.13 3.91 15.40
FK1 70.85 62.53 8.28 6.44 11.26 1.84 9.66
HS1 64.05 54.25 7.82 14.25 8.05 1.61 14.02
HS2 67.93 56.78 10.57 18.62 1.38 1.61 11.03
HS3 70.23 59.77 8.74 18.62 2.53 1.61 8.74
PRGR5 []

download these results as csv

PresegmentedKeyRobbieWilliams
Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
BD1 70.79 64.55 6.88 8.99 0.53 2.65 16.40
BD2 56.83 46.56 8.99 16.40 4.23 2.65 21.16
CN1 62.28 53.97 10.58 7.94 3.17 7.41 16.93
FK1 76.30 68.25 12.17 3.70 4.23 0.00 11.64
HS1 77.14 68.78 10.05 8.99 3.17 0.00 8.99
HS2 67.78 52.38 11.11 31.75 1.59 0.00 3.17
HS3 70.53 56.61 10.05 28.57 1.59 0.00 3.17
PRGR5 []

download these results as csv

Billboard2012Key
Algorithm Mirex Correct Adjacent Relative Parallel Chromatic Other
BD1 78.84 67.12 18.49 5.48 4.11 0.68 4.11
BD2 46.71 29.45 3.42 41.78 15.07 0.00 10.27
CN1 67.40 58.22 12.33 5.48 6.85 6.85 10.27
FK1 78.22 69.86 6.85 6.85 14.38 0.00 2.05
HS1 74.79 65.75 6.85 10.96 11.64 0.00 4.79
HS2 82.33 75.34 7.53 10.27 0.68 0.00 6.16
HS3 81.85 73.97 9.59 10.27 0.00 0.68 5.48
PRGR5 16.03 3.42 4.11 24.66 15.75 2.05 50.00

download these results as csv

The submission PRGR5 is currently not able to complete the task without crashing, but hopefully this can still be remedied. These tables will be updated as soon as this is the case.

Note

This page will be further updated with more detailed info and extended results (extra statistics, per-file results, confusion matrices) once I get back to a country where the wifi is better and Google's services aren't blocked (which includes the captcha's for this bloody wiki). That will be around November 10th. --Johan