Difference between revisions of "2009:Audio Chord Detection Results"

From MIREX Wiki
(Overall Summary Results)
m (Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2009:\1))
Line 1: Line 1:
 
==Introduction==
 
==Introduction==
These are the results for the 2009 running of the Audio Chord Detection task set. For background information about this task set please refer to the [[Audio Chord Detection]] page.
+
These are the results for the 2009 running of the Audio Chord Detection task set. For background information about this task set please refer to the [[2009:Audio Chord Detection]] page.
  
 
===Task Descriptions===
 
===Task Descriptions===
Line 50: Line 50:
  
 
=====Task 1 Results Per Artist/Album=====
 
=====Task 1 Results Per Artist/Album=====
[[Audio Chord Detection Results Per Album#Task 1]]
+
[[2009:Audio Chord Detection Results Per Album#Task 1]]
  
 
=====Task 1 Friedman's Test for Significant Differences=====
 
=====Task 1 Friedman's Test for Significant Differences=====
Line 71: Line 71:
  
 
=====Task 2 Results Per Artist/Album=====
 
=====Task 2 Results Per Artist/Album=====
[[Audio Chord Detection Results Per Album#Task 2]]
+
[[2009:Audio Chord Detection Results Per Album#Task 2]]
  
 
=====Task 2 Friedman's Test for Significant Differences=====
 
=====Task 2 Friedman's Test for Significant Differences=====

Revision as of 13:31, 13 May 2010

Introduction

These are the results for the 2009 running of the Audio Chord Detection task set. For background information about this task set please refer to the 2009:Audio Chord Detection page.

Task Descriptions

Beatles data donated by Christopher Harte and extra dataset of around 37 songs donated by Matthias Mauch was used. 7 songs were excluded from the list where the average score for these songs across participants were less than 0.25

Task 1 (Pretrained Systems) Go to Task 1 Results:


Task 2 (Train-Test Systems) Go to Task 2 Results: System trained on ~2/3 of the dataset and tested on ~1/3. Folds were divided such that 2/3 of each album was represented in the train list and 1/3 in the test list.

Overlap score was calculated as the ratio between the overlap of the ground truth and detected chords and ground truth duration. Also a secondary overlap score was calculated by ignoring the major-minor variations of the detected chord (e.g., C major == C minor, etc.). The weighted average is computed across the songs by weighting each score by its the duration.

General Legend

Team ID

CH = Christopher Harte
DE = Dan Ellis
KO1 = Maksim Khadkevich, Maurizio Omologo
KO2 = Maksim Khadkevich, Maurizio Omologo
MD = Matthias Mauch, Katy Noland, Simon Dixon
OGF1 = Laurent Oudre, Yves Grenier,Cedric Fevotte (majmin)
OGF2 = Laurent Oudre, Yves Grenier,Cedric Fevotte (majmin7)
PP = Helene Papadopoulos, Geoffroy Peeters
PVM1 = Johan Pauwels, Matthias Varewyck,Jean-Pierre Martens
PVM2 = Johan Pauwels, Matthias Varewyck,Jean-Pierre Martens
RRHS1 = Thomas Rocher, Matthias Robine, Pierre Hanna,Robert Strandh
RSHS2 = Thomas Rocher, Matthias Robine, Pierre Hanna,Robert Strandh
RSHS3 = Thomas Rocher, Matthias Robine, Pierre Hanna,Robert Strandh
RUSUSL = J.T.Reed,Yushi Ueda,S.Siniscalchi,Yuki Uchiyama,Shigeki Sagayama,C.H.Lee
WEJ1 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ2 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ3 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ4 = Adrian Weller, Daniel Ellis, Tony Jebara

Overall Summary Results

Task 1 Results

Task 1 Overall Results

file /nema-raid/www/mirex/results/chord/task1/summary/task1.overall.csv not found

Task 1 Run Times

TBA file /nema-raid/www/mirex/results/chord/task1/task1_runtimes.csv not found

Task 1 Results Per Artist/Album

2009:Audio Chord Detection Results Per Album#Task 1

Task 1 Friedman's Test for Significant Differences

The Tukey-Kramer HSD multi-comparison data below was generate using the following MATLAB instruction.

Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/chord/task1/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv not found

https://music-ir.org/mirex/2009/results/chord/task1/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png

Task 2 Results

Task 2 Overall Results

file /nema-raid/www/mirex/results/chord/task2/summary/task2.overall.csv not found

Task 2 Run Times

TBA file /nema-raid/www/mirex/results/chord/task2/task2_runtimes.csv not found

Task 2 Results Per Artist/Album

2009:Audio Chord Detection Results Per Album#Task 2

Task 2 Friedman's Test for Significant Differences

The Tukey-Kramer HSD multi-comparison data below was generate using the following MATLAB instruction.

Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/chord/task2/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv not found

https://music-ir.org/mirex/2009/results/chord/task2/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png

Individual Results

Task1

CH = Christopher Harte
DE = Dan Ellis
KO1 = Maksim Khadkevich, Maurizio Omologo
KO2 = Maksim Khadkevich, Maurizio Omologo
MD = Matthias Mauch, Simon Dixon
OGF1 = Laurent Oudre,Yves Grenier,Cedric Fevotte
OGF2 = Laurent Oudre,Yves Grenier,Cedric Fevotte
PP = Helene Papadopoulos, Geoffroy Peeters
PVM1 = Johan Pauwels,Matthias Varewyck,Jean-Pierre Martens
PVM2 = Johan Pauwels,Matthias Varewyck,Jean-Pierre Martens
RRHS1 = Thomas Rocher, Matthias Robine,Pierre Hanna,Robert Strandh
RSHS2 = Thomas Rocher, Matthias Robine,Pierre Hanna,Robert Strandh
RSHS3 = Thomas Rocher, Matthias Robine,Pierre Hanna,Robert Strandh

Task2

RUSUSL = J.T.Reed,Yushi Ueda,S.Siniscalchi,Yuki Uchiyama,Shigeki Sagayama,C.H.Lee
WEJ1 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ2 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ3 = Adrian Weller, Daniel Ellis, Tony Jebara
WEJ4 = Adrian Weller, Daniel Ellis, Tony Jebara

Evaluations by Christopher Harte's Metrics

TBA