Difference between revisions of "2008:Audio Artist Identification Results"

From MIREX Wiki
(→‎MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds)
(→‎Results By Algorithm)
 
(21 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
==Introduction==
 
==Introduction==
−
These are the results for the 2008 running of the Audio Artist Identification task. For background information about this task set please refer to the [[Audio Artist Identification]] page.  
+
These are the results for the 2008 running of the Audio Artist Identification task. For background information about this task set please refer to the [[2008:Audio Artist Identification]] page.  
  
 
===General Legend===
 
===General Legend===
 
====Team ID====
 
====Team ID====
−
'''LRPPI1''' = [https://www.music-ir.org/mirex/2008/abs/.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 1]<br />
+
'''GP2''' = [https://www.music-ir.org/mirex/abstracts/2008/Peeters_2008_ISMIR_MIREX.pdf G. Peeters]<br />
−
'''LRPPI2''' = [https://www.music-ir.org/mirex/2008/abs/.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 2]<br />
+
'''GT1 (mono)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br />
−
'''LRPPI3''' = [https://www.music-ir.org/mirex/2008/abs/.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 3]<br />
+
'''GT2 (stereo)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br />
−
'''LRPPI4''' = [https://www.music-ir.org/mirex/2008/abs/.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de Leon, J. M. I├▒esta 4]<br />
+
'''GT3 (multicore)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br />
−
'''ME1''' = [https://www.music-ir.org/mirex/2008/abs/AI_CC_GC_MC_AS_mandel.pdf M. I. Mandel, D. P. W. Ellis 1]<br />
+
'''LRPPI1''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1]<br />
−
'''ME2''' = [https://www.music-ir.org/mirex/2008/abs/AI_CC_GC_MC_AS_mandel.pdf M. I. Mandel, D. P. W. Ellis 2]<br />
+
'''LRPPI2''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2]<br />
−
'''ME3''' = [https://www.music-ir.org/mirex/2008/abs/AI_CC_GC_MC_AS_mandel.pdf M. I. Mandel, D. P. W. Ellis 3]<br />
+
'''LRPPI3''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3]<br />
−
'''GP''' = [https://www.music-ir.org/mirex/2008/abs/.pdf G. Peeters]<br />
+
'''LRPPI4''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4]<br />
−
'''GT1''' = [https://www.music-ir.org/mirex/2008/abs/.pdf G. Tzanetakis]<br />
+
'''ME1''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf M. I. Mandel, D. P. W. Ellis 1]<br />
−
'''GT2''' = [https://www.music-ir.org/mirex/2008/abs/.pdf G. Tzanetakis]<br />
+
'''ME2''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf M. I. Mandel, D. P. W. Ellis 2]<br />
−
'''GT3''' = [https://www.music-ir.org/mirex/2008/abs/.pdf G. Tzanetakis]<br />
+
'''ME3''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf M. I. Mandel, D. P. W. Ellis 3]<br />
  
 
==Overall Summary Results==
 
==Overall Summary Results==
 
===MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds===
 
===MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds===
  
−
<csv>artist/artist.avg.results.csv</csv>
+
<csv>2008/artist/audioartist.avg.results.csv</csv>
 +
 
 +
=====Accuracy Across Folds=====
 +
 
 +
<csv>2008/artist/audioartist.results.fold.csv</csv>
 +
 
 +
=====Accuracy Across Categories=====
 +
 
 +
<csv>2008/artist/audioartist.results.class.csv</csv>
  
 
===MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices===
 
===MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices===
  
 +
====MIREX 2008 Audio Artist Classification Run Times====
 +
 +
<csv>2008/artist.runtime.csv</csv>
 +
 +
====CSV Files Without Rounding====
 +
[https://www.music-ir.org/mirex/results/2008/artist/audioartist_results_fold.csv audioartist_results_fold.csv]<br />
 +
[https://www.music-ir.org/mirex/results/2008/artist/audioartist_results_class.csv audioartist_results_class.csv]<br />
 +
 +
====Results By Algorithm====
 +
(.tar.gz) <br />
 +
'''GP2''' = [https://www.music-ir.org/mirex/results/2008/artist/GP2.tar.gz G. Peeters]<br />
 +
'''GT1''' = [https://www.music-ir.org/mirex/results/2008/artist/GT1.tar.gz G. Tzanetakis]<br />
 +
'''GT2''' = [https://www.music-ir.org/mirex/results/2008/artist/GT2.tar.gz G. Tzanetakis]<br />
 +
'''GT3''' = [https://www.music-ir.org/mirex/results/2008/artist/GT3.tar.gz G. Tzanetakis]<br />
 +
'''LRPPI1''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI1.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1]<br />
 +
'''LRPPI2''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI2.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2]<br />
 +
'''LRPPI3''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI3.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3]<br />
 +
'''LRPPI4''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI4.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4]<br />
 +
'''ME1''' = [https://www.music-ir.org/mirex/results/2008/artist/ME1.tar.gz I. M. Mandel, D. P. W. Ellis 1]<br />
 +
'''ME2''' = [https://www.music-ir.org/mirex/results/2008/artist/ME2.tar.gz I. M. Mandel, D. P. W. Ellis 2]<br />
 +
'''ME3''' = [https://www.music-ir.org/mirex/results/2008/artist/ME3.tar.gz I. M. Mandel, D. P. W. Ellis 3]<br />
 +
 +
===Friedman's Test for Significant Differences===
 +
====Classes vs. Systems====
 +
The Friedman test was run in MATLAB against the average accuracy for each class.
 +
 +
=====Friedman's Anova Table=====
 +
 +
<csv>2008/artist/perClassAccuracy.friedman.csv</csv>
 +
 +
=====Tukey-Kramer HSD Multi-Comparison=====
 +
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction.
 +
Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
 +
 +
<csv>2008/artist/perClassAccuracy.friedman.detail.csv</csv>
 +
 +
[[Image:2008_artist.perclassaccuracy.friedman.tukeykramerhsd.png]]
 +
 +
====Folds vs. Systems====
 +
The Friedman test was run in MATLAB against the accuracy for each fold.
 +
 +
=====Friedman's Anova Table=====
 +
 +
<csv>2008/artist/perFoldAccuracy.friedman.csv</csv>
  
 +
=====Tukey-Kramer HSD Multi-Comparison=====
 +
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction.
 +
Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
  
−
===MIREX 2008 Audio Artist Classification Run Times===
+
<csv>2008/artist/perFoldAccuracy.friedman.detail.csv</csv>
  
−
<csv>artist.runtime.csv</csv>
+
[[Image:2008_artist.perfoldaccuracy.friedman.tukeykramerhsd.png]]
  
 
[[Category: Results]]
 
[[Category: Results]]

Latest revision as of 17:01, 23 July 2010

Introduction

These are the results for the 2008 running of the Audio Artist Identification task. For background information about this task set please refer to the 2008:Audio Artist Identification page.

General Legend

Team ID

GP2 = G. Peeters
GT1 (mono) = G. Tzanetakis
GT2 (stereo) = G. Tzanetakis
GT3 (multicore) = G. Tzanetakis
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4
ME1 = M. I. Mandel, D. P. W. Ellis 1
ME2 = M. I. Mandel, D. P. W. Ellis 2
ME3 = M. I. Mandel, D. P. W. Ellis 3

Overall Summary Results

MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds

Participant Average Classifcation Accuracy
GP2 1.11%
GT1 33.66%
GT2 43.47%
GT3 35.27%
LRPPI1 35.42%
LRPPI2 33.20%
LRPPI3 29.87%
LRPPI4 32.52%
ME1 47.65%
ME2 47.16%
ME3 47.25%

download these results as csv

Accuracy Across Folds
Classification fold GP2 GT1 GT2 GT3 LRPPI1 LRPPI2 LRPPI3 LRPPI4 ME1 ME2 ME3
0 0.005 0.342 0.443 0.344 0.367 0.348 0.306 0.344 0.467 0.460 0.462
1 0.018 0.329 0.447 0.341 0.333 0.313 0.288 0.308 0.481 0.477 0.475
2 0.011 0.339 0.413 0.373 0.363 0.335 0.303 0.324 0.482 0.478 0.481

download these results as csv

Accuracy Across Categories
Class GP2 GT1 GT2 GT3 LRPPI1 LRPPI2 LRPPI3 LRPPI4 ME1 ME2 ME3
112 0.000 0.633 0.600 0.433 0.300 0.333 0.300 0.233 0.633 0.633 0.633
ABBA 0.000 0.300 0.233 0.333 0.300 0.167 0.200 0.367 0.500 0.500 0.500
ACDC 0.067 0.433 0.533 0.533 0.533 0.467 0.500 0.467 0.500 0.500 0.500
AEROSMITH 0.100 0.195 0.161 0.195 0.067 0.100 0.100 0.200 0.127 0.127 0.127
ARTHURBIGBOYCRUDUP 0.000 0.933 1.000 1.000 0.900 0.800 0.667 0.733 1.000 1.000 1.000
ARTIESHAW 0.000 0.400 0.700 0.300 0.333 0.400 0.333 0.367 0.433 0.400 0.400
BENNYCARTER 0.000 0.300 0.667 0.367 0.200 0.167 0.133 0.100 0.300 0.300 0.300
BENNYGOODMAN 0.000 0.133 0.367 0.233 0.200 0.233 0.300 0.167 0.367 0.400 0.400
BENNYMOTEN 0.000 0.467 0.700 0.333 0.433 0.267 0.167 0.333 0.433 0.433 0.467
BESSIESMITH 0.033 1.000 1.000 0.900 1.000 0.933 0.967 0.967 1.000 1.000 1.000
BIGBILLBROONZY 0.000 0.600 0.800 0.600 0.533 0.500 0.400 0.500 0.733 0.733 0.733
BILLCOLEMAN 0.000 0.200 0.700 0.133 0.233 0.100 0.133 0.133 0.267 0.267 0.267
BILLIEHOLIDAY 0.000 0.400 0.633 0.333 0.400 0.433 0.300 0.367 0.700 0.700 0.667
BIXBEIDERBECKE 0.000 0.600 0.800 0.600 0.800 0.667 0.600 0.700 0.733 0.733 0.733
BLINK182 0.000 0.167 0.267 0.467 0.200 0.367 0.333 0.333 0.367 0.367 0.367
BOBMARLEYTHEWAILERS 0.000 0.567 0.600 0.633 0.833 0.700 0.733 0.800 0.700 0.700 0.700
BONJOVI 0.000 0.600 0.533 0.567 0.600 0.633 0.633 0.633 0.600 0.533 0.533
BROWNIEMCGHEE 0.000 0.800 0.867 0.833 0.800 0.933 0.700 0.833 0.933 0.933 0.933
BRUCESPRINGSTEEN 0.033 0.067 0.133 0.067 0.367 0.267 0.267 0.300 0.133 0.133 0.133
BRYANADAMS 0.000 0.200 0.400 0.200 0.167 0.133 0.133 0.100 0.367 0.367 0.367
BUNNYBERIGAN 0.000 0.333 0.500 0.433 0.333 0.433 0.433 0.300 0.467 0.433 0.433
CABCALLOWAY 0.000 0.467 0.533 0.500 0.133 0.133 0.133 0.200 0.600 0.600 0.633
CARLPERKINS 0.000 0.567 0.567 0.567 0.400 0.267 0.167 0.300 0.700 0.700 0.700
CHARLIEBARNET 0.000 0.200 0.400 0.300 0.567 0.300 0.333 0.400 0.267 0.267 0.267
CHICKWEBB 0.000 0.267 0.767 0.300 0.300 0.267 0.100 0.167 0.467 0.433 0.433
COUNTBASIE 0.000 0.167 0.300 0.200 0.100 0.133 0.133 0.167 0.333 0.333 0.333
CREEDENCECLEARWATERREVIVAL 0.000 0.500 0.633 0.500 0.533 0.367 0.500 0.533 0.400 0.367 0.367
DAVEMATTHEWSBAND 0.000 0.100 0.133 0.167 0.067 0.133 0.067 0.033 0.367 0.333 0.333
DEANMARTIN 0.000 0.467 0.667 0.567 0.467 0.367 0.367 0.233 0.767 0.767 0.767
DEFLEPPARD 0.000 0.467 0.433 0.667 0.467 0.567 0.467 0.533 0.467 0.467 0.467
DEPECHEMODE 0.000 0.233 0.333 0.200 0.333 0.400 0.167 0.267 0.600 0.600 0.600
DMX 0.067 0.700 0.800 0.700 0.700 0.733 0.733 0.767 0.900 0.900 0.900
DUKEELLINGTON 0.000 0.233 0.400 0.300 0.333 0.333 0.300 0.300 0.300 0.300 0.300
EDDIECONDON 0.000 0.733 0.767 0.733 0.467 0.467 0.533 0.567 0.467 0.467 0.467
ENYA 0.000 0.833 0.833 0.733 0.633 0.733 0.667 0.633 0.667 0.667 0.667
ERICCLAPTON 0.000 0.146 0.305 0.144 0.089 0.057 0.113 0.079 0.144 0.144 0.144
ERSKINEHAWKINS 0.000 0.067 0.100 0.067 0.333 0.333 0.233 0.367 0.333 0.333 0.333
EURYTHMICS 0.000 0.200 0.167 0.200 0.500 0.300 0.400 0.433 0.367 0.367 0.367
FATSWALLER 0.000 0.300 0.667 0.300 0.533 0.433 0.200 0.433 0.700 0.700 0.733
FLEETWOODMAC 0.000 0.000 0.000 0.000 0.033 0.000 0.000 0.033 0.200 0.200 0.200
FLETCHERHENDERSON 0.000 0.100 0.433 0.133 0.167 0.200 0.133 0.100 0.367 0.300 0.300
FOOFIGHTERS 0.300 0.400 0.533 0.667 0.033 0.033 0.133 0.033 0.400 0.400 0.433
FRANKSINATRA 0.000 0.433 0.467 0.400 0.400 0.433 0.467 0.433 0.700 0.667 0.667
FREDDYFENDER 0.000 0.433 0.400 0.433 0.500 0.400 0.200 0.333 0.600 0.533 0.533
GARTHBROOKS 0.000 0.733 0.767 0.633 0.733 0.633 0.533 0.667 0.733 0.733 0.733
GENEKRUPA 0.000 0.333 0.433 0.333 0.200 0.200 0.167 0.233 0.433 0.433 0.467
GENESIS 0.000 0.131 0.083 0.125 0.239 0.175 0.214 0.159 0.042 0.000 0.000
GEORGEJONES 0.000 0.533 0.767 0.567 0.567 0.600 0.600 0.533 0.700 0.700 0.700
GEORGEWETTLING 0.000 0.333 0.533 0.400 0.267 0.100 0.133 0.167 0.233 0.233 0.200
GLENNMILLER 0.000 0.100 0.300 0.200 0.267 0.333 0.133 0.367 0.300 0.300 0.367
GREENDAY 0.000 0.067 0.000 0.100 0.167 0.500 0.433 0.300 0.167 0.167 0.167
HARRYJAMES 0.000 0.267 0.400 0.267 0.233 0.167 0.233 0.167 0.600 0.600 0.600
INXS 0.000 0.033 0.067 0.133 0.133 0.100 0.067 0.100 0.367 0.367 0.367
JACKTEAGARDEN 0.000 0.133 0.233 0.100 0.067 0.167 0.133 0.100 0.400 0.367 0.367
JELLYROLLMORTON 0.000 0.067 0.233 0.100 0.333 0.300 0.233 0.333 0.467 0.467 0.467
JIMMIELUNCEFORD 0.000 0.167 0.267 0.167 0.133 0.300 0.167 0.233 0.400 0.400 0.433
JIMMIENOONE 0.000 0.400 0.733 0.333 0.367 0.200 0.167 0.200 0.467 0.500 0.500
JOHNNYDODDS 0.000 0.100 0.167 0.033 0.200 0.133 0.100 0.133 0.233 0.233 0.233
KOKOMOARNOLD 0.000 0.800 0.867 0.767 0.733 0.867 0.767 0.800 0.867 0.867 0.867
LEADBELLY 0.000 0.533 0.500 0.533 0.367 0.333 0.367 0.333 0.667 0.667 0.667
LIGHTNINHOPKINS 0.000 0.633 0.733 0.667 0.600 0.733 0.733 0.733 0.767 0.767 0.767
LIONELHAMPTON 0.000 0.233 0.433 0.300 0.167 0.333 0.167 0.233 0.467 0.400 0.467
LLCOOLJ 0.000 0.367 0.467 0.400 0.167 0.200 0.167 0.167 0.833 0.833 0.800
LOUISARMSTRONG 0.000 0.000 0.367 0.067 0.233 0.167 0.167 0.167 0.167 0.167 0.167
MADONNA 0.000 0.133 0.067 0.167 0.067 0.033 0.000 0.033 0.367 0.367 0.367
MARIAHCAREY 0.000 0.433 0.600 0.600 0.400 0.400 0.433 0.467 0.800 0.800 0.833
MARILYNMANSON 0.133 0.000 0.067 0.033 0.067 0.100 0.067 0.067 0.367 0.367 0.333
METALLICA 0.000 0.417 0.507 0.424 0.507 0.424 0.444 0.382 0.583 0.583 0.583
MICHAELJACKSON 0.000 0.300 0.267 0.367 0.300 0.267 0.167 0.400 0.267 0.267 0.267
MICKEYGILLEY 0.000 0.567 0.600 0.500 0.667 0.633 0.400 0.467 0.667 0.667 0.633
MUDDYWATERS 0.000 0.500 0.500 0.533 0.333 0.333 0.233 0.433 0.700 0.700 0.700
MUGGSYSPANIER 0.000 0.233 0.267 0.367 0.367 0.300 0.233 0.300 0.533 0.500 0.500
NATKINGCOLE 0.000 0.333 0.633 0.400 0.433 0.400 0.367 0.400 0.700 0.633 0.633
NINEINCHNAILS 0.300 0.033 0.133 0.033 0.067 0.000 0.100 0.000 0.133 0.167 0.100
NIRVANA 0.000 0.167 0.067 0.067 0.233 0.300 0.233 0.233 0.333 0.367 0.367
ORIGINALDIXIELANDJAZZBAND 0.000 1.000 1.000 1.000 0.900 0.733 0.767 0.833 0.900 0.900 0.933
PHILCOLLINS 0.000 0.067 0.100 0.067 0.133 0.233 0.300 0.167 0.433 0.433 0.433
PINKFLOYD 0.000 0.098 0.183 0.278 0.299 0.444 0.156 0.217 0.169 0.169 0.169
QUEEN 0.000 0.067 0.167 0.100 0.267 0.167 0.167 0.200 0.367 0.367 0.367
RADIOHEAD 0.067 0.067 0.067 0.033 0.233 0.167 0.133 0.133 0.033 0.033 0.033
RAYCHARLES 0.000 0.302 0.446 0.346 0.164 0.201 0.204 0.134 0.594 0.594 0.594
REDNICHOLS 0.000 0.367 0.367 0.500 0.367 0.300 0.367 0.400 0.500 0.500 0.500
RKELLY 0.000 0.467 0.433 0.300 0.533 0.400 0.433 0.500 0.533 0.533 0.533
RODSTEWART 0.000 0.067 0.033 0.033 0.167 0.133 0.233 0.100 0.167 0.167 0.167
ROXETTE 0.000 0.167 0.367 0.200 0.167 0.133 0.133 0.133 0.433 0.433 0.433
SADE 0.000 0.133 0.133 0.167 0.133 0.133 0.133 0.133 0.367 0.367 0.367
SANTANA 0.000 0.200 0.133 0.067 0.100 0.067 0.067 0.100 0.133 0.133 0.133
SIDNEYBECHET 0.000 0.300 0.467 0.233 0.300 0.267 0.167 0.100 0.300 0.300 0.300
SONNYBOYWILLIAMSONI 0.000 0.700 0.800 0.733 0.633 0.633 0.633 0.567 0.867 0.867 0.867
STANKENTON 0.000 0.500 0.567 0.533 0.467 0.433 0.300 0.400 0.500 0.467 0.467
SUBLIME 0.000 0.100 0.233 0.233 0.233 0.033 0.167 0.200 0.400 0.400 0.400
THEBEATLES 0.000 0.389 0.167 0.250 0.167 0.222 0.306 0.250 0.278 0.278 0.278
THEDOORS 0.000 0.033 0.067 0.067 0.033 0.000 0.000 0.067 0.233 0.200 0.200
THEROLLINGSTONES 0.028 0.361 0.222 0.278 0.306 0.111 0.083 0.167 0.139 0.139 0.139
TIMMCGRAW 0.000 0.433 0.500 0.300 0.500 0.467 0.400 0.500 0.833 0.867 0.833
TINATURNER 0.000 0.033 0.033 0.095 0.151 0.192 0.167 0.159 0.323 0.323 0.323
TOMJONES 0.000 0.767 0.967 0.833 0.767 0.733 0.633 0.667 0.967 0.967 0.967
TOMMYDORSEY 0.000 0.433 0.433 0.467 0.400 0.300 0.233 0.233 0.433 0.433 0.433
U2 0.000 0.233 0.367 0.233 0.300 0.267 0.133 0.200 0.333 0.333 0.333
VANHALEN 0.000 0.174 0.278 0.174 0.375 0.207 0.269 0.278 0.238 0.238 0.204
WILLIENELSON 0.000 0.667 0.800 0.733 0.733 0.733 0.800 0.867 0.900 0.933 0.933
WOODYHERMAN 0.000 0.033 0.300 0.083 0.167 0.100 0.033 0.033 0.267 0.267 0.267

download these results as csv

MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices

MIREX 2008 Audio Artist Classification Run Times

Participant Runtime (hh:mm) / Fold
GP2 Feat Ex: 05:05 Train/Classify: 00:09
GT1 Feat Ex/Train/Classify: 00:16
GT2 Feat Ex/Train/Classify: 00:38
GT3 Feat Ex: 00:06 Train/Classify: 00:00 (11 sec)
LRPPI1 Feat Ex: 13:35 Train/Classify: 00:57
LRPPI2 Feat Ex: 13:35 Train/Classify: 1:15
LRPPI3 Feat Ex: 13:35 Train/Classify: 01:18
LRPPI4 Feat Ex: 13:35 Train/Classify: 01:53
ME1 Feat Ex: 1:27 Train/Classify: 00:01
ME2 Feat Ex: 1:27 Train/Classify: 00:01
ME3 Feat Ex: 1:27 Train/Classify: 00:01

download these results as csv

CSV Files Without Rounding

audioartist_results_fold.csv
audioartist_results_class.csv

Results By Algorithm

(.tar.gz)
GP2 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3

Friedman's Test for Significant Differences

Classes vs. Systems

The Friedman test was run in MATLAB against the average accuracy for each class.

Friedman's Anova Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 4775.9 10 477.59 452.84 0
Error 5981.6 1010 5.922
Total 10757.5 1121

download these results as csv

Tukey-Kramer HSD Multi-Comparison

The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
GP2 GT1 -1.2725 0.1912 1.6549 FALSE
GP2 GT2 -1.2431 0.2206 1.6843 FALSE
GP2 GT3 -0.3215 1.1422 2.6058 FALSE
GP2 LRPPI1 1.4481 2.9118 4.3754 TRUE
GP2 LRPPI2 1.1932 2.6569 4.1205 TRUE
GP2 LRPPI3 1.9824 3.4461 4.9098 TRUE
GP2 LRPPI4 1.9285 3.3922 4.8558 TRUE
GP2 ME1 1.9040 3.3676 4.8313 TRUE
GP2 ME2 2.7177 4.1814 5.6450 TRUE
GP2 ME3 5.8206 7.2843 8.7480 TRUE
GT1 GT2 -1.4343 0.0294 1.4931 FALSE
GT1 GT3 -0.5127 0.9510 2.4147 FALSE
GT1 LRPPI1 1.2569 2.7206 4.1843 TRUE
GT1 LRPPI2 1.0020 2.4657 3.9294 TRUE
GT1 LRPPI3 1.7912 3.2549 4.7186 TRUE
GT1 LRPPI4 1.7373 3.2010 4.6647 TRUE
GT1 ME1 1.7128 3.1765 4.6401 TRUE
GT1 ME2 2.5265 3.9902 5.4539 TRUE
GT1 ME3 5.6295 7.0931 8.5568 TRUE
GT2 GT3 -0.5421 0.9216 2.3852 FALSE
GT2 LRPPI1 1.2275 2.6912 4.1549 TRUE
GT2 LRPPI2 0.9726 2.4363 3.8999 TRUE
GT2 LRPPI3 1.7618 3.2255 4.6892 TRUE
GT2 LRPPI4 1.7079 3.1716 4.6352 TRUE
GT2 ME1 1.6834 3.1471 4.6107 TRUE
GT2 ME2 2.4971 3.9608 5.4245 TRUE
GT2 ME3 5.6001 7.0637 8.5274 TRUE
GT3 LRPPI1 0.3059 1.7696 3.2333 TRUE
GT3 LRPPI2 0.0510 1.5147 2.9784 TRUE
GT3 LRPPI3 0.8402 2.3039 3.7676 TRUE
GT3 LRPPI4 0.7863 2.2500 3.7137 TRUE
GT3 ME1 0.7618 2.2255 3.6892 TRUE
GT3 ME2 1.5755 3.0392 4.5029 TRUE
GT3 ME3 4.6785 6.1422 7.6058 TRUE
LRPPI1 LRPPI2 -1.7186 -0.2549 1.2088 FALSE
LRPPI1 LRPPI3 -0.9294 0.5343 1.9980 FALSE
LRPPI1 LRPPI4 -0.9833 0.4804 1.9441 FALSE
LRPPI1 ME1 -1.0078 0.4559 1.9196 FALSE
LRPPI1 ME2 -0.1941 1.2696 2.7333 FALSE
LRPPI1 ME3 2.9089 4.3725 5.8362 TRUE
LRPPI2 LRPPI3 -0.6745 0.7892 2.2529 FALSE
LRPPI2 LRPPI4 -0.7284 0.7353 2.1990 FALSE
LRPPI2 ME1 -0.7529 0.7108 2.1745 FALSE
LRPPI2 ME2 0.0608 1.5245 2.9882 TRUE
LRPPI2 ME3 3.1638 4.6275 6.0911 TRUE
LRPPI3 LRPPI4 -1.5176 -0.0539 1.4098 FALSE
LRPPI3 ME1 -1.5421 -0.0784 1.3852 FALSE
LRPPI3 ME2 -0.7284 0.7353 2.1990 FALSE
LRPPI3 ME3 2.3746 3.8382 5.3019 TRUE
LRPPI4 ME1 -1.4882 -0.0245 1.4392 FALSE
LRPPI4 ME2 -0.6745 0.7892 2.2529 FALSE
LRPPI4 ME3 2.4285 3.8922 5.3558 TRUE
ME1 ME2 -0.6499 0.8137 2.2774 FALSE
ME1 ME3 2.4530 3.9167 5.3803 TRUE
ME2 ME3 1.6393 3.1029 4.5666 TRUE

download these results as csv

2008 artist.perclassaccuracy.friedman.tukeykramerhsd.png

Folds vs. Systems

The Friedman test was run in MATLAB against the accuracy for each fold.

Friedman's Anova Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 319.333 10 31.9333 29.03 0.0012
Error 10.667 20 0.5333
Total 330 32

download these results as csv

Tukey-Kramer HSD Multi-Comparison

The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
GP2 GT1 -7.3828 1.3333 10.0495 FALSE
GP2 GT2 -7.0495 1.6667 10.3828 FALSE
GP2 GT3 -5.7162 3.0000 11.7162 FALSE
GP2 LRPPI1 -4.0495 4.6667 13.3828 FALSE
GP2 LRPPI2 -4.0495 4.6667 13.3828 FALSE
GP2 LRPPI3 -2.0495 6.6667 15.3828 FALSE
GP2 LRPPI4 -2.3828 6.3333 15.0495 FALSE
GP2 ME1 -1.0495 7.6667 16.3828 FALSE
GP2 ME2 0.2838 9.0000 17.7162 TRUE
GP2 ME3 1.2838 10.0000 18.7162 TRUE
GT1 GT2 -8.3828 0.3333 9.0495 FALSE
GT1 GT3 -7.0495 1.6667 10.3828 FALSE
GT1 LRPPI1 -5.3828 3.3333 12.0495 FALSE
GT1 LRPPI2 -5.3828 3.3333 12.0495 FALSE
GT1 LRPPI3 -3.3828 5.3333 14.0495 FALSE
GT1 LRPPI4 -3.7162 5.0000 13.7162 FALSE
GT1 ME1 -2.3828 6.3333 15.0495 FALSE
GT1 ME2 -1.0495 7.6667 16.3828 FALSE
GT1 ME3 -0.0495 8.6667 17.3828 FALSE
GT2 GT3 -7.3828 1.3333 10.0495 FALSE
GT2 LRPPI1 -5.7162 3.0000 11.7162 FALSE
GT2 LRPPI2 -5.7162 3.0000 11.7162 FALSE
GT2 LRPPI3 -3.7162 5.0000 13.7162 FALSE
GT2 LRPPI4 -4.0495 4.6667 13.3828 FALSE
GT2 ME1 -2.7162 6.0000 14.7162 FALSE
GT2 ME2 -1.3828 7.3333 16.0495 FALSE
GT2 ME3 -0.3828 8.3333 17.0495 FALSE
GT3 LRPPI1 -7.0495 1.6667 10.3828 FALSE
GT3 LRPPI2 -7.0495 1.6667 10.3828 FALSE
GT3 LRPPI3 -5.0495 3.6667 12.3828 FALSE
GT3 LRPPI4 -5.3828 3.3333 12.0495 FALSE
GT3 ME1 -4.0495 4.6667 13.3828 FALSE
GT3 ME2 -2.7162 6.0000 14.7162 FALSE
GT3 ME3 -1.7162 7.0000 15.7162 FALSE
LRPPI1 LRPPI2 -8.7162 0.0000 8.7162 FALSE
LRPPI1 LRPPI3 -6.7162 2.0000 10.7162 FALSE
LRPPI1 LRPPI4 -7.0495 1.6667 10.3828 FALSE
LRPPI1 ME1 -5.7162 3.0000 11.7162 FALSE
LRPPI1 ME2 -4.3828 4.3333 13.0495 FALSE
LRPPI1 ME3 -3.3828 5.3333 14.0495 FALSE
LRPPI2 LRPPI3 -6.7162 2.0000 10.7162 FALSE
LRPPI2 LRPPI4 -7.0495 1.6667 10.3828 FALSE
LRPPI2 ME1 -5.7162 3.0000 11.7162 FALSE
LRPPI2 ME2 -4.3828 4.3333 13.0495 FALSE
LRPPI2 ME3 -3.3828 5.3333 14.0495 FALSE
LRPPI3 LRPPI4 -9.0495 -0.3333 8.3828 FALSE
LRPPI3 ME1 -7.7162 1.0000 9.7162 FALSE
LRPPI3 ME2 -6.3828 2.3333 11.0495 FALSE
LRPPI3 ME3 -5.3828 3.3333 12.0495 FALSE
LRPPI4 ME1 -7.3828 1.3333 10.0495 FALSE
LRPPI4 ME2 -6.0495 2.6667 11.3828 FALSE
LRPPI4 ME3 -5.0495 3.6667 12.3828 FALSE
ME1 ME2 -7.3828 1.3333 10.0495 FALSE
ME1 ME3 -6.3828 2.3333 11.0495 FALSE
ME2 ME3 -7.7162 1.0000 9.7162 FALSE

download these results as csv

2008 artist.perfoldaccuracy.friedman.tukeykramerhsd.png

Retrieved from "https://www.music-ir.org/mirex/w/index.php?title=2008:Audio_Artist_Identification_Results&oldid=7376"