Difference between revisions of "2008:Audio Artist Identification Results"
(→Results By Algorithm) |
(→Results By Algorithm) |
||
(13 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
− | These are the results for the 2008 running of the Audio Artist Identification task. For background information about this task set please refer to the [[Audio Artist Identification]] page. | + | These are the results for the 2008 running of the Audio Artist Identification task. For background information about this task set please refer to the [[2008:Audio Artist Identification]] page. |
===General Legend=== | ===General Legend=== | ||
====Team ID==== | ====Team ID==== | ||
− | '''GP2''' = [https://www.music-ir.org/mirex/2008/ | + | '''GP2''' = [https://www.music-ir.org/mirex/abstracts/2008/Peeters_2008_ISMIR_MIREX.pdf G. Peeters]<br /> |
− | '''GT1 (mono)''' = [https://www.music-ir.org/mirex/2008/ | + | '''GT1 (mono)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br /> |
− | '''GT2 (stereo)''' = [https://www.music-ir.org/mirex/2008/ | + | '''GT2 (stereo)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br /> |
− | '''GT3 (multicore)''' = [https://www.music-ir.org/mirex/2008/ | + | '''GT3 (multicore)''' = [https://www.music-ir.org/mirex/abstracts/2008/mirex2007.pdf G. Tzanetakis]<br /> |
− | '''LRPPI1''' = [https://www.music-ir.org/mirex/2008/ | + | '''LRPPI1''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1]<br /> |
− | '''LRPPI2''' = [https://www.music-ir.org/mirex/2008/ | + | '''LRPPI2''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2]<br /> |
− | '''LRPPI3''' = [https://www.music-ir.org/mirex/2008/ | + | '''LRPPI3''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3]<br /> |
− | '''LRPPI4''' = [https://www.music-ir.org/mirex/2008/ | + | '''LRPPI4''' = [https://www.music-ir.org/mirex/abstracts/2008/abstract_mirex08_class.pdf T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4]<br /> |
− | '''ME1''' = [https://www.music-ir.org/mirex/2008/ | + | '''ME1''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf M. I. Mandel, D. P. W. Ellis 1]<br /> |
− | '''ME2''' = [https://www.music-ir.org/mirex/2008/ | + | '''ME2''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf M. I. Mandel, D. P. W. Ellis 2]<br /> |
− | '''ME3''' = [https://www.music-ir.org/mirex/2008/ | + | '''ME3''' = [https://www.music-ir.org/mirex/abstracts/2008/AA_AG_AT_MM_CC_mandel.pdf M. I. Mandel, D. P. W. Ellis 3]<br /> |
==Overall Summary Results== | ==Overall Summary Results== | ||
===MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds=== | ===MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds=== | ||
− | <csv>artist/audioartist.avg.results.csv</csv> | + | <csv>2008/artist/audioartist.avg.results.csv</csv> |
=====Accuracy Across Folds===== | =====Accuracy Across Folds===== | ||
− | <csv>artist/audioartist.results.fold.csv</csv> | + | <csv>2008/artist/audioartist.results.fold.csv</csv> |
=====Accuracy Across Categories===== | =====Accuracy Across Categories===== | ||
− | <csv>artist/audioartist.results.class.csv</csv> | + | <csv>2008/artist/audioartist.results.class.csv</csv> |
===MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices=== | ===MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices=== | ||
− | ===MIREX 2008 Audio Artist Classification Run Times=== | + | ====MIREX 2008 Audio Artist Classification Run Times==== |
− | <csv>artist.runtime.csv</csv> | + | <csv>2008/artist.runtime.csv</csv> |
====CSV Files Without Rounding==== | ====CSV Files Without Rounding==== | ||
− | [https://www.music-ir.org/mirex/2008 | + | [https://www.music-ir.org/mirex/results/2008/artist/audioartist_results_fold.csv audioartist_results_fold.csv]<br /> |
− | [https://www.music-ir.org/mirex/2008 | + | [https://www.music-ir.org/mirex/results/2008/artist/audioartist_results_class.csv audioartist_results_class.csv]<br /> |
====Results By Algorithm==== | ====Results By Algorithm==== | ||
(.tar.gz) <br /> | (.tar.gz) <br /> | ||
− | '''GP2''' = [https://www.music-ir.org/mirex/2008 | + | '''GP2''' = [https://www.music-ir.org/mirex/results/2008/artist/GP2.tar.gz G. Peeters]<br /> |
− | '''GT1''' = [https://www.music-ir.org/mirex/2008 | + | '''GT1''' = [https://www.music-ir.org/mirex/results/2008/artist/GT1.tar.gz G. Tzanetakis]<br /> |
− | '''GT2''' = [https://www.music-ir.org/mirex/2008 | + | '''GT2''' = [https://www.music-ir.org/mirex/results/2008/artist/GT2.tar.gz G. Tzanetakis]<br /> |
− | '''GT3''' = [https://www.music-ir.org/mirex/2008 | + | '''GT3''' = [https://www.music-ir.org/mirex/results/2008/artist/GT3.tar.gz G. Tzanetakis]<br /> |
− | '''LRPPI1''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI1''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI1.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1]<br /> |
− | '''LRPPI2''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI2''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI2.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2]<br /> |
− | '''LRPPI3''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI3''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI3.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3]<br /> |
− | '''LRPPI4''' = [https://www.music-ir.org/mirex/2008 | + | '''LRPPI4''' = [https://www.music-ir.org/mirex/results/2008/artist/LRPPI4.tar.gz T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4]<br /> |
− | '''ME1''' = [https://www.music-ir.org/mirex/2008 | + | '''ME1''' = [https://www.music-ir.org/mirex/results/2008/artist/ME1.tar.gz I. M. Mandel, D. P. W. Ellis 1]<br /> |
− | '''ME2''' = [https://www.music-ir.org/mirex/2008 | + | '''ME2''' = [https://www.music-ir.org/mirex/results/2008/artist/ME2.tar.gz I. M. Mandel, D. P. W. Ellis 2]<br /> |
− | '''ME3''' = [https://www.music-ir.org/mirex/2008 | + | '''ME3''' = [https://www.music-ir.org/mirex/results/2008/artist/ME3.tar.gz I. M. Mandel, D. P. W. Ellis 3]<br /> |
+ | |||
+ | ===Friedman's Test for Significant Differences=== | ||
+ | ====Classes vs. Systems==== | ||
+ | The Friedman test was run in MATLAB against the average accuracy for each class. | ||
+ | |||
+ | =====Friedman's Anova Table===== | ||
+ | |||
+ | <csv>2008/artist/perClassAccuracy.friedman.csv</csv> | ||
+ | |||
+ | =====Tukey-Kramer HSD Multi-Comparison===== | ||
+ | The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. | ||
+ | Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05); | ||
+ | |||
+ | <csv>2008/artist/perClassAccuracy.friedman.detail.csv</csv> | ||
+ | |||
+ | [[Image:2008_artist.perclassaccuracy.friedman.tukeykramerhsd.png]] | ||
+ | |||
+ | ====Folds vs. Systems==== | ||
+ | The Friedman test was run in MATLAB against the accuracy for each fold. | ||
+ | |||
+ | =====Friedman's Anova Table===== | ||
+ | |||
+ | <csv>2008/artist/perFoldAccuracy.friedman.csv</csv> | ||
+ | |||
+ | =====Tukey-Kramer HSD Multi-Comparison===== | ||
+ | The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. | ||
+ | Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05); | ||
+ | |||
+ | <csv>2008/artist/perFoldAccuracy.friedman.detail.csv</csv> | ||
+ | |||
+ | [[Image:2008_artist.perfoldaccuracy.friedman.tukeykramerhsd.png]] | ||
[[Category: Results]] | [[Category: Results]] |
Latest revision as of 17:01, 23 July 2010
Contents
Introduction
These are the results for the 2008 running of the Audio Artist Identification task. For background information about this task set please refer to the 2008:Audio Artist Identification page.
General Legend
Team ID
GP2 = G. Peeters
GT1 (mono) = G. Tzanetakis
GT2 (stereo) = G. Tzanetakis
GT3 (multicore) = G. Tzanetakis
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4
ME1 = M. I. Mandel, D. P. W. Ellis 1
ME2 = M. I. Mandel, D. P. W. Ellis 2
ME3 = M. I. Mandel, D. P. W. Ellis 3
Overall Summary Results
MIREX 2008 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds
Participant | Average Classifcation Accuracy |
---|---|
GP2 | 1.11% |
GT1 | 33.66% |
GT2 | 43.47% |
GT3 | 35.27% |
LRPPI1 | 35.42% |
LRPPI2 | 33.20% |
LRPPI3 | 29.87% |
LRPPI4 | 32.52% |
ME1 | 47.65% |
ME2 | 47.16% |
ME3 | 47.25% |
Accuracy Across Folds
Classification fold | GP2 | GT1 | GT2 | GT3 | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.005 | 0.342 | 0.443 | 0.344 | 0.367 | 0.348 | 0.306 | 0.344 | 0.467 | 0.460 | 0.462 |
1 | 0.018 | 0.329 | 0.447 | 0.341 | 0.333 | 0.313 | 0.288 | 0.308 | 0.481 | 0.477 | 0.475 |
2 | 0.011 | 0.339 | 0.413 | 0.373 | 0.363 | 0.335 | 0.303 | 0.324 | 0.482 | 0.478 | 0.481 |
Accuracy Across Categories
Class | GP2 | GT1 | GT2 | GT3 | LRPPI1 | LRPPI2 | LRPPI3 | LRPPI4 | ME1 | ME2 | ME3 |
---|---|---|---|---|---|---|---|---|---|---|---|
112 | 0.000 | 0.633 | 0.600 | 0.433 | 0.300 | 0.333 | 0.300 | 0.233 | 0.633 | 0.633 | 0.633 |
ABBA | 0.000 | 0.300 | 0.233 | 0.333 | 0.300 | 0.167 | 0.200 | 0.367 | 0.500 | 0.500 | 0.500 |
ACDC | 0.067 | 0.433 | 0.533 | 0.533 | 0.533 | 0.467 | 0.500 | 0.467 | 0.500 | 0.500 | 0.500 |
AEROSMITH | 0.100 | 0.195 | 0.161 | 0.195 | 0.067 | 0.100 | 0.100 | 0.200 | 0.127 | 0.127 | 0.127 |
ARTHURBIGBOYCRUDUP | 0.000 | 0.933 | 1.000 | 1.000 | 0.900 | 0.800 | 0.667 | 0.733 | 1.000 | 1.000 | 1.000 |
ARTIESHAW | 0.000 | 0.400 | 0.700 | 0.300 | 0.333 | 0.400 | 0.333 | 0.367 | 0.433 | 0.400 | 0.400 |
BENNYCARTER | 0.000 | 0.300 | 0.667 | 0.367 | 0.200 | 0.167 | 0.133 | 0.100 | 0.300 | 0.300 | 0.300 |
BENNYGOODMAN | 0.000 | 0.133 | 0.367 | 0.233 | 0.200 | 0.233 | 0.300 | 0.167 | 0.367 | 0.400 | 0.400 |
BENNYMOTEN | 0.000 | 0.467 | 0.700 | 0.333 | 0.433 | 0.267 | 0.167 | 0.333 | 0.433 | 0.433 | 0.467 |
BESSIESMITH | 0.033 | 1.000 | 1.000 | 0.900 | 1.000 | 0.933 | 0.967 | 0.967 | 1.000 | 1.000 | 1.000 |
BIGBILLBROONZY | 0.000 | 0.600 | 0.800 | 0.600 | 0.533 | 0.500 | 0.400 | 0.500 | 0.733 | 0.733 | 0.733 |
BILLCOLEMAN | 0.000 | 0.200 | 0.700 | 0.133 | 0.233 | 0.100 | 0.133 | 0.133 | 0.267 | 0.267 | 0.267 |
BILLIEHOLIDAY | 0.000 | 0.400 | 0.633 | 0.333 | 0.400 | 0.433 | 0.300 | 0.367 | 0.700 | 0.700 | 0.667 |
BIXBEIDERBECKE | 0.000 | 0.600 | 0.800 | 0.600 | 0.800 | 0.667 | 0.600 | 0.700 | 0.733 | 0.733 | 0.733 |
BLINK182 | 0.000 | 0.167 | 0.267 | 0.467 | 0.200 | 0.367 | 0.333 | 0.333 | 0.367 | 0.367 | 0.367 |
BOBMARLEYTHEWAILERS | 0.000 | 0.567 | 0.600 | 0.633 | 0.833 | 0.700 | 0.733 | 0.800 | 0.700 | 0.700 | 0.700 |
BONJOVI | 0.000 | 0.600 | 0.533 | 0.567 | 0.600 | 0.633 | 0.633 | 0.633 | 0.600 | 0.533 | 0.533 |
BROWNIEMCGHEE | 0.000 | 0.800 | 0.867 | 0.833 | 0.800 | 0.933 | 0.700 | 0.833 | 0.933 | 0.933 | 0.933 |
BRUCESPRINGSTEEN | 0.033 | 0.067 | 0.133 | 0.067 | 0.367 | 0.267 | 0.267 | 0.300 | 0.133 | 0.133 | 0.133 |
BRYANADAMS | 0.000 | 0.200 | 0.400 | 0.200 | 0.167 | 0.133 | 0.133 | 0.100 | 0.367 | 0.367 | 0.367 |
BUNNYBERIGAN | 0.000 | 0.333 | 0.500 | 0.433 | 0.333 | 0.433 | 0.433 | 0.300 | 0.467 | 0.433 | 0.433 |
CABCALLOWAY | 0.000 | 0.467 | 0.533 | 0.500 | 0.133 | 0.133 | 0.133 | 0.200 | 0.600 | 0.600 | 0.633 |
CARLPERKINS | 0.000 | 0.567 | 0.567 | 0.567 | 0.400 | 0.267 | 0.167 | 0.300 | 0.700 | 0.700 | 0.700 |
CHARLIEBARNET | 0.000 | 0.200 | 0.400 | 0.300 | 0.567 | 0.300 | 0.333 | 0.400 | 0.267 | 0.267 | 0.267 |
CHICKWEBB | 0.000 | 0.267 | 0.767 | 0.300 | 0.300 | 0.267 | 0.100 | 0.167 | 0.467 | 0.433 | 0.433 |
COUNTBASIE | 0.000 | 0.167 | 0.300 | 0.200 | 0.100 | 0.133 | 0.133 | 0.167 | 0.333 | 0.333 | 0.333 |
CREEDENCECLEARWATERREVIVAL | 0.000 | 0.500 | 0.633 | 0.500 | 0.533 | 0.367 | 0.500 | 0.533 | 0.400 | 0.367 | 0.367 |
DAVEMATTHEWSBAND | 0.000 | 0.100 | 0.133 | 0.167 | 0.067 | 0.133 | 0.067 | 0.033 | 0.367 | 0.333 | 0.333 |
DEANMARTIN | 0.000 | 0.467 | 0.667 | 0.567 | 0.467 | 0.367 | 0.367 | 0.233 | 0.767 | 0.767 | 0.767 |
DEFLEPPARD | 0.000 | 0.467 | 0.433 | 0.667 | 0.467 | 0.567 | 0.467 | 0.533 | 0.467 | 0.467 | 0.467 |
DEPECHEMODE | 0.000 | 0.233 | 0.333 | 0.200 | 0.333 | 0.400 | 0.167 | 0.267 | 0.600 | 0.600 | 0.600 |
DMX | 0.067 | 0.700 | 0.800 | 0.700 | 0.700 | 0.733 | 0.733 | 0.767 | 0.900 | 0.900 | 0.900 |
DUKEELLINGTON | 0.000 | 0.233 | 0.400 | 0.300 | 0.333 | 0.333 | 0.300 | 0.300 | 0.300 | 0.300 | 0.300 |
EDDIECONDON | 0.000 | 0.733 | 0.767 | 0.733 | 0.467 | 0.467 | 0.533 | 0.567 | 0.467 | 0.467 | 0.467 |
ENYA | 0.000 | 0.833 | 0.833 | 0.733 | 0.633 | 0.733 | 0.667 | 0.633 | 0.667 | 0.667 | 0.667 |
ERICCLAPTON | 0.000 | 0.146 | 0.305 | 0.144 | 0.089 | 0.057 | 0.113 | 0.079 | 0.144 | 0.144 | 0.144 |
ERSKINEHAWKINS | 0.000 | 0.067 | 0.100 | 0.067 | 0.333 | 0.333 | 0.233 | 0.367 | 0.333 | 0.333 | 0.333 |
EURYTHMICS | 0.000 | 0.200 | 0.167 | 0.200 | 0.500 | 0.300 | 0.400 | 0.433 | 0.367 | 0.367 | 0.367 |
FATSWALLER | 0.000 | 0.300 | 0.667 | 0.300 | 0.533 | 0.433 | 0.200 | 0.433 | 0.700 | 0.700 | 0.733 |
FLEETWOODMAC | 0.000 | 0.000 | 0.000 | 0.000 | 0.033 | 0.000 | 0.000 | 0.033 | 0.200 | 0.200 | 0.200 |
FLETCHERHENDERSON | 0.000 | 0.100 | 0.433 | 0.133 | 0.167 | 0.200 | 0.133 | 0.100 | 0.367 | 0.300 | 0.300 |
FOOFIGHTERS | 0.300 | 0.400 | 0.533 | 0.667 | 0.033 | 0.033 | 0.133 | 0.033 | 0.400 | 0.400 | 0.433 |
FRANKSINATRA | 0.000 | 0.433 | 0.467 | 0.400 | 0.400 | 0.433 | 0.467 | 0.433 | 0.700 | 0.667 | 0.667 |
FREDDYFENDER | 0.000 | 0.433 | 0.400 | 0.433 | 0.500 | 0.400 | 0.200 | 0.333 | 0.600 | 0.533 | 0.533 |
GARTHBROOKS | 0.000 | 0.733 | 0.767 | 0.633 | 0.733 | 0.633 | 0.533 | 0.667 | 0.733 | 0.733 | 0.733 |
GENEKRUPA | 0.000 | 0.333 | 0.433 | 0.333 | 0.200 | 0.200 | 0.167 | 0.233 | 0.433 | 0.433 | 0.467 |
GENESIS | 0.000 | 0.131 | 0.083 | 0.125 | 0.239 | 0.175 | 0.214 | 0.159 | 0.042 | 0.000 | 0.000 |
GEORGEJONES | 0.000 | 0.533 | 0.767 | 0.567 | 0.567 | 0.600 | 0.600 | 0.533 | 0.700 | 0.700 | 0.700 |
GEORGEWETTLING | 0.000 | 0.333 | 0.533 | 0.400 | 0.267 | 0.100 | 0.133 | 0.167 | 0.233 | 0.233 | 0.200 |
GLENNMILLER | 0.000 | 0.100 | 0.300 | 0.200 | 0.267 | 0.333 | 0.133 | 0.367 | 0.300 | 0.300 | 0.367 |
GREENDAY | 0.000 | 0.067 | 0.000 | 0.100 | 0.167 | 0.500 | 0.433 | 0.300 | 0.167 | 0.167 | 0.167 |
HARRYJAMES | 0.000 | 0.267 | 0.400 | 0.267 | 0.233 | 0.167 | 0.233 | 0.167 | 0.600 | 0.600 | 0.600 |
INXS | 0.000 | 0.033 | 0.067 | 0.133 | 0.133 | 0.100 | 0.067 | 0.100 | 0.367 | 0.367 | 0.367 |
JACKTEAGARDEN | 0.000 | 0.133 | 0.233 | 0.100 | 0.067 | 0.167 | 0.133 | 0.100 | 0.400 | 0.367 | 0.367 |
JELLYROLLMORTON | 0.000 | 0.067 | 0.233 | 0.100 | 0.333 | 0.300 | 0.233 | 0.333 | 0.467 | 0.467 | 0.467 |
JIMMIELUNCEFORD | 0.000 | 0.167 | 0.267 | 0.167 | 0.133 | 0.300 | 0.167 | 0.233 | 0.400 | 0.400 | 0.433 |
JIMMIENOONE | 0.000 | 0.400 | 0.733 | 0.333 | 0.367 | 0.200 | 0.167 | 0.200 | 0.467 | 0.500 | 0.500 |
JOHNNYDODDS | 0.000 | 0.100 | 0.167 | 0.033 | 0.200 | 0.133 | 0.100 | 0.133 | 0.233 | 0.233 | 0.233 |
KOKOMOARNOLD | 0.000 | 0.800 | 0.867 | 0.767 | 0.733 | 0.867 | 0.767 | 0.800 | 0.867 | 0.867 | 0.867 |
LEADBELLY | 0.000 | 0.533 | 0.500 | 0.533 | 0.367 | 0.333 | 0.367 | 0.333 | 0.667 | 0.667 | 0.667 |
LIGHTNINHOPKINS | 0.000 | 0.633 | 0.733 | 0.667 | 0.600 | 0.733 | 0.733 | 0.733 | 0.767 | 0.767 | 0.767 |
LIONELHAMPTON | 0.000 | 0.233 | 0.433 | 0.300 | 0.167 | 0.333 | 0.167 | 0.233 | 0.467 | 0.400 | 0.467 |
LLCOOLJ | 0.000 | 0.367 | 0.467 | 0.400 | 0.167 | 0.200 | 0.167 | 0.167 | 0.833 | 0.833 | 0.800 |
LOUISARMSTRONG | 0.000 | 0.000 | 0.367 | 0.067 | 0.233 | 0.167 | 0.167 | 0.167 | 0.167 | 0.167 | 0.167 |
MADONNA | 0.000 | 0.133 | 0.067 | 0.167 | 0.067 | 0.033 | 0.000 | 0.033 | 0.367 | 0.367 | 0.367 |
MARIAHCAREY | 0.000 | 0.433 | 0.600 | 0.600 | 0.400 | 0.400 | 0.433 | 0.467 | 0.800 | 0.800 | 0.833 |
MARILYNMANSON | 0.133 | 0.000 | 0.067 | 0.033 | 0.067 | 0.100 | 0.067 | 0.067 | 0.367 | 0.367 | 0.333 |
METALLICA | 0.000 | 0.417 | 0.507 | 0.424 | 0.507 | 0.424 | 0.444 | 0.382 | 0.583 | 0.583 | 0.583 |
MICHAELJACKSON | 0.000 | 0.300 | 0.267 | 0.367 | 0.300 | 0.267 | 0.167 | 0.400 | 0.267 | 0.267 | 0.267 |
MICKEYGILLEY | 0.000 | 0.567 | 0.600 | 0.500 | 0.667 | 0.633 | 0.400 | 0.467 | 0.667 | 0.667 | 0.633 |
MUDDYWATERS | 0.000 | 0.500 | 0.500 | 0.533 | 0.333 | 0.333 | 0.233 | 0.433 | 0.700 | 0.700 | 0.700 |
MUGGSYSPANIER | 0.000 | 0.233 | 0.267 | 0.367 | 0.367 | 0.300 | 0.233 | 0.300 | 0.533 | 0.500 | 0.500 |
NATKINGCOLE | 0.000 | 0.333 | 0.633 | 0.400 | 0.433 | 0.400 | 0.367 | 0.400 | 0.700 | 0.633 | 0.633 |
NINEINCHNAILS | 0.300 | 0.033 | 0.133 | 0.033 | 0.067 | 0.000 | 0.100 | 0.000 | 0.133 | 0.167 | 0.100 |
NIRVANA | 0.000 | 0.167 | 0.067 | 0.067 | 0.233 | 0.300 | 0.233 | 0.233 | 0.333 | 0.367 | 0.367 |
ORIGINALDIXIELANDJAZZBAND | 0.000 | 1.000 | 1.000 | 1.000 | 0.900 | 0.733 | 0.767 | 0.833 | 0.900 | 0.900 | 0.933 |
PHILCOLLINS | 0.000 | 0.067 | 0.100 | 0.067 | 0.133 | 0.233 | 0.300 | 0.167 | 0.433 | 0.433 | 0.433 |
PINKFLOYD | 0.000 | 0.098 | 0.183 | 0.278 | 0.299 | 0.444 | 0.156 | 0.217 | 0.169 | 0.169 | 0.169 |
QUEEN | 0.000 | 0.067 | 0.167 | 0.100 | 0.267 | 0.167 | 0.167 | 0.200 | 0.367 | 0.367 | 0.367 |
RADIOHEAD | 0.067 | 0.067 | 0.067 | 0.033 | 0.233 | 0.167 | 0.133 | 0.133 | 0.033 | 0.033 | 0.033 |
RAYCHARLES | 0.000 | 0.302 | 0.446 | 0.346 | 0.164 | 0.201 | 0.204 | 0.134 | 0.594 | 0.594 | 0.594 |
REDNICHOLS | 0.000 | 0.367 | 0.367 | 0.500 | 0.367 | 0.300 | 0.367 | 0.400 | 0.500 | 0.500 | 0.500 |
RKELLY | 0.000 | 0.467 | 0.433 | 0.300 | 0.533 | 0.400 | 0.433 | 0.500 | 0.533 | 0.533 | 0.533 |
RODSTEWART | 0.000 | 0.067 | 0.033 | 0.033 | 0.167 | 0.133 | 0.233 | 0.100 | 0.167 | 0.167 | 0.167 |
ROXETTE | 0.000 | 0.167 | 0.367 | 0.200 | 0.167 | 0.133 | 0.133 | 0.133 | 0.433 | 0.433 | 0.433 |
SADE | 0.000 | 0.133 | 0.133 | 0.167 | 0.133 | 0.133 | 0.133 | 0.133 | 0.367 | 0.367 | 0.367 |
SANTANA | 0.000 | 0.200 | 0.133 | 0.067 | 0.100 | 0.067 | 0.067 | 0.100 | 0.133 | 0.133 | 0.133 |
SIDNEYBECHET | 0.000 | 0.300 | 0.467 | 0.233 | 0.300 | 0.267 | 0.167 | 0.100 | 0.300 | 0.300 | 0.300 |
SONNYBOYWILLIAMSONI | 0.000 | 0.700 | 0.800 | 0.733 | 0.633 | 0.633 | 0.633 | 0.567 | 0.867 | 0.867 | 0.867 |
STANKENTON | 0.000 | 0.500 | 0.567 | 0.533 | 0.467 | 0.433 | 0.300 | 0.400 | 0.500 | 0.467 | 0.467 |
SUBLIME | 0.000 | 0.100 | 0.233 | 0.233 | 0.233 | 0.033 | 0.167 | 0.200 | 0.400 | 0.400 | 0.400 |
THEBEATLES | 0.000 | 0.389 | 0.167 | 0.250 | 0.167 | 0.222 | 0.306 | 0.250 | 0.278 | 0.278 | 0.278 |
THEDOORS | 0.000 | 0.033 | 0.067 | 0.067 | 0.033 | 0.000 | 0.000 | 0.067 | 0.233 | 0.200 | 0.200 |
THEROLLINGSTONES | 0.028 | 0.361 | 0.222 | 0.278 | 0.306 | 0.111 | 0.083 | 0.167 | 0.139 | 0.139 | 0.139 |
TIMMCGRAW | 0.000 | 0.433 | 0.500 | 0.300 | 0.500 | 0.467 | 0.400 | 0.500 | 0.833 | 0.867 | 0.833 |
TINATURNER | 0.000 | 0.033 | 0.033 | 0.095 | 0.151 | 0.192 | 0.167 | 0.159 | 0.323 | 0.323 | 0.323 |
TOMJONES | 0.000 | 0.767 | 0.967 | 0.833 | 0.767 | 0.733 | 0.633 | 0.667 | 0.967 | 0.967 | 0.967 |
TOMMYDORSEY | 0.000 | 0.433 | 0.433 | 0.467 | 0.400 | 0.300 | 0.233 | 0.233 | 0.433 | 0.433 | 0.433 |
U2 | 0.000 | 0.233 | 0.367 | 0.233 | 0.300 | 0.267 | 0.133 | 0.200 | 0.333 | 0.333 | 0.333 |
VANHALEN | 0.000 | 0.174 | 0.278 | 0.174 | 0.375 | 0.207 | 0.269 | 0.278 | 0.238 | 0.238 | 0.204 |
WILLIENELSON | 0.000 | 0.667 | 0.800 | 0.733 | 0.733 | 0.733 | 0.800 | 0.867 | 0.900 | 0.933 | 0.933 |
WOODYHERMAN | 0.000 | 0.033 | 0.300 | 0.083 | 0.167 | 0.100 | 0.033 | 0.033 | 0.267 | 0.267 | 0.267 |
MIREX 2008 Audio Artist Classification Evaluation Logs and Confusion Matrices
MIREX 2008 Audio Artist Classification Run Times
Participant | Runtime (hh:mm) / Fold |
---|---|
GP2 | Feat Ex: 05:05 Train/Classify: 00:09 |
GT1 | Feat Ex/Train/Classify: 00:16 |
GT2 | Feat Ex/Train/Classify: 00:38 |
GT3 | Feat Ex: 00:06 Train/Classify: 00:00 (11 sec) |
LRPPI1 | Feat Ex: 13:35 Train/Classify: 00:57 |
LRPPI2 | Feat Ex: 13:35 Train/Classify: 1:15 |
LRPPI3 | Feat Ex: 13:35 Train/Classify: 01:18 |
LRPPI4 | Feat Ex: 13:35 Train/Classify: 01:53 |
ME1 | Feat Ex: 1:27 Train/Classify: 00:01 |
ME2 | Feat Ex: 1:27 Train/Classify: 00:01 |
ME3 | Feat Ex: 1:27 Train/Classify: 00:01 |
CSV Files Without Rounding
audioartist_results_fold.csv
audioartist_results_class.csv
Results By Algorithm
(.tar.gz)
GP2 = G. Peeters
GT1 = G. Tzanetakis
GT2 = G. Tzanetakis
GT3 = G. Tzanetakis
LRPPI1 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 1
LRPPI2 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 2
LRPPI3 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 3
LRPPI4 = T. Lidy, A. Rauber, A. Pertusa, P. Peonce de León, J. M. Iñesta 4
ME1 = I. M. Mandel, D. P. W. Ellis 1
ME2 = I. M. Mandel, D. P. W. Ellis 2
ME3 = I. M. Mandel, D. P. W. Ellis 3
Friedman's Test for Significant Differences
Classes vs. Systems
The Friedman test was run in MATLAB against the average accuracy for each class.
Friedman's Anova Table
Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
---|---|---|---|---|---|
Columns | 4775.9 | 10 | 477.59 | 452.84 | 0 |
Error | 5981.6 | 1010 | 5.922 | ||
Total | 10757.5 | 1121 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
GP2 | GT1 | -1.2725 | 0.1912 | 1.6549 | FALSE |
GP2 | GT2 | -1.2431 | 0.2206 | 1.6843 | FALSE |
GP2 | GT3 | -0.3215 | 1.1422 | 2.6058 | FALSE |
GP2 | LRPPI1 | 1.4481 | 2.9118 | 4.3754 | TRUE |
GP2 | LRPPI2 | 1.1932 | 2.6569 | 4.1205 | TRUE |
GP2 | LRPPI3 | 1.9824 | 3.4461 | 4.9098 | TRUE |
GP2 | LRPPI4 | 1.9285 | 3.3922 | 4.8558 | TRUE |
GP2 | ME1 | 1.9040 | 3.3676 | 4.8313 | TRUE |
GP2 | ME2 | 2.7177 | 4.1814 | 5.6450 | TRUE |
GP2 | ME3 | 5.8206 | 7.2843 | 8.7480 | TRUE |
GT1 | GT2 | -1.4343 | 0.0294 | 1.4931 | FALSE |
GT1 | GT3 | -0.5127 | 0.9510 | 2.4147 | FALSE |
GT1 | LRPPI1 | 1.2569 | 2.7206 | 4.1843 | TRUE |
GT1 | LRPPI2 | 1.0020 | 2.4657 | 3.9294 | TRUE |
GT1 | LRPPI3 | 1.7912 | 3.2549 | 4.7186 | TRUE |
GT1 | LRPPI4 | 1.7373 | 3.2010 | 4.6647 | TRUE |
GT1 | ME1 | 1.7128 | 3.1765 | 4.6401 | TRUE |
GT1 | ME2 | 2.5265 | 3.9902 | 5.4539 | TRUE |
GT1 | ME3 | 5.6295 | 7.0931 | 8.5568 | TRUE |
GT2 | GT3 | -0.5421 | 0.9216 | 2.3852 | FALSE |
GT2 | LRPPI1 | 1.2275 | 2.6912 | 4.1549 | TRUE |
GT2 | LRPPI2 | 0.9726 | 2.4363 | 3.8999 | TRUE |
GT2 | LRPPI3 | 1.7618 | 3.2255 | 4.6892 | TRUE |
GT2 | LRPPI4 | 1.7079 | 3.1716 | 4.6352 | TRUE |
GT2 | ME1 | 1.6834 | 3.1471 | 4.6107 | TRUE |
GT2 | ME2 | 2.4971 | 3.9608 | 5.4245 | TRUE |
GT2 | ME3 | 5.6001 | 7.0637 | 8.5274 | TRUE |
GT3 | LRPPI1 | 0.3059 | 1.7696 | 3.2333 | TRUE |
GT3 | LRPPI2 | 0.0510 | 1.5147 | 2.9784 | TRUE |
GT3 | LRPPI3 | 0.8402 | 2.3039 | 3.7676 | TRUE |
GT3 | LRPPI4 | 0.7863 | 2.2500 | 3.7137 | TRUE |
GT3 | ME1 | 0.7618 | 2.2255 | 3.6892 | TRUE |
GT3 | ME2 | 1.5755 | 3.0392 | 4.5029 | TRUE |
GT3 | ME3 | 4.6785 | 6.1422 | 7.6058 | TRUE |
LRPPI1 | LRPPI2 | -1.7186 | -0.2549 | 1.2088 | FALSE |
LRPPI1 | LRPPI3 | -0.9294 | 0.5343 | 1.9980 | FALSE |
LRPPI1 | LRPPI4 | -0.9833 | 0.4804 | 1.9441 | FALSE |
LRPPI1 | ME1 | -1.0078 | 0.4559 | 1.9196 | FALSE |
LRPPI1 | ME2 | -0.1941 | 1.2696 | 2.7333 | FALSE |
LRPPI1 | ME3 | 2.9089 | 4.3725 | 5.8362 | TRUE |
LRPPI2 | LRPPI3 | -0.6745 | 0.7892 | 2.2529 | FALSE |
LRPPI2 | LRPPI4 | -0.7284 | 0.7353 | 2.1990 | FALSE |
LRPPI2 | ME1 | -0.7529 | 0.7108 | 2.1745 | FALSE |
LRPPI2 | ME2 | 0.0608 | 1.5245 | 2.9882 | TRUE |
LRPPI2 | ME3 | 3.1638 | 4.6275 | 6.0911 | TRUE |
LRPPI3 | LRPPI4 | -1.5176 | -0.0539 | 1.4098 | FALSE |
LRPPI3 | ME1 | -1.5421 | -0.0784 | 1.3852 | FALSE |
LRPPI3 | ME2 | -0.7284 | 0.7353 | 2.1990 | FALSE |
LRPPI3 | ME3 | 2.3746 | 3.8382 | 5.3019 | TRUE |
LRPPI4 | ME1 | -1.4882 | -0.0245 | 1.4392 | FALSE |
LRPPI4 | ME2 | -0.6745 | 0.7892 | 2.2529 | FALSE |
LRPPI4 | ME3 | 2.4285 | 3.8922 | 5.3558 | TRUE |
ME1 | ME2 | -0.6499 | 0.8137 | 2.2774 | FALSE |
ME1 | ME3 | 2.4530 | 3.9167 | 5.3803 | TRUE |
ME2 | ME3 | 1.6393 | 3.1029 | 4.5666 | TRUE |
Folds vs. Systems
The Friedman test was run in MATLAB against the accuracy for each fold.
Friedman's Anova Table
Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
---|---|---|---|---|---|
Columns | 319.333 | 10 | 31.9333 | 29.03 | 0.0012 |
Error | 10.667 | 20 | 0.5333 | ||
Total | 330 | 32 |
Tukey-Kramer HSD Multi-Comparison
The Tukey-Kramer HSD multi-comparison data below was generated using the following MATLAB instruction. Command: [c, m, h, gnames] = multicompare(stats, 'ctype', 'tukey-kramer', 'estimate', 'friedman', 'alpha', 0.05);
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
GP2 | GT1 | -7.3828 | 1.3333 | 10.0495 | FALSE |
GP2 | GT2 | -7.0495 | 1.6667 | 10.3828 | FALSE |
GP2 | GT3 | -5.7162 | 3.0000 | 11.7162 | FALSE |
GP2 | LRPPI1 | -4.0495 | 4.6667 | 13.3828 | FALSE |
GP2 | LRPPI2 | -4.0495 | 4.6667 | 13.3828 | FALSE |
GP2 | LRPPI3 | -2.0495 | 6.6667 | 15.3828 | FALSE |
GP2 | LRPPI4 | -2.3828 | 6.3333 | 15.0495 | FALSE |
GP2 | ME1 | -1.0495 | 7.6667 | 16.3828 | FALSE |
GP2 | ME2 | 0.2838 | 9.0000 | 17.7162 | TRUE |
GP2 | ME3 | 1.2838 | 10.0000 | 18.7162 | TRUE |
GT1 | GT2 | -8.3828 | 0.3333 | 9.0495 | FALSE |
GT1 | GT3 | -7.0495 | 1.6667 | 10.3828 | FALSE |
GT1 | LRPPI1 | -5.3828 | 3.3333 | 12.0495 | FALSE |
GT1 | LRPPI2 | -5.3828 | 3.3333 | 12.0495 | FALSE |
GT1 | LRPPI3 | -3.3828 | 5.3333 | 14.0495 | FALSE |
GT1 | LRPPI4 | -3.7162 | 5.0000 | 13.7162 | FALSE |
GT1 | ME1 | -2.3828 | 6.3333 | 15.0495 | FALSE |
GT1 | ME2 | -1.0495 | 7.6667 | 16.3828 | FALSE |
GT1 | ME3 | -0.0495 | 8.6667 | 17.3828 | FALSE |
GT2 | GT3 | -7.3828 | 1.3333 | 10.0495 | FALSE |
GT2 | LRPPI1 | -5.7162 | 3.0000 | 11.7162 | FALSE |
GT2 | LRPPI2 | -5.7162 | 3.0000 | 11.7162 | FALSE |
GT2 | LRPPI3 | -3.7162 | 5.0000 | 13.7162 | FALSE |
GT2 | LRPPI4 | -4.0495 | 4.6667 | 13.3828 | FALSE |
GT2 | ME1 | -2.7162 | 6.0000 | 14.7162 | FALSE |
GT2 | ME2 | -1.3828 | 7.3333 | 16.0495 | FALSE |
GT2 | ME3 | -0.3828 | 8.3333 | 17.0495 | FALSE |
GT3 | LRPPI1 | -7.0495 | 1.6667 | 10.3828 | FALSE |
GT3 | LRPPI2 | -7.0495 | 1.6667 | 10.3828 | FALSE |
GT3 | LRPPI3 | -5.0495 | 3.6667 | 12.3828 | FALSE |
GT3 | LRPPI4 | -5.3828 | 3.3333 | 12.0495 | FALSE |
GT3 | ME1 | -4.0495 | 4.6667 | 13.3828 | FALSE |
GT3 | ME2 | -2.7162 | 6.0000 | 14.7162 | FALSE |
GT3 | ME3 | -1.7162 | 7.0000 | 15.7162 | FALSE |
LRPPI1 | LRPPI2 | -8.7162 | 0.0000 | 8.7162 | FALSE |
LRPPI1 | LRPPI3 | -6.7162 | 2.0000 | 10.7162 | FALSE |
LRPPI1 | LRPPI4 | -7.0495 | 1.6667 | 10.3828 | FALSE |
LRPPI1 | ME1 | -5.7162 | 3.0000 | 11.7162 | FALSE |
LRPPI1 | ME2 | -4.3828 | 4.3333 | 13.0495 | FALSE |
LRPPI1 | ME3 | -3.3828 | 5.3333 | 14.0495 | FALSE |
LRPPI2 | LRPPI3 | -6.7162 | 2.0000 | 10.7162 | FALSE |
LRPPI2 | LRPPI4 | -7.0495 | 1.6667 | 10.3828 | FALSE |
LRPPI2 | ME1 | -5.7162 | 3.0000 | 11.7162 | FALSE |
LRPPI2 | ME2 | -4.3828 | 4.3333 | 13.0495 | FALSE |
LRPPI2 | ME3 | -3.3828 | 5.3333 | 14.0495 | FALSE |
LRPPI3 | LRPPI4 | -9.0495 | -0.3333 | 8.3828 | FALSE |
LRPPI3 | ME1 | -7.7162 | 1.0000 | 9.7162 | FALSE |
LRPPI3 | ME2 | -6.3828 | 2.3333 | 11.0495 | FALSE |
LRPPI3 | ME3 | -5.3828 | 3.3333 | 12.0495 | FALSE |
LRPPI4 | ME1 | -7.3828 | 1.3333 | 10.0495 | FALSE |
LRPPI4 | ME2 | -6.0495 | 2.6667 | 11.3828 | FALSE |
LRPPI4 | ME3 | -5.0495 | 3.6667 | 12.3828 | FALSE |
ME1 | ME2 | -7.3828 | 1.3333 | 10.0495 | FALSE |
ME1 | ME3 | -6.3828 | 2.3333 | 11.0495 | FALSE |
ME2 | ME3 | -7.7162 | 1.0000 | 9.7162 | FALSE |