University of Illinois · Graduate School of Library and Information Science · ISRL

2005 MIREX Contest Results - Audio Artist Identification (Contest wiki)


NOTE: As of October 6, 2005, there are still 1 set of evaluations to be completed.

Goal: To identify artist from music audio (in PCM format).

Dataset: Two sets of data were used: Magnatune and USPOP. The audio sampling rates used were either 44.1 KHz or 22.05 KHz (mono). More data information is in the following table.

Dataset  Size (@ 44.1 KHz)
Number of Training Files Number of Testing Files
Magnatune  35.2 GB 1158 642
USPOP 37.3 GB 1158
653


OVERALL
Rank Participant Mean of Magnatune Raw Classification Accuracy
and USPOP Raw Classification Accuracy
1 Mandel & Ellis 72.45%
2 Bergstra, Casagrande, & Eck (1) 68.57%
3
Bergstra, Casagrande, & Eck (2) 66.71%
4
Pampalk, E. 61.28%
5 West & Lamere 47.24%
6
Tzanetakis, G. 42.05%
7
Logan, B 25.95%


Magnatune Dataset
Rank Participant Raw Classification Accuracy Normalized Raw lassification Accuracy Runtime (s) Machine Confusion Matrix Files
1
Bergstra, Casagrande, & Eck (1) 77.26%
79.64%
24 hours
B0
BCE_1_MTeval.txt
2
Mandel & Ellis 76.60% 76.62% 11073 R ME_MTeval.txt
3
Bergstra, Casagrande, & Eck (2) 74.45%
74.51%


BCE_2_MTeval.txt
4
Pampalk, E. 66.36%
66.48%
4272
B1
P_MTeval.txt
5
Tzanetakis, G. 55.45%
55.59%
2632
B0
T_MTeval.txt
6
West & Lamere
53.43%
53.48%
27480
B3
WL_MTeval.txt
7
Logan, B
37.07%
37.10%
N/A
B3
L_MTeval.txt
8
Lidy & Rauber (SSD+RH) TO *




8
Lidy & Rauber (RP+SSD) TO *




8
Lidy & Rauber (RP+SSD+RH) TO *




 

USPOP Dataset
Rank Participant Raw Classification Accuracy Normalized Raw Classification Accuracy Runtime (s) Machine Confusion Matrix Files
1
Mandel & Ellis
68.30%
67.96%
10240
R
ME_USeval.txt
2
Bergstra, Casagrande, & Eck (1)
59.88%
60.90%
24 Hours
B0
BCE_1_USeval.txt
3
Bergstra, Casagrande, & Eck (2) 58.96%
58.96%


BCE_2_USeval.txt
4
Pampalk, E.
56.20%
56.03%
4321
B1
P_USeval.txt
5
West & Lamere
41.04%
41.00%
26871
B3
WL_USeval.txt
6
Tzanetakis, G. 28.64% 28.48% 2443 B0 T_USeval.txt
7
Logan, B.
14.83%
14.76%
N/A
B3
L_USeval.txt
8
Lidy & Rauber (SSD+RH) TO *




8
Lidy & Rauber (RP+SSD) TO *




8
Lidy & Rauber (RP+SSD+RH) TO *





Note:
DNC: did not complete ( error in execution).
          TO: timed out (did not complete within 24 hours).


Maintained by :J Stephen Downie
Comments to : jdownie at uiuc dot edu
Last modified: 25 October 2005