Audio Tag Classification Results

From MIREX2008

Jump to: navigation, search

Contents

Introduction

These are the results for the 2008 running of the Audio Tag Classification task. For background information about this task set please refer to the Audio Tag Classification page.

General Legend

Team ID

LB = L. Barrington, D. Turnbull, G. Lanckriet
BBE 1 = T. Bertin-Mahieux, Y. Bengio, D. Eck (KNN)
BBE 2 = T. Bertin-Mahieux, Y. Bengio, D. Eck (NNet)
BBE 3 = T. Bertin-Mahieux, D. Eck, P. Lamere, Y. Bengio (Thierry/Lamere Boosting)
TB = T. Bertin-Mahieux (dumb/smurf)
ME1 = M. I. Mandel, D. P. W. Ellis 1
ME2 = M. I. Mandel, D. P. W. Ellis 2
ME3 = M. I. Mandel, D. P. W. Ellis 3
GP1 = G. Peeters 1
GP2 = G. Peeters 2
TTKV = K. Trohidis, G. Tsoumakas, G. Kalliris, I. Vlahavas

Overall Summary Results

Measure BBE1 BBE2 BBE3 ME1 ME2 ME3 GP1 GP2 LB TB TTKV
Average Tag Positive Example Accuracy 0.05 1.00 0.85 0.67 0.68 0.66 0.04 0.03 0.28 0.91 0.03
Average Tag Negative Example Accuracy 0.99 0.00 0.37 0.71 0.73 0.73 0.98 0.98 0.94 0.09 0.97
Average Tag F-Measure 0.06 0.15 0.19 0.24 0.26 0.26 0.03 0.02 0.28 0.15 0.04
Average Tag Accuracy 0.91 0.09 0.43 0.71 0.73 0.72 0.90 0.89 0.90 0.17 0.90
Average AUC-ROC Clip 0.82 0.49 0.81 0.77 0.79 0.78 n/a n/a 0.84 0.69 0.78
Average AUC-ROC Tag 0.66 0.50 0.74 0.75 0.77 0.76 n/a n/a 0.77 0.50 0.50
Overall Beta-Binomial Positive Example Accuracy 0.01 1.00 0.89 0.68 0.69 0.67 0.00 0.00 0.26 1.00 0.00
Overall Beta-Binomial Negative Example Accuracy 1.00 0.00 0.35 0.72 0.74 0.73 1.00 1.00 0.95 0.00 1.00

download these results as csv


Summary Positive Example Accuracy (Average Across All Folds)

Tag Positive examples Negative examples BBE1 BBE2 BBE3 ME1 ME2 ME3 GP1 GP2 LB TB TTKV
metal 35 2180 0 1 0.75 0.78 0.66 0.67 0 0.1 0.18 0.33 0
ambient 60 2155 0 0.99 0.43 0.56 0.62 0.64 0 0.08 0.27 1 0
fast 109 2106 0 1 0.89 0.7 0.7 0.74 0 0.02 0.22 1 0
solo 53 2162 0 1 0.98 0.61 0.59 0.59 0 0 0.1 1 0
jazz 138 2077 0.05 1 0.91 0.7 0.74 0.72 0.04 0 0.4 1 0.02
instrumental 102 2113 0 1 0.73 0.59 0.62 0.59 0.33 0.04 0.08 1 0
horns 40 2175 0 1 0.8 0.54 0.54 0.47 0 0.06 0.02 0.67 0
house 65 2150 0 1 0.87 0.82 0.77 0.74 0 0.06 0.2 1 0
male 738 1477 0.12 1 1 0.71 0.72 0.72 0.01 0 0.49 1 0.2
beat 137 2078 0 1 0.8 0.74 0.78 0.74 0 0.04 0.25 1 0
strings 55 2160 0.08 1 0.85 0.71 0.73 0.78 0 0 0.22 1 0
saxophone 68 2147 0 1 0.85 0.8 0.82 0.72 0 0 0.33 1 0
piano 185 2030 0.06 1 0.65 0.67 0.72 0.74 0 0 0.34 1 0
loud 37 2178 0 1 0.8 0.71 0.67 0.71 0 0 0.17 0.67 0
noise 42 2173 0.26 0.97 0.71 0.59 0.51 0.55 0 0 0.16 0.67 0
80s 117 2098 0.01 1 0.93 0.6 0.59 0.63 0 0.17 0.23 1 0
pop 479 1736 0.1 1 0.98 0.67 0.69 0.71 0.02 0 0.45 1 0
female 345 1870 0.12 1 0.95 0.75 0.72 0.73 0.01 0.02 0.44 1 0
slow 159 2056 0 1 0.97 0.64 0.69 0.68 0.08 0 0.33 1 0
funk 37 2178 0 1 0.74 0.72 0.75 0.69 0 0.02 0.03 0.67 0
keyboard 42 2173 0 1 0.91 0.45 0.54 0.46 0 0 0 1 0
electronica 166 2049 0.06 1 0.94 0.68 0.72 0.68 0 0.03 0.28 1 0
rock 664 1551 0.24 1 0.98 0.75 0.75 0.73 0.05 0.01 0.69 1 0.24
vocal 270 1945 0 1 0.94 0.61 0.63 0.6 0 0 0.22 1 0
acoustic 38 2177 0 1 0.74 0.65 0.62 0.64 0 0 0.18 0.33 0
hip hop 152 2063 0.12 1 0.9 0.83 0.86 0.82 0.23 0 0.58 1 0.04
guitar 849 1366 0.12 1 0.99 0.72 0.72 0.73 0.15 0 0.64 1 0.37
bass 420 1795 0 1 0.95 0.5 0.53 0.54 0 0.07 0.23 1 0
british 82 2133 0.01 1 0.84 0.62 0.63 0.66 0 0 0.11 1 0
dance 324 1891 0.04 1 0.99 0.73 0.75 0.72 0.04 0 0.54 1 0.04
r&b 38 2177 0.02 1 0.56 0.6 0.65 0.7 0.02 0 0.18 0.33 0
electronic 485 1730 0.2 1 0.97 0.68 0.7 0.69 0.18 0 0.52 1 0.16
drum 927 1288 0.01 1 1 0.61 0.59 0.57 0.35 0 0.5 1 0.28
soft 62 2153 0 1 0.92 0.74 0.81 0.69 0.09 0 0.2 1 0
punk 51 2164 0 1 0.71 0.76 0.71 0.63 0 0 0.3 1 0
country 74 2141 0.18 1 0.98 0.6 0.62 0.61 0.05 0 0.15 1 0
drum machine 89 2126 0 1 0.89 0.66 0.69 0.69 0 0 0.21 1 0
voice 148 2067 0 1 0.97 0.53 0.59 0.52 0 0.36 0.08 1 0
quiet 43 2172 0.16 0.86 0.55 0.63 0.75 0.7 0 0.13 0.25 1 0
distortion 60 2155 0 1 0.82 0.68 0.63 0.64 0 0 0.13 1 0
synth 480 1735 0.02 1 0.97 0.68 0.63 0.62 0 0 0.4 1 0.02
rap 157 2058 0.46 1 0.68 0.87 0.88 0.85 0 0 0.65 1 0.05
techno 248 1967 0 1 0.92 0.76 0.8 0.75 0 0 0.5 1 0.02
organ 35 2180 0 1 0.68 0.4 0.31 0.33 0 0 0 0.67 0
trumpet 39 2176 0 1 0.81 0.63 0.69 0.58 0 0 0.14 0.67 0
MEAN 198.76 2016.24 0.05 1 0.85 0.67 0.68 0.66 0.04 0.03 0.28 0.91 0.03

download these results as csv

Summary Negative Example Accuracy (Average Across All Folds)

Tag Positive examples Negative examples BBE1 BBE2 BBE3 ME1 ME2 ME3 GP1 GP2 LB TB TTKV
metal 35 2180 1 0 0.64 0.79 0.79 0.79 1 0.97 0.99 0.67 1
ambient 60 2155 1 0 0.87 0.75 0.81 0.75 1 0.97 0.98 0 1
fast 109 2106 1 0 0.41 0.68 0.69 0.7 1 0.97 0.96 0 1
solo 53 2162 1 0 0.17 0.74 0.73 0.74 1 1 0.98 0 1
jazz 138 2077 0.99 0 0.47 0.85 0.87 0.87 1 1 0.96 0 0.98
instrumental 102 2113 1 0 0.37 0.63 0.65 0.64 0.67 0.95 0.96 0 1
horns 40 2175 1 0 0.31 0.66 0.67 0.64 1 0.94 0.98 0.33 1
house 65 2150 1 0 0.41 0.76 0.8 0.76 1 0.96 0.98 0 1
male 738 1477 0.91 0 0.05 0.72 0.74 0.74 0.99 1 0.75 0 0.8
beat 137 2078 1 0 0.55 0.7 0.74 0.75 1 0.95 0.95 0 1
strings 55 2160 0.97 0 0.46 0.76 0.75 0.77 1 1 0.98 0 1
saxophone 68 2147 1 0 0.45 0.85 0.85 0.84 1 0.99 0.98 0 1
piano 185 2030 0.96 0 0.44 0.65 0.68 0.69 1 1 0.94 0 1
loud 37 2178 1 0 0.56 0.81 0.82 0.81 1 1 0.99 0.33 1
noise 42 2173 0.95 0 0.69 0.76 0.78 0.78 1 0.99 0.98 0.33 1
80s 117 2098 1 0 0.13 0.68 0.68 0.7 1 0.92 0.96 0 1
pop 479 1736 0.93 0 0.1 0.68 0.68 0.68 0.99 1 0.85 0 1
female 345 1870 0.93 0 0.17 0.81 0.82 0.8 1 0.98 0.9 0 1
slow 159 2056 1 0 0.27 0.65 0.68 0.68 0.98 1 0.95 0 1
funk 37 2178 1 0 0.43 0.71 0.71 0.7 1 0.98 0.98 0.33 1
keyboard 42 2173 1 0 0.16 0.59 0.56 0.53 1 1 0.98 0 1
electronica 166 2049 0.99 0 0.39 0.68 0.73 0.73 1 0.96 0.94 0 1
rock 664 1551 0.92 0 0.28 0.79 0.81 0.8 0.99 0.99 0.87 0 0.72
vocal 270 1945 1 0 0.08 0.58 0.61 0.6 1 0.99 0.89 0 1
acoustic 38 2177 1 0 0.45 0.68 0.68 0.66 1 0.96 0.99 0.67 1
hip hop 152 2063 0.99 0 0.67 0.83 0.87 0.87 0.99 1 0.97 0 0.95
guitar 849 1366 0.94 0.01 0.22 0.71 0.73 0.73 0.95 1 0.78 0 0.62
bass 420 1795 1 0 0.05 0.57 0.58 0.6 1 0.94 0.82 0 1
british 82 2133 1 0 0.22 0.69 0.7 0.67 1 1 0.97 0 1
dance 324 1891 1 0 0.19 0.75 0.8 0.8 0.99 1 0.92 0 0.94
r&b 38 2177 0.99 0 0.66 0.75 0.76 0.74 1 1 0.99 0.67 1
electronic 485 1730 0.97 0 0.27 0.72 0.77 0.77 0.96 1 0.86 0 0.83
drum 927 1288 1 0.01 0.02 0.56 0.57 0.6 0.66 1 0.64 0 0.7
soft 62 2153 1 0 0.3 0.68 0.69 0.68 0.99 1 0.98 0 1
punk 51 2164 1 0 0.6 0.8 0.82 0.81 1 1 0.98 0 1
country 74 2141 0.95 0 0.19 0.73 0.73 0.76 0.98 1 0.97 0 1
drum machine 89 2126 1 0 0.43 0.65 0.69 0.68 1 1 0.97 0 1
voice 148 2067 1 0 0.06 0.52 0.54 0.54 1 0.65 0.93 0 1
quiet 43 2172 0.99 4.42E-004 0.83 0.84 0.85 0.85 1 0.97 0.99 0 1
distortion 60 2155 1 0 0.53 0.77 0.78 0.78 1 1 0.98 0 1
synth 480 1735 1 0 0.14 0.65 0.68 0.68 1 1 0.83 0 0.98
rap 157 2058 0.97 0 0.64 0.85 0.89 0.89 1 1 0.97 0 0.95
techno 248 1967 1 0 0.52 0.74 0.8 0.8 1 1 0.94 0 0.97
organ 35 2180 1 0 0.39 0.55 0.55 0.53 1 0.99 0.98 0.33 1
trumpet 39 2176 1 0 0.39 0.77 0.76 0.74 1 1 0.98 0.33 1
MEAN 198.76 2016.24 0.99 0 0.37 0.71 0.73 0.73 0.98 0.98 0.94 0.09 0.97

download these results as csv

Summary Binary relevance F-Measure (Average Across All Folds)

Tag Positive Examples Negative Examples BBE1 BBE2 BBE3 ME1 ME2 ME3 GP1 GP2 LB TB TTKV
metal 35 2180 0 0.03 0.09 0.1 0.09 0.09 0 0.06 0.17 0.01 0
ambient 60 2155 0 0.05 0.1 0.09 0.14 0.12 0 0.07 0.26 0.05 0
fast 109 2106 0 0.09 0.14 0.17 0.18 0.19 0 0.02 0.21 0.09 0
solo 53 2162 0 0.05 0.06 0.1 0.09 0.1 0 0 0.1 0.05 0
jazz 138 2077 0.08 0.12 0.21 0.36 0.4 0.39 0.07 0 0.4 0.12 0.02
instrumental 102 2113 0 0.09 0.1 0.13 0.14 0.13 0.03 0.02 0.08 0.09 0
horns 40 2175 0 0.04 0.04 0.05 0.06 0.05 0 0.02 0.02 0.02 0
house 65 2150 0 0.06 0.08 0.17 0.19 0.15 0 0.04 0.2 0.06 0
male 738 1477 0.17 0.5 0.51 0.63 0.64 0.65 0.02 0 0.49 0.5 0.24
beat 137 2078 0 0.12 0.2 0.24 0.28 0.27 0 0.03 0.25 0.12 0
strings 55 2160 0.05 0.05 0.08 0.13 0.13 0.14 0 0 0.22 0.05 0
saxophone 68 2147 0 0.06 0.09 0.26 0.26 0.22 0 0 0.31 0.06 0
piano 185 2030 0.07 0.15 0.11 0.24 0.27 0.29 0 0 0.34 0.15 0
loud 37 2178 0 0.03 0.1 0.11 0.11 0.11 0 0 0.19 0.02 0
noise 42 2173 0.05 0.04 0.09 0.08 0.08 0.09 0 0 0.15 0.02 0
80s 117 2098 0.01 0.1 0.11 0.16 0.16 0.18 0 0.05 0.23 0.1 0
pop 479 1736 0.12 0.36 0.38 0.47 0.49 0.5 0.04 0 0.45 0.36 0
female 345 1870 0.14 0.27 0.29 0.54 0.52 0.51 0.01 0.03 0.42 0.27 0.01
slow 159 2056 0 0.13 0.17 0.21 0.24 0.23 0.08 0 0.33 0.13 0
funk 37 2178 0 0.03 0.05 0.08 0.08 0.07 0 0.02 0.03 0.02 0
keyboard 42 2173 0 0.04 0.04 0.04 0.05 0.04 0 0 0 0.04 0
electronica 166 2049 0.09 0.14 0.2 0.24 0.28 0.27 0 0.03 0.27 0.14 0
rock 664 1551 0.27 0.46 0.54 0.67 0.68 0.67 0.08 0.02 0.69 0.46 0.26
vocal 270 1945 0 0.22 0.22 0.26 0.28 0.27 0 0 0.22 0.22 0
acoustic 38 2177 0 0.03 0.05 0.07 0.06 0.06 0 0 0.18 0.01 0
hip hop 152 2063 0.15 0.13 0.28 0.4 0.47 0.46 0.22 0 0.56 0.13 0.05
guitar 849 1366 0.18 0.56 0.61 0.66 0.67 0.67 0.18 0 0.64 0.55 0.37
bass 420 1795 0 0.32 0.32 0.3 0.32 0.33 0 0.1 0.23 0.32 0
british 82 2133 0.02 0.07 0.08 0.13 0.13 0.13 0 0 0.1 0.07 0
dance 324 1891 0.07 0.25 0.29 0.46 0.52 0.5 0.06 0 0.54 0.25 0.06
r&b 38 2177 0.04 0.03 0.06 0.08 0.09 0.09 0.04 0 0.18 0.01 0
electronic 485 1730 0.28 0.36 0.42 0.5 0.55 0.55 0.19 0 0.51 0.36 0.18
drum 927 1288 0.01 0.59 0.6 0.55 0.54 0.53 0.23 0 0.5 0.59 0.33
soft 62 2153 0 0.05 0.07 0.11 0.13 0.11 0.08 0 0.2 0.05 0
punk 51 2164 0 0.05 0.08 0.15 0.15 0.13 0 0 0.29 0.05 0
country 74 2141 0.13 0.06 0.08 0.14 0.14 0.15 0.03 0 0.15 0.06 0
drum machine 89 2126 0 0.08 0.12 0.13 0.15 0.15 0 0 0.22 0.08 0
voice 148 2067 0 0.13 0.13 0.13 0.15 0.13 0 0.12 0.08 0.13 0
quiet 43 2172 0.19 0.03 0.21 0.13 0.16 0.15 0 0.09 0.25 0.04 0
distortion 60 2155 0 0.05 0.09 0.14 0.13 0.13 0 0 0.12 0.05 0
synth 480 1735 0.03 0.36 0.38 0.46 0.45 0.45 0 0 0.39 0.35 0.04
rap 157 2058 0.48 0.13 0.21 0.46 0.52 0.52 0 0 0.63 0.13 0.06
techno 248 1967 0 0.2 0.33 0.39 0.47 0.45 0 0 0.49 0.2 0.02
organ 35 2180 0 0.03 0.03 0.03 0.02 0.02 0 0 0 0.02 0
trumpet 39 2176 0 0.03 0.05 0.08 0.09 0.07 0 0 0.13 0.02 0
MEAN 198.76 2016.24 0.06 0.15 0.19 0.24 0.26 0.26 0.03 0.02 0.28 0.15 0.04

download these results as csv

Summary Binary Accuracy (Average Across All Folds)

Tag Positive Examples Negative Examples BBE1 BBE2 BBE3 ME1 ME2 ME3 GP1 GP2 LB TB TTKV
metal 35 2180 0.98 0.02 0.64 0.79 0.79 0.78 0.98 0.95 0.97 0.66 0.98
ambient 60 2155 0.97 0.03 0.86 0.74 0.8 0.74 0.97 0.95 0.96 0.03 0.97
fast 109 2106 0.95 0.05 0.43 0.68 0.69 0.7 0.95 0.92 0.92 0.05 0.95
solo 53 2162 0.98 0.03 0.19 0.74 0.72 0.73 0.98 0.98 0.96 0.02 0.98
jazz 138 2077 0.93 0.07 0.5 0.84 0.86 0.86 0.94 0.94 0.93 0.06 0.92
instrumental 102 2113 0.95 0.05 0.39 0.63 0.65 0.63 0.65 0.91 0.92 0.05 0.95
horns 40 2175 0.98 0.02 0.32 0.66 0.67 0.64 0.98 0.92 0.96 0.34 0.98
house 65 2150 0.97 0.03 0.43 0.76 0.8 0.76 0.97 0.93 0.95 0.03 0.97
male 738 1477 0.64 0.34 0.37 0.72 0.73 0.74 0.66 0.66 0.66 0.34 0.59
beat 137 2078 0.94 0.06 0.57 0.71 0.75 0.75 0.94 0.9 0.91 0.06 0.94
strings 55 2160 0.95 0.03 0.47 0.76 0.75 0.77 0.97 0.98 0.96 0.02 0.98
saxophone 68 2147 0.97 0.03 0.46 0.85 0.85 0.84 0.97 0.95 0.96 0.03 0.97
piano 185 2030 0.88 0.09 0.45 0.66 0.68 0.7 0.92 0.92 0.89 0.08 0.92
loud 37 2178 0.98 0.02 0.56 0.81 0.82 0.81 0.98 0.98 0.97 0.34 0.98
noise 42 2173 0.93 0.02 0.69 0.76 0.77 0.78 0.98 0.97 0.97 0.34 0.98
80s 117 2098 0.95 0.06 0.17 0.68 0.68 0.69 0.95 0.88 0.92 0.05 0.95
pop 479 1736 0.75 0.22 0.29 0.68 0.68 0.69 0.78 0.78 0.76 0.22 0.78
female 345 1870 0.8 0.16 0.29 0.8 0.8 0.79 0.85 0.83 0.82 0.15 0.85
slow 159 2056 0.93 0.07 0.32 0.65 0.69 0.68 0.92 0.93 0.9 0.07 0.93
funk 37 2178 0.98 0.02 0.43 0.71 0.71 0.7 0.98 0.96 0.97 0.34 0.98
keyboard 42 2173 0.98 0.02 0.17 0.59 0.56 0.53 0.98 0.98 0.96 0.02 0.98
electronica 166 2049 0.92 0.08 0.42 0.69 0.73 0.73 0.93 0.89 0.89 0.07 0.93
rock 664 1551 0.72 0.3 0.49 0.78 0.79 0.78 0.71 0.7 0.81 0.3 0.58
vocal 270 1945 0.88 0.13 0.19 0.59 0.61 0.6 0.88 0.87 0.81 0.12 0.88
acoustic 38 2177 0.98 0.02 0.45 0.68 0.68 0.66 0.98 0.94 0.97 0.66 0.98
hip hop 152 2063 0.93 0.07 0.69 0.83 0.87 0.87 0.94 0.93 0.94 0.07 0.89
guitar 849 1366 0.63 0.39 0.52 0.72 0.73 0.73 0.64 0.62 0.72 0.38 0.52
bass 420 1795 0.81 0.19 0.22 0.56 0.57 0.59 0.81 0.78 0.71 0.19 0.81
british 82 2133 0.96 0.04 0.24 0.69 0.7 0.67 0.96 0.96 0.93 0.04 0.96
dance 324 1891 0.86 0.15 0.31 0.75 0.8 0.79 0.85 0.85 0.87 0.15 0.81
r&b 38 2177 0.98 0.02 0.66 0.75 0.76 0.74 0.98 0.98 0.97 0.66 0.98
electronic 485 1730 0.8 0.22 0.42 0.71 0.76 0.76 0.8 0.78 0.79 0.22 0.69
drum 927 1288 0.58 0.42 0.43 0.58 0.58 0.59 0.52 0.58 0.58 0.42 0.53
soft 62 2153 0.97 0.03 0.32 0.68 0.69 0.68 0.97 0.97 0.95 0.03 0.97
punk 51 2164 0.98 0.03 0.6 0.8 0.81 0.8 0.98 0.98 0.97 0.02 0.98
country 74 2141 0.92 0.04 0.22 0.73 0.73 0.75 0.95 0.97 0.94 0.03 0.97
drum machine 89 2126 0.96 0.04 0.45 0.65 0.69 0.69 0.96 0.96 0.94 0.04 0.96
voice 148 2067 0.93 0.07 0.12 0.52 0.54 0.54 0.93 0.63 0.88 0.07 0.93
quiet 43 2172 0.98 0.02 0.83 0.84 0.85 0.84 0.98 0.95 0.97 0.02 0.98
distortion 60 2155 0.97 0.03 0.54 0.77 0.78 0.77 0.97 0.97 0.95 0.03 0.97
synth 480 1735 0.79 0.22 0.32 0.66 0.67 0.67 0.78 0.78 0.74 0.22 0.77
rap 157 2058 0.93 0.07 0.65 0.86 0.89 0.89 0.93 0.93 0.95 0.07 0.89
techno 248 1967 0.89 0.11 0.56 0.74 0.8 0.8 0.89 0.89 0.89 0.11 0.86
organ 35 2180 0.98 0.02 0.39 0.55 0.54