2009:Audio Tag Classification Tagatune Results

From MIREX Wiki

Introduction

This task compares various algorithms' abilities to associate tags with 29-second audio clips of songs. The tags used were collected by the Tagatune game and algorithms were evaluated using the previously collected tags (using the same statistical procedures as the the other MIREX 2009 tag classification tasks) and in the Tagatune game itself (the Tagatune metric).

What is Tagatune?

Tagatune is a two-player game designed to extract information about music. In each round of the game, two players are each shown a song, either they are shown the same song or two different songs. Each player describes his given song by typing in any number of tags, which are immediately revealed to the partner. After reviewing each other's tags, the players must each decide whether they have been given the same piece of music as their partner. After both players have voted, the game reveals the true answer (whether the songs given to the pair of players are the same or different) and prepares the next round. Tagatune is live at www.gwap.com

http://www.cs.cmu.edu/~elaw/tagatune.jpg

Since Tagatune is a two-player game, when no partner is available for a player, a bot (a computer program or algorithm) is instituted to play against that player. In each round of the game, the bot generates a set of appropriate tags for a song and reveals these tags to the player. The player then decides his votes for same or different by comparing what he is listening to and the tags revealed by his bot partner. If the songs given to the bot and the player are identical, and the tags generated by the bot are accurate for the song, then the player will have a high probability of guessing correctly that the songs are the same. Otherwise, we would expect the player to make more mistakes in making this judgment. In short, the hypothesis is that better algorithms generate tags that are more fitting descriptions of songs, which in turn, allows players to have a higher chance of guessing correctly.

What is the goal of the MIREX Special Tagatune Evaluation?

The goal of the MIREX Special Tagatune Evaluation competition is to investigate a new method of evaluating music tagging algorithms, by using them as bots in Tagatune, and measuring the number of mistakes players make in guessing whether they are listening to the same or different songs (we will call this the Tagatune metric) when paired against different algorithm bots. We are particularly interested in whether there is a statistical correlation between the ranking of the algorithms induced by the Tagatune metric versus the classical metrics used in MIREX. For the motivation behind this evaluation, see this paper.

There are three main steps to this evaluation.

Step 1: Algorithm to Tags

All submitted algorithms are trained using the Tagatune training set and tested on the Tagatune test set. Artist filtering was used in the production of the test and training split, I.e. the training and test sets contained different artists. The trained algorithm must generate a set of tags for each of the songs in the test set, and rank the tags in a particular order (e.g. by confidence, saliency, relevance etc). This part of the evaluation is very similar, if not identical, to the MIREX 2009 Audio Tag Classification tasks where two outputs are produced by each algorithm:

  • a set of binary classifications indicating which tags are relevant to each example,
  • a set of 'affinity' scores which indicate the degree to which each tag applies to each track.

These different outputs allow the algorithms to be evaluated both on tag 'classification' and tag 'ranking' (where the tags may be ranked for each track and tracks ranked for each tag).

Step 2: Tagatune Experiments

The tags returned as 'relevant' by each algorithm were subsequently displayed to players of Tagatune in an internet-wide experiment. The number of mistakes players make in guessing whether the songs were the same or different was recorded for each algorithm.

Step 3: Ranking

The submitted algorithm were then evaluated by two methods:

(1) ranking using the MIREX metrics

(2) ranking using the Tagatune metric


The Tagatune Dataset

The Tagatune training and test set consist of music clips that are 29 seconds long, and are associated with 6622 tracks, 517 albums and 270 artists. The genres include classical, new age, electronica, rock, pop, world, jazz, blues, metal, punk etc. The tags used in the experiments are each associated with more than fifty songs, where each song is associated with a tag by more than two players independently. The following table shows the minimum, maximum and average number of songs associated with any tags in the training set, test set and the complete set used in this evaluation.


Training Set Test Set Complete Set
MIN 18 15 50
MAX 2103 3767 5870
AVG 212 288 502


Number of samples in training set: 9598

Number of samples in test set: 13194


The following is a list of 160 tags found in the Tagatune dataset.

no voicesingerduethard rock
worldharpsichordsitarchorus
female operamale vocalvocalsclarinet
heavysilencebeatsfunky
no stringschimesforeignno piano
hornsclassicalfemalespacey
jazzguitarquietno beat
banjoelectricsoloviolins
folkfemale voicewindambient
new agesynthfunkno singing
middle easterntrumpetpercussiondrum
airyvoicerepetitivebirds
stringsbassharpsicordmedieval
male voicegirlacousticloud
classicstringdrumselectronic
not classicalchantingno violinnot rock
no guitarorganno vocaltalking
choralweirdoperafast
electric guitarmale singerman singingclassical guitar
countryviolinelectrotribal
darkmale operano vocalsirish
electronicahornoperaticarabic
lowinstrumentaltrancechant
strangeheavy metalmodernbells
mandeepfast beathard
harpno flutepoplute
female vocaloboemelloworchestral
lightpianocelticmale vocals
orchestraeasternoldflutes
punkspanishsadsax
slowmalebluesvocal
indianindiawomanwoman singing
rockdancepiano sologuitars
no drumsjazzysingingcello
calmfemale vocalsvoicestechno
clappinghouseflutenot opera
not englishorientalbeatupbeat
softnoisechoirfemale singer
rapmetalhip hopwater
baroquewomenfiddleenglish


NOTE: An interesting effect of Tagatune is that we have collected many negative tags, which indicates the absence of an instrument (e.g. no piano, no guitar) or the genre that the song does not belong to (e.g. not classical, not rock). Participants of this evaluation might want to tailor their algorithms to take advantage of these negative tags that are not available on the MIREX 2008/2009 datasets.


MIREX Statistical Evaluation

Participating algorithms were evaluated over a single artist-filtered test/train split using both the full test set and only the 100 query subset used in Tagatune evaluation.

Binary (Classification) Evaluation

Algorithms are evaluated on their performance at tag classification using F-measure. Results are also reported for simple accuracy, however, as this statistic is dominated by the negative example accuracy it is not a reliable indicator of performance (as a system that returns no tags for any example will achieve a high score on this statistic). However, the accuracies are also reported for positive and negative examples separately as these can help elucidate the behaviour of an algorithm (for example demonstrating if the system is under of over predicting).

Affinity (Ranking) Evaluation

Algorithms are evaluated on their performance at tag ranking using the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The affinity scores for each tag to be applied to a track are sorted prior to the computation of the AUC-ROC statistic, which gives higher scores to ranked tag sets where the correct tags appear towards the top of the set.

General Legend

Team ID

Mandel = Michael Mandel
Manzagol = Pierre-Antoine Manzagol
Marsyas = George Tzanetakis
Zhi = Zhi-Sheng Chen
LabX = Anonymous

Results

The following sections provide detail the evaluation statistics computed. The results of the task are also detailed in the paper Evaluation of Algorithms Using Games: The Case of Music Tagging.

Overall Summary Results (Tagatune)

Measure Human LabX Mandel Manzagol Marsyas Zhi
Tagatune Metric 93.00% 26.80% 70.10% 67.50% 68.60% 60.90%

download these results as csv

Friedman's Test Results

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Tagatune metric for each track in the test. The tags generated by the algorithms are pre-processed to remove redundant or contradictory tags, which is important to maintain a minimum quality for the algorithm bots. This pre-processing is not done on the data for which other metrics are computed.

TeamID TeamID Lowerbound Mean Upperbound Significance
"Human" "Mandel" 0.563 1.265 1.967 TRUE
"Human" "Manzagol" 0.513 1.215 1.917 TRUE
"Human" "Marsyas" 0.773 1.475 2.177 TRUE
"Human" "Zhi" 1.258 1.960 2.662 TRUE
"Human" "LabX" 2.233 2.935 3.637 TRUE
"Mandel" "Manzagol" -0.752 -0.050 0.652 FALSE
"Mandel" "Marsyas" -0.492 0.210 0.912 FALSE
"Mandel" "Zhi" -0.007 0.695 1.397 FALSE
"Mandel" "LabX" 0.968 1.670 2.372 TRUE
"Manzagol" "Marsyas" -0.442 0.260 0.962 FALSE
"Manzagol" "Zhi" 0.043 0.745 1.447 TRUE
"Manzagol" "LabX" 1.018 1.720 2.422 TRUE
"Marsyas" "Zhi" -0.217 0.485 1.187 FALSE
"Marsyas" "LabX" 0.758 1.460 2.162 TRUE
"Zhi" "LabX" 0.273 0.975 1.677 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/tagatune_correctness.friedman.tukeyKramerHSD.png

Tagatune Correctness

Track Human LabX Mandel Manzagol Marsyas Zhi
4303 1 0 0.800 1 1 0
33934 1 0.200 1 0.800 0.800 1
48608 0.800 0.400 0.800 0.800 0.800 1
41810 1 0 0.600 0.200 1 1
42165 1 0.600 0.600 1 1 0.800
36668 0.800 0.200 0.800 0.800 0.600 0.400
54698 1 0.400 0.800 0.600 0.400 0.600
23910 1 0.600 0.400 0.600 0.600 0
55057 1 0.200 1 1 1 0.400
10943 1 0.800 1 1 0 0.800
32635 1 0.200 0.400 1 0.600 0.600
8802 1 0 0.800 1 0.600 1
48455 0.800 0.200 1 0.600 0.600 0.600
31267 0.600 0.600 1 0.800 0.600 0.400
25699 1 0.400 1 1 1 0
42361 1 0 1 0.800 0.400 0
21267 1 1 0.600 0 0 0.400
9956 1 0.200 0.400 0 0.400 0.400
44920 1 0 0.600 0.800 0.800 1
7313 1 0 0.400 0.400 0.400 0.400
28222 0.800 0 0.600 1 0.800 0.800
28224 1 0.200 0.600 0.800 0.600 0.800
19209 0.800 0.400 0.600 1 1 0.400
23773 1 0 0.600 1 1 1
43598 0.800 0 0.400 0.400 0 0.800
88 1 0 1 1 0.600 0.800
9494 1 0.200 0.200 0.400 0 0.600
16864 0.800 0.800 0.800 0.800 1 0.600
31905 1 0.200 0.800 1 0 0.200
15023 1 0 1 1 0.800 0.600
27304 1 0.200 0.400 0 0.800 0.600
16385 0.800 0.200 0.600 0.600 0.800 0.200
40029 0.800 1 1 1 0.800 0.200
43295 0.800 0 1 0 1 0.800
12795 0.600 1 0.200 0.200 1 0
44560 0.800 0 0.200 1 0 0.400
15325 1 0.200 1 1 0.600 0.400
33941 1 0.200 0.600 1 1 0.200
15134 0.800 0.400 0.600 0.600 1 0.400
4815 0.800 0.600 0.800 0.200 0.200 0.400
20022 1 0.400 0.400 1 1 0.600
26382 1 0.200 0.800 0.200 0.200 0.600
35687 1 0.200 1 0 0 0.200
45842 1 0 1 0.800 1 0.600
2456 0.800 0.800 0.600 0 0.200 0.600
15128 1 0 0.800 0.200 1 0.400
25228 0.800 0 0.600 0.800 0.600 0
46943 1 0 1 0.800 0 0.200
24215 0.800 0.400 0.400 0 0.200 0
20132 0.600 0.600 0.400 0.400 0.800 0.200
19370 1 0 0.800 0.800 0.800 0.200
2053 1 1 1 1 1 1
3217 1 0 0.800 0.400 0.200 1
49877 1 0 0.200 0.800 0.200 1
20030 1 0 0.600 1 1 0.600
55361 0.800 0.600 0.600 1 0.600 0.800
24920 1 0 0.800 1 0.800 0.600
25635 0.600 0 0.800 1 1 0.800
43638 0.800 0.800 0.800 0.600 0 0.200
13047 1 0.600 1 0.600 1 0.600
46941 0.600 0 0.400 0.200 0.400 0.400
34281 1 0 1 1 1 0.600
15093 0.600 0.400 0.800 0.400 0.200 0.400
36940 1 0 0.800 0.400 0.200 1
18122 0.600 0.800 0.200 0.200 0.400 0.200
3074 1 0 0.800 1 1 1
16429 1 0 0.600 0 1 1
44091 1 0.200 0.600 1 1 0.200
23230 1 0.200 0 0 0.400 0
29086 1 0.600 1 1 1 0.800
7561 1 0.800 1 0 0.600 1
42554 1 0.400 0.800 0.800 1 0.200
40638 0.800 0 0.400 1 0.600 0.200
19220 1 0.600 0.400 1 0.800 0.800
3227 1 0 0.800 1 0.600 0.600
14874 1 0.200 1 0.800 1 1
31684 1 0.800 1 0.400 0 0.800
48236 1 0.600 0.200 0.600 0.800 0.200
14241 1 0.200 0.200 1 1 0.800
48424 0.800 0 0.400 0.800 0.800 0.200
37579 0.800 0.200 1 1 1 1
27476 0.800 0.200 0.400 1 0.200 0.600
31220 1 0 0.400 0.800 0 0.800
36649 1 0.200 0.800 1 0 0
36168 1 0 0.800 0.600 1 1
36646 1 0.200 0.200 0.400 0.400 0
21872 1 0 1 0 0.600 0.400
13232 1 0.200 1 1 0.800 0.800
30803 0.800 0 1 0.400 0.600 1
45897 1 0.200 1 0.200 1 1
18287 0.800 1 0.600 0.800 0.800 0.800
26461 1 0.400 1 1 1 1
10949 1 0 0.800 1 0 0.400
51093 1 0 1 1 0.800 1
32514 1 0.200 0.800 0.600 0.800 0.600
53102 0.600 0.800 1 1 0.800 0
46769 1 0 1 1 1 0.200
4853 1 0.800 0.200 0.600 1 0.400
10944 1 1 0.800 1 0.600 0.800
36811 0.800 0.200 0.600 0.200 0.200 0.200

download these results as csv


Overall Summary Results (MIREX Statistical evaluation - Binary)

Full dataset

Measure LabX Mandel Manzagol Marsyas Zhi
Average Tag F-measure 0.001 0.132 0.098 0.125 0.138
Average Tag Accuracy 0.972 0.789 0.967 0.948 0.914
Average Positive Tag Accuracy 0.004 0.698 0.120 0.223 0.413
Average Negative Tag Accuracy 0.994 0.790 0.983 0.954 0.922

download these results as csv

100 query subset used in Tagatune evaluation

Measure LabX Mandel Manzagol Marsyas Zhi
Average Tag F-measure 0.002 0.304 0.110 0.212 0.250
Average Tag Accuracy 0.921 0.799 0.925 0.924 0.886
Average Positive Tag Accuracy 0.005 0.688 0.095 0.224 0.361
Average Negative Tag Accuracy 0.993 0.806 0.987 0.956 0.923

download these results as csv


Binary Relevance F-Measure

Full dataset

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
nostrings 13.000 6486.000 0.000 0.005 0.000 0.000 0.000
chimes 22.000 6477.000 0.022 0.016 0.000 0.035 0.046
sad 18.000 6481.000 0.006 0.014 0.062 0.000 0.026
nodrums 48.000 6451.000 0.000 0.019 0.000 0.000 0.017
femalevoice 105.000 6394.000 0.000 0.142 0.081 0.147 0.177
horn 7 6492.000 0.000 0.003 0.000 0.000 0.014
pop 196.000 6303.000 0.000 0.166 0.184 0.254 0.158
rock 601.000 5898.000 0.000 0.562 0.502 0.551 0.523
house 22.000 6477.000 0.000 0.025 0.028 0.000 0.029
birds 7 6492.000 0.000 0.012 0.000 0.034 0.020
harpsicord 59.000 6440.000 0.000 0.165 0.127 0.209 0.094
strange 22.000 6477.000 0.000 0.015 0.000 0.043 0.022
noflute 35.000 6464.000 0.000 0.008 0.000 0.000 0.015
novocal 263.000 6236.000 0.000 0.099 0.006 0.104 0.082
solo 217.000 6282.000 0.000 0.202 0.085 0.183 0.188
notenglish 11.000 6488.000 0.000 0.019 0.000 0.031 0.052
novoice 146.000 6353.000 0.000 0.058 0.011 0.058 0.048
newage 157.000 6342.000 0.000 0.139 0.000 0.174 0.116
synth 294.000 6205.000 0.000 0.190 0.083 0.233 0.192
upbeat 52.000 6447.000 0.000 0.040 0.029 0.070 0.057
slow 1043.000 5456.000 0.000 0.437 0.256 0.452 0.392
deep 12.000 6487.000 0.000 0.013 0.065 0.000 0.019
fiddle 14.000 6485.000 0.000 0.018 0.000 0.027 0.018
orchestral 12.000 6487.000 0.000 0.008 0.000 0.000 0.029
notclassical 14.000 6485.000 0.000 0.006 0.000 0.000 0.017
mansinging 46.000 6453.000 0.000 0.042 0.011 0.063 0.070
wind 22.000 6477.000 0.048 0.025 0.022 0.000 0.032
piano 630.000 5869.000 0.000 0.550 0.528 0.392 0.534
spanish 65.000 6434.000 0.000 0.050 0.011 0.060 0.068
femalesinger 30.000 6469.000 0.000 0.047 0.075 0.120 0.085
singing 242.000 6257.000 0.000 0.226 0.116 0.262 0.221
quiet 263.000 6236.000 0.000 0.219 0.054 0.342 0.212
oboe 12.000 6487.000 0.000 0.009 0.026 0.000 0.004
tribal 40.000 6459.000 0.000 0.022 0.036 0.090 0.082
noguitar 46.000 6453.000 0.000 0.018 0.000 0.011 0.049
femalevocal 126.000 6373.000 0.000 0.189 0.076 0.208 0.202
fastbeat 33.000 6466.000 0.000 0.029 0.000 0.000 0.057
hiphop 32.000 6467.000 0.000 0.058 0.222 0.000 0.121
instrumental 102.000 6397.000 0.000 0.045 0.026 0.053 0.048
chorus 50.000 6449.000 0.000 0.161 0.255 0.000 0.234
silence 12.000 6487.000 0.000 0.030 0.075 0.000 0.029
duet 18.000 6481.000 0.000 0.014 0.000 0.000 0.015
sax 20.000 6479.000 0.000 0.012 0.000 0.026 0.000
nobeat 14.000 6485.000 0.000 0.008 0.000 0.000 0.031
nopiano 90.000 6409.000 0.017 0.033 0.000 0.007 0.023
novocals 326.000 6173.000 0.000 0.119 0.006 0.117 0.100
pianosolo 13.000 6486.000 0.000 0.019 0.117 0.000 0.052
low 35.000 6464.000 0.000 0.039 0.097 0.031 0.030
weird 120.000 6379.000 0.000 0.075 0.036 0.145 0.103
dance 184.000 6315.000 0.000 0.216 0.214 0.179 0.265
harp 137.000 6362.000 0.000 0.137 0.085 0.148 0.128
horns 12.000 6487.000 0.000 0.009 0.035 0.000 0.012
funky 66.000 6433.000 0.000 0.073 0.082 0.000 0.105
hardrock 80.000 6419.000 0.000 0.182 0.115 0.000 0.171
bells 36.000 6463.000 0.000 0.021 0.028 0.046 0.042
punk 42.000 6457.000 0.000 0.122 0.159 0.000 0.120
electricguitar 51.000 6448.000 0.000 0.049 0.071 0.113 0.065
techno 827.000 5672.000 0.000 0.584 0.441 0.609 0.621
modern 73.000 6426.000 0.000 0.037 0.046 0.051 0.053
violins 258.000 6241.000 0.000 0.269 0.155 0.220 0.251
noviolin 18.000 6481.000 0.000 0.007 0.077 0.028 0.009
opera 325.000 6174.000 0.000 0.667 0.592 0.372 0.630
india 22.000 6477.000 0.000 0.025 0.074 0.033 0.167
cello 145.000 6354.000 0.000 0.376 0.268 0.208 0.266
sitar 250.000 6249.000 0.000 0.377 0.454 0.400 0.321
hard 25.000 6474.000 0.000 0.060 0.051 0.000 0.063
banjo 15.000 6484.000 0.000 0.015 0.051 0.013 0.026
blues 42.000 6457.000 0.000 0.095 0.103 0.121 0.053
man 128.000 6371.000 0.000 0.132 0.025 0.231 0.194
water 12.000 6487.000 0.000 0.027 0.000 0.000 0.026
femalevocals 90.000 6409.000 0.000 0.128 0.068 0.145 0.140
beat 534.000 5965.000 0.000 0.370 0.269 0.527 0.459
vocal 346.000 6153.000 0.000 0.277 0.102 0.295 0.228
jazz 88.000 6411.000 0.000 0.099 0.076 0.153 0.118
male 316.000 6183.000 0.000 0.310 0.193 0.327 0.293
maleopera 18.000 6481.000 0.000 0.125 0.147 0.000 0.128
drums 663.000 5836.000 0.000 0.374 0.235 0.417 0.367
electronic 578.000 5921.000 0.000 0.364 0.156 0.411 0.383
talking 27.000 6472.000 0.000 0.034 0.061 0.000 0.020
violin 908.000 5591.000 0.000 0.666 0.620 0.560 0.587
bass 73.000 6426.000 0.000 0.053 0.037 0.123 0.091
notrock 19.000 6480.000 0.000 0.004 0.000 0.034 0.035
string 91.000 6408.000 0.000 0.068 0.032 0.076 0.047
womansinging 32.000 6467.000 0.000 0.059 0.031 0.119 0.108
guitar 1166.000 5333.000 0.000 0.584 0.464 0.507 0.544
medieval 39.000 6460.000 0.000 0.044 0.072 0.012 0.038
clarinet 16.000 6483.000 0.000 0.028 0.000 0.000 0.036
world 14.000 6485.000 0.000 0.007 0.080 0.000 0.069
old 14.000 6485.000 0.000 0.012 0.000 0.040 0.012
middleeastern 17.000 6482.000 0.000 0.009 0.027 0.044 0.013
baroque 81.000 6418.000 0.019 0.120 0.015 0.155 0.093
oriental 50.000 6449.000 0.000 0.038 0.055 0.072 0.072
trumpet 17.000 6482.000 0.000 0.016 0.080 0.000 0.000
irish 49.000 6450.000 0.000 0.070 0.018 0.092 0.030
ambient 419.000 6080.000 0.000 0.432 0.028 0.397 0.308
funk 32.000 6467.000 0.000 0.062 0.086 0.000 0.057
metal 159.000 6340.000 0.006 0.333 0.187 0.000 0.295
woman 186.000 6313.000 0.000 0.292 0.106 0.274 0.335
dark 36.000 6463.000 0.000 0.045 0.035 0.000 0.027
acoustic 66.000 6433.000 0.012 0.081 0.097 0.106 0.124
light 16.000 6483.000 0.000 0.009 0.000 0.054 0.007
repetitive 24.000 6475.000 0.000 0.012 0.000 0.000 0.000
trance 51.000 6448.000 0.000 0.049 0.021 0.057 0.063
celtic 27.000 6472.000 0.000 0.024 0.000 0.067 0.017
electric 44.000 6455.000 0.000 0.022 0.000 0.013 0.039
malevocals 123.000 6376.000 0.066 0.122 0.122 0.178 0.101
heavy 59.000 6440.000 0.000 0.110 0.161 0.000 0.120
jazzy 68.000 6431.000 0.000 0.081 0.099 0.112 0.112
country 122.000 6377.000 0.000 0.179 0.141 0.190 0.128
beats 157.000 6342.000 0.009 0.131 0.123 0.240 0.223
loud 313.000 6186.000 0.000 0.307 0.148 0.448 0.318
classical 1544.000 4955.000 0.000 0.727 0.244 0.523 0.618
voices 39.000 6460.000 0.000 0.045 0.000 0.038 0.103
flutes 54.000 6445.000 0.000 0.124 0.298 0.000 0.227
choral 104.000 6395.000 0.000 0.360 0.282 0.256 0.475
harpsichord 263.000 6236.000 0.000 0.398 0.375 0.384 0.290
eastern 80.000 6419.000 0.000 0.068 0.100 0.185 0.110
foreign 51.000 6448.000 0.000 0.061 0.023 0.124 0.142
fast 616.000 5883.000 0.000 0.324 0.154 0.384 0.291
english 11.000 6488.000 0.000 0.005 0.000 0.000 0.014
spacey 27.000 6472.000 0.000 0.047 0.000 0.052 0.033
electro 87.000 6412.000 0.000 0.064 0.015 0.079 0.091
calm 33.000 6466.000 0.000 0.016 0.000 0.038 0.023
lute 15.000 6484.000 0.000 0.049 0.088 0.000 0.051
arabic 10.000 6489.000 0.000 0.003 0.019 0.000 0.000
voice 111.000 6388.000 0.000 0.086 0.057 0.115 0.086
vocals 256.000 6243.000 0.000 0.176 0.084 0.203 0.187
rap 41.000 6458.000 0.000 0.111 0.219 0.000 0.296
singer 25.000 6474.000 0.000 0.023 0.029 0.021 0.000
strings 997.000 5502.000 0.000 0.551 0.141 0.509 0.461
orchestra 98.000 6401.000 0.000 0.117 0.108 0.118 0.110
guitars 25.000 6474.000 0.000 0.011 0.000 0.000 0.036
chant 51.000 6448.000 0.000 0.190 0.182 0.350 0.262
heavymetal 43.000 6456.000 0.000 0.106 0.109 0.000 0.123
girl 10.000 6489.000 0.000 0.011 0.000 0.000 0.041
percussion 26.000 6473.000 0.004 0.020 0.000 0.032 0.059
flute 455.000 6044.000 0.000 0.609 0.589 0.479 0.475
drum 89.000 6410.000 0.000 0.066 0.060 0.110 0.101
classic 235.000 6264.000 0.000 0.210 0.106 0.180 0.185
nosinging 51.000 6448.000 0.000 0.019 0.000 0.038 0.012
chanting 32.000 6467.000 0.000 0.068 0.053 0.086 0.122
folk 48.000 6451.000 0.000 0.036 0.018 0.044 0.063
malesinger 39.000 6460.000 0.000 0.033 0.045 0.084 0.051
mellow 29.000 6470.000 0.000 0.012 0.000 0.000 0.015
indian 313.000 6186.000 0.000 0.284 0.185 0.319 0.269
electronica 39.000 6460.000 0.000 0.028 0.000 0.055 0.045
women 22.000 6477.000 0.000 0.047 0.057 0.111 0.129
notopera 19.000 6480.000 0.000 0.004 0.000 0.000 0.028
noise 16.000 6483.000 0.000 0.016 0.065 0.000 0.021
soft 248.000 6251.000 0.000 0.166 0.075 0.214 0.169
femaleopera 27.000 6472.000 0.000 0.107 0.123 0.000 0.126
malevoice 155.000 6344.000 0.000 0.141 0.062 0.192 0.153
organ 17.000 6482.000 0.000 0.007 0.006 0.011 0.009
female 320.000 6179.000 0.000 0.460 0.244 0.322 0.408
classicalguitar 38.000 6461.000 0.000 0.180 0.104 0.000 0.129
operatic 17.000 6482.000 0.000 0.036 0.000 0.000 0.070
airy 12.000 6487.000 0.026 0.013 0.000 0.038 0.023
malevocal 271.000 6228.000 0.000 0.273 0.200 0.304 0.218
clapping 12.000 6487.000 0.000 0.008 0.000 0.000 0.000
choir 161.000 6338.000 0.000 0.508 0.544 0.429 0.590

download these results as csv

100 query subset used in Tagatune evaluation

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
sad 5 95.000 0.125 0.167 0.333 0.000 0.214
nodrums 4 96.000 0.000 0.102 0.000 0.000 0.148
femalevoice 6 94.000 0.000 0.385 0.000 0.308 0.154
pop 10.000 90.000 0.000 0.462 0.522 0.700 0.500
rock 14.000 86.000 0.000 0.710 0.333 0.718 0.688
birds 1 99.000 0.000 0.200 0.000 0.000 0.000
harpsicord 3 97.000 0.000 0.400 0.000 0.000 0.200
strange 2 98.000 0.000 0.085 0.000 0.000 0.000
novocal 12.000 88.000 0.000 0.328 0.154 0.100 0.083
solo 11.000 89.000 0.000 0.276 0.154 0.235 0.211
notenglish 1 99.000 0.000 0.000 0.000 0.000 0.000
novoice 4 96.000 0.000 0.143 0.000 0.000 0.000
newage 12.000 88.000 0.000 0.421 0.000 0.400 0.333
synth 11.000 89.000 0.000 0.514 0.167 0.667 0.333
upbeat 4 96.000 0.000 0.214 0.000 0.000 0.308
slow 44.000 56.000 0.000 0.769 0.259 0.733 0.485
deep 2 98.000 0.000 0.091 0.000 0.000 0.000
fiddle 1 99.000 0.000 0.069 0.000 0.000 0.000
orchestral 2 98.000 0.000 0.062 0.000 0.000 0.121
mansinging 2 98.000 0.000 0.167 0.000 0.286 0.333
wind 1 99.000 0.000 0.095 0.000 0.000 0.105
piano 9 91.000 0.000 0.286 0.100 0.286 0.267
femalesinger 4 96.000 0.000 0.240 0.000 0.000 0.000
singing 13.000 87.000 0.000 0.667 0.235 0.480 0.500
quiet 16.000 84.000 0.000 0.500 0.118 0.611 0.471
tribal 1 99.000 0.000 0.167 0.000 0.000 1.000
noguitar 2 98.000 0.000 0.073 0.000 0.000 0.000
femalevocal 13.000 87.000 0.000 0.545 0.190 0.609 0.421
fastbeat 1 99.000 0.000 0.000 0.000 0.000 0.000
instrumental 4 96.000 0.000 0.131 0.000 0.000 0.125
chorus 3 97.000 0.000 0.444 0.000 0.000 0.500
silence 1 99.000 0.000 0.200 0.000 0.000 0.333
sax 2 98.000 0.000 0.000 0.000 0.000 0.000
nobeat 1 99.000 0.000 0.044 0.000 0.000 0.000
nopiano 3 97.000 0.000 0.095 0.000 0.333 0.000
novocals 15.000 85.000 0.000 0.353 0.000 0.222 0.087
low 5 95.000 0.000 0.207 0.000 0.000 0.000
weird 3 97.000 0.000 0.121 0.000 0.000 0.000
dance 4 96.000 0.000 0.615 0.250 0.444 0.500
harp 3 97.000 0.000 0.000 0.000 0.000 0.286
horns 3 97.000 0.000 0.000 0.000 0.000 0.000
funky 1 99.000 0.000 0.118 0.000 0.000 0.000
hardrock 4 96.000 0.000 0.533 0.000 0.000 0.421
bells 2 98.000 0.000 0.080 0.667 0.000 0.000
punk 4 96.000 0.000 0.615 0.857 0.000 0.500
techno 12.000 88.000 0.000 0.621 0.500 0.571 0.480
modern 5 95.000 0.000 0.158 0.333 0.250 0.364
violins 25.000 75.000 0.000 0.656 0.258 0.353 0.538
opera 6 94.000 0.000 0.571 0.182 0.462 0.571
cello 21.000 79.000 0.000 0.723 0.345 0.378 0.467
sitar 1 99.000 0.000 0.000 0.000 0.000 0.500
man 4 96.000 0.000 0.167 0.400 0.400 0.000
femalevocals 9 91.000 0.000 0.462 0.000 0.353 0.286
beat 13.000 87.000 0.000 0.457 0.000 0.545 0.500
vocal 22.000 78.000 0.000 0.708 0.000 0.718 0.345
jazz 3 97.000 0.000 0.333 0.000 0.000 0.333
male 10.000 90.000 0.000 0.438 0.133 0.545 0.000
drums 14.000 86.000 0.000 0.564 0.222 0.533 0.480
electronic 16.000 84.000 0.000 0.571 0.316 0.690 0.533
violin 44.000 56.000 0.000 0.886 0.800 0.835 0.685
bass 5 95.000 0.000 0.171 0.000 0.250 0.000
string 12.000 88.000 0.000 0.333 0.000 0.000 0.160
womansinging 3 97.000 0.000 0.240 0.000 0.000 0.000
guitar 15.000 85.000 0.000 0.500 0.500 0.393 0.381
medieval 5 95.000 0.000 0.267 0.000 0.000 0.000
old 1 99.000 0.000 0.050 0.000 0.000 0.061
middleeastern 1 99.000 0.000 0.000 0.000 0.000 0.000
baroque 7 93.000 0.000 0.276 0.000 0.000 0.238
oriental 1 99.000 0.000 0.000 0.000 0.000 0.000
trumpet 2 98.000 0.000 0.000 0.000 0.000 0.000
irish 1 99.000 0.000 0.091 0.000 0.000 0.222
ambient 14.000 86.000 0.000 0.625 0.000 0.667 0.333
funk 2 98.000 0.000 0.444 0.000 0.000 0.500
metal 5 95.000 0.160 0.714 0.333 0.000 0.500
woman 14.000 86.000 0.000 0.710 0.000 0.583 0.381
dark 1 99.000 0.000 0.000 0.000 0.000 0.000
acoustic 3 97.000 0.000 0.375 0.000 0.500 0.500
light 1 99.000 0.000 0.040 0.000 0.000 0.500
trance 4 96.000 0.000 0.316 0.000 0.000 0.462
celtic 1 99.000 0.000 0.057 0.000 0.000 0.000
electric 1 99.000 0.000 0.074 0.000 0.000 0.000
malevocals 6 94.000 0.000 0.154 0.250 0.267 0.300
heavy 4 96.000 0.000 0.250 0.400 0.000 0.353
jazzy 3 97.000 0.000 0.267 0.750 0.000 0.462
country 2 98.000 0.000 0.250 0.000 0.222 0.000
beats 4 96.000 0.000 0.087 0.000 0.000 0.000
loud 11.000 89.000 0.000 0.545 0.154 0.588 0.571
classical 48.000 52.000 0.000 0.839 0.038 0.821 0.771
voices 2 98.000 0.000 0.250 0.000 0.000 0.400
choral 7 93.000 0.000 0.500 0.000 0.000 0.333
harpsichord 12.000 88.000 0.000 0.621 0.435 0.741 0.458
eastern 1 99.000 0.000 0.091 0.000 0.000 0.667
foreign 2 98.000 0.000 0.111 0.000 0.000 0.000
fast 17.000 83.000 0.000 0.389 0.000 0.545 0.357
english 1 99.000 0.000 0.000 0.000 0.000 0.000
spacey 2 98.000 0.000 0.211 0.000 0.667 0.095
electro 4 96.000 0.000 0.231 0.000 0.000 0.308
calm 6 94.000 0.000 0.087 0.000 0.000 0.000
voice 10.000 90.000 0.000 0.545 0.000 0.333 0.143
vocals 16.000 84.000 0.000 0.512 0.273 0.387 0.200
singer 1 99.000 0.000 0.069 0.500 0.000 0.000
strings 47.000 53.000 0.000 0.905 0.151 0.872 0.780
orchestra 5 95.000 0.000 0.138 0.000 0.000 0.000
chant 4 96.000 0.000 0.444 0.000 0.400 0.667
heavymetal 2 98.000 0.000 0.364 0.000 0.000 0.308
girl 2 98.000 0.000 0.083 0.000 0.000 0.333
flute 8 92.000 0.000 0.444 0.500 0.316 0.143
drum 3 97.000 0.000 0.160 0.000 0.500 0.000
classic 24.000 76.000 0.000 0.687 0.148 0.368 0.462
nosinging 3 97.000 0.000 0.094 0.000 0.000 0.000
chanting 2 98.000 0.000 0.222 0.000 0.000 0.000
folk 2 98.000 0.000 0.105 0.000 0.000 0.000
malesinger 1 99.000 0.000 0.077 0.500 0.286 0.000
mellow 5 95.000 0.000 0.176 0.000 0.000 0.000
indian 3 97.000 0.000 0.154 0.000 0.000 0.400
electronica 2 98.000 0.000 0.125 0.000 0.000 0.200
women 3 97.000 0.000 0.235 0.000 0.500 0.333
soft 21.000 79.000 0.000 0.353 0.000 0.308 0.242
malevoice 9 91.000 0.000 0.452 0.000 0.471 0.143
organ 1 99.000 0.000 0.000 0.000 0.000 0.000
female 19.000 81.000 0.000 0.842 0.516 0.688 0.444
classicalguitar 1 99.000 0.000 0.500 0.000 0.000 0.500
airy 4 96.000 0.000 0.258 0.000 0.400 0.381
malevocal 11.000 89.000 0.000 0.545 0.333 0.571 0.286
clapping 2 98.000 0.000 0.000 0.000 0.000 0.000
choir 7 93.000 0.000 0.615 0.250 0.308 0.500

download these results as csv

Binary Accuracy

Full dataset

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
nostrings 13.000 6486.000 0.998 0.496 0.998 0.992 0.986
chimes 22.000 6477.000 0.986 0.778 0.994 0.992 0.981
sad 18.000 6481.000 0.894 0.705 0.995 0.989 0.862
nodrums 48.000 6451.000 0.993 0.569 0.991 0.977 0.824
femalevoice 105.000 6394.000 0.984 0.851 0.962 0.945 0.950
horn 7 6492.000 0.996 0.801 0.997 0.991 0.978
pop 196.000 6303.000 0.970 0.749 0.877 0.921 0.838
rock 601.000 5898.000 0.908 0.878 0.922 0.863 0.859
house 22.000 6477.000 0.997 0.793 0.989 0.991 0.939
birds 7 6492.000 0.998 0.922 0.997 0.991 0.923
harpsicord 59.000 6440.000 0.991 0.929 0.979 0.964 0.864
strange 22.000 6477.000 0.997 0.588 0.996 0.986 0.960
noflute 35.000 6464.000 0.987 0.559 0.994 0.982 0.939
novocal 263.000 6236.000 0.960 0.548 0.952 0.870 0.816
solo 217.000 6282.000 0.966 0.831 0.960 0.890 0.861
notenglish 11.000 6488.000 0.998 0.839 0.998 0.990 0.972
novoice 146.000 6353.000 0.978 0.569 0.972 0.935 0.865
newage 157.000 6342.000 0.976 0.759 0.974 0.936 0.783
synth 294.000 6205.000 0.955 0.696 0.949 0.874 0.789
upbeat 52.000 6447.000 0.992 0.693 0.979 0.980 0.883
slow 1043.000 5456.000 0.840 0.708 0.814 0.716 0.743
deep 12.000 6487.000 0.998 0.830 0.996 0.991 0.953
fiddle 14.000 6485.000 0.998 0.817 0.991 0.989 0.950
orchestral 12.000 6487.000 0.998 0.779 0.995 0.989 0.908
notclassical 14.000 6485.000 0.998 0.660 0.996 0.992 0.945
mansinging 46.000 6453.000 0.993 0.748 0.972 0.982 0.955
wind 22.000 6477.000 0.994 0.834 0.987 0.984 0.878
piano 630.000 5869.000 0.903 0.879 0.891 0.758 0.889
spanish 65.000 6434.000 0.990 0.826 0.973 0.976 0.954
femalesinger 30.000 6469.000 0.995 0.870 0.992 0.984 0.974
singing 242.000 6257.000 0.963 0.802 0.942 0.889 0.931
quiet 263.000 6236.000 0.960 0.792 0.957 0.890 0.813
oboe 12.000 6487.000 0.998 0.935 0.977 0.977 0.917
tribal 40.000 6459.000 0.994 0.770 0.992 0.975 0.973
noguitar 46.000 6453.000 0.993 0.560 0.992 0.972 0.916
femalevocal 126.000 6373.000 0.981 0.870 0.936 0.933 0.949
fastbeat 33.000 6466.000 0.995 0.784 0.991 0.989 0.914
hiphop 32.000 6467.000 0.995 0.906 0.992 0.991 0.978
instrumental 102.000 6397.000 0.976 0.598 0.965 0.950 0.819
chorus 50.000 6449.000 0.992 0.939 0.982 0.970 0.965
silence 12.000 6487.000 0.983 0.901 0.992 0.991 0.959
duet 18.000 6481.000 0.997 0.890 0.986 0.988 0.938
sax 20.000 6479.000 0.974 0.896 0.986 0.988 0.960
nobeat 14.000 6485.000 0.994 0.677 0.997 0.990 0.943
nopiano 90.000 6409.000 0.948 0.571 0.983 0.956 0.894
novocals 326.000 6173.000 0.950 0.546 0.948 0.856 0.814
pianosolo 13.000 6486.000 0.998 0.854 0.986 0.991 0.932
low 35.000 6464.000 0.995 0.816 0.986 0.981 0.872
weird 120.000 6379.000 0.982 0.641 0.975 0.946 0.914
dance 184.000 6315.000 0.972 0.823 0.941 0.939 0.888
harp 137.000 6362.000 0.979 0.885 0.974 0.932 0.882
horns 12.000 6487.000 0.989 0.935 0.992 0.988 0.975
funky 66.000 6433.000 0.990 0.774 0.972 0.979 0.916
hardrock 80.000 6419.000 0.988 0.894 0.981 0.968 0.887
bells 36.000 6463.000 0.994 0.731 0.989 0.981 0.972
punk 42.000 6457.000 0.964 0.927 0.959 0.986 0.924
electricguitar 51.000 6448.000 0.992 0.820 0.988 0.978 0.868
techno 827.000 5672.000 0.873 0.832 0.892 0.853 0.877
modern 73.000 6426.000 0.989 0.592 0.975 0.965 0.896
violins 258.000 6241.000 0.939 0.819 0.933 0.922 0.849
noviolin 18.000 6481.000 0.997 0.519 0.996 0.989 0.965
opera 325.000 6174.000 0.950 0.954 0.955 0.847 0.955
india 22.000 6477.000 0.996 0.770 0.996 0.991 0.989
cello 145.000 6354.000 0.978 0.947 0.950 0.913 0.955
sitar 250.000 6249.000 0.962 0.890 0.960 0.928 0.890
hard 25.000 6474.000 0.991 0.884 0.994 0.990 0.894
banjo 15.000 6484.000 0.998 0.959 0.994 0.977 0.942
blues 42.000 6457.000 0.994 0.921 0.987 0.982 0.989
man 128.000 6371.000 0.980 0.792 0.964 0.951 0.958
water 12.000 6487.000 0.998 0.911 0.994 0.991 0.896
femalevocals 90.000 6409.000 0.986 0.867 0.962 0.957 0.964
beat 534.000 5965.000 0.918 0.755 0.910 0.893 0.865
vocal 346.000 6153.000 0.947 0.784 0.930 0.847 0.916
jazz 88.000 6411.000 0.986 0.812 0.940 0.957 0.905
male 316.000 6183.000 0.951 0.822 0.934 0.895 0.922
maleopera 18.000 6481.000 0.997 0.966 0.991 0.988 0.966
drums 663.000 5836.000 0.898 0.714 0.863 0.803 0.819
electronic 578.000 5921.000 0.911 0.737 0.902 0.818 0.817
talking 27.000 6472.000 0.996 0.834 0.995 0.988 0.985
violin 908.000 5591.000 0.860 0.876 0.872 0.806 0.863
bass 73.000 6426.000 0.989 0.698 0.976 0.965 0.902
notrock 19.000 6480.000 0.997 0.493 0.996 0.991 0.983
string 91.000 6408.000 0.986 0.719 0.972 0.959 0.846
womansinging 32.000 6467.000 0.995 0.854 0.990 0.982 0.982
guitar 1166.000 5333.000 0.821 0.821 0.851 0.691 0.843
medieval 39.000 6460.000 0.994 0.791 0.988 0.976 0.874
clarinet 16.000 6483.000 0.998 0.893 0.992 0.989 0.950
world 14.000 6485.000 0.998 0.605 0.996 0.992 0.983
old 14.000 6485.000 0.998 0.713 0.995 0.993 0.821
middleeastern 17.000 6482.000 0.968 0.680 0.989 0.987 0.953
baroque 81.000 6418.000 0.854 0.843 0.980 0.955 0.796
oriental 50.000 6449.000 0.992 0.725 0.974 0.980 0.936
trumpet 17.000 6482.000 0.958 0.847 0.996 0.990 0.992
irish 49.000 6450.000 0.992 0.858 0.983 0.976 0.912
ambient 419.000 6080.000 0.936 0.866 0.936 0.858 0.814
funk 32.000 6467.000 0.995 0.870 0.987 0.990 0.944
metal 159.000 6340.000 0.842 0.912 0.973 0.939 0.887
woman 186.000 6313.000 0.971 0.889 0.956 0.914 0.948
dark 36.000 6463.000 0.994 0.798 0.992 0.979 0.856
acoustic 66.000 6433.000 0.974 0.842 0.966 0.979 0.941
light 16.000 6483.000 0.998 0.607 0.996 0.989 0.954
repetitive 24.000 6475.000 0.948 0.755 0.996 0.992 0.989
trance 51.000 6448.000 0.992 0.757 0.985 0.975 0.881
celtic 27.000 6472.000 0.996 0.751 0.992 0.987 0.963
electric 44.000 6455.000 0.993 0.610 0.971 0.976 0.932
malevocals 123.000 6376.000 0.930 0.776 0.958 0.952 0.872
heavy 59.000 6440.000 0.928 0.873 0.984 0.975 0.876
jazzy 68.000 6431.000 0.990 0.806 0.939 0.968 0.919
country 122.000 6377.000 0.981 0.880 0.925 0.945 0.952
beats 157.000 6342.000 0.967 0.721 0.952 0.947 0.881
loud 313.000 6186.000 0.952 0.827 0.947 0.923 0.842
classical 1544.000 4955.000 0.762 0.848 0.768 0.569 0.788
voices 39.000 6460.000 0.994 0.842 0.990 0.976 0.965
flutes 54.000 6445.000 0.992 0.904 0.982 0.976 0.957
choral 104.000 6395.000 0.984 0.952 0.984 0.959 0.971
harpsichord 263.000 6236.000 0.960 0.906 0.908 0.894 0.828
eastern 80.000 6419.000 0.988 0.736 0.978 0.959 0.920
foreign 51.000 6448.000 0.992 0.824 0.974 0.970 0.980
fast 616.000 5883.000 0.905 0.723 0.902 0.804 0.799
english 11.000 6488.000 0.998 0.768 0.991 0.991 0.978
spacey 27.000 6472.000 0.996 0.857 0.994 0.983 0.892
electro 87.000 6412.000 0.985 0.686 0.980 0.957 0.874
calm 33.000 6466.000 0.984 0.653 0.989 0.985 0.921
lute 15.000 6484.000 0.998 0.922 0.990 0.984 0.937
arabic 10.000 6489.000 0.998 0.732 0.984 0.988 0.971
voice 111.000 6388.000 0.983 0.708 0.964 0.938 0.938
vocals 256.000 6243.000 0.961 0.743 0.906 0.886 0.920
rap 41.000 6458.000 0.994 0.934 0.978 0.986 0.985
singer 25.000 6474.000 0.996 0.736 0.979 0.986 0.985
strings 997.000 5502.000 0.847 0.795 0.841 0.756 0.771
orchestra 98.000 6401.000 0.985 0.840 0.975 0.945 0.860
guitars 25.000 6474.000 0.996 0.730 0.992 0.990 0.919
chant 51.000 6448.000 0.992 0.950 0.989 0.982 0.971
heavymetal 43.000 6456.000 0.993 0.904 0.982 0.986 0.918
girl 10.000 6489.000 0.976 0.756 0.997 0.992 0.971
percussion 26.000 6473.000 0.862 0.729 0.993 0.981 0.961
flute 455.000 6044.000 0.930 0.927 0.944 0.889 0.902
drum 89.000 6410.000 0.986 0.689 0.976 0.960 0.931
classic 235.000 6264.000 0.964 0.760 0.938 0.888 0.807
nosinging 51.000 6448.000 0.992 0.529 0.987 0.977 0.926
chanting 32.000 6467.000 0.995 0.945 0.994 0.984 0.969
folk 48.000 6451.000 0.993 0.802 0.983 0.973 0.968
malesinger 39.000 6460.000 0.994 0.748 0.927 0.980 0.942
mellow 29.000 6470.000 0.992 0.649 0.992 0.987 0.958
indian 313.000 6186.000 0.952 0.813 0.942 0.878 0.927
electronica 39.000 6460.000 0.994 0.629 0.988 0.979 0.921
women 22.000 6477.000 0.997 0.886 0.995 0.990 0.981
notopera 19.000 6480.000 0.997 0.558 0.997 0.991 0.989
noise 16.000 6483.000 0.998 0.792 0.996 0.981 0.901
soft 248.000 6251.000 0.962 0.754 0.947 0.889 0.823
femaleopera 27.000 6472.000 0.996 0.936 0.985 0.980 0.949
malevoice 155.000 6344.000 0.976 0.766 0.949 0.940 0.933
organ 17.000 6482.000 0.997 0.786 0.951 0.973 0.867
female 320.000 6179.000 0.951 0.905 0.896 0.856 0.936
classicalguitar 38.000 6461.000 0.994 0.958 0.973 0.975 0.932
operatic 17.000 6482.000 0.997 0.867 0.996 0.989 0.943
airy 12.000 6487.000 0.988 0.762 0.997 0.992 0.881
malevocal 271.000 6228.000 0.958 0.810 0.908 0.911 0.889
clapping 12.000 6487.000 0.998 0.815 0.997 0.992 0.996
choir 161.000 6338.000 0.975 0.958 0.983 0.951 0.974

download these results as csv

100 query subset used in Tagatune evaluation

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
sad 5 95.000 0.860 0.600 0.960 0.950 0.780
nodrums 4 96.000 0.960 0.470 0.960 0.950 0.770
femalevoice 6 94.000 0.940 0.840 0.900 0.910 0.890
pop 10.000 90.000 0.900 0.790 0.890 0.940 0.860
rock 14.000 86.000 0.860 0.910 0.880 0.890 0.900
birds 1 99.000 0.990 0.920 0.990 0.990 0.830
harpsicord 3 97.000 0.970 0.910 0.950 0.970 0.760
strange 2 98.000 0.980 0.570 0.980 0.960 0.930
novocal 12.000 88.000 0.880 0.590 0.890 0.820 0.780
solo 11.000 89.000 0.890 0.790 0.890 0.870 0.850
notenglish 1 99.000 0.990 0.820 0.990 0.990 0.930
novoice 4 96.000 0.960 0.640 0.960 0.940 0.930
newage 12.000 88.000 0.880 0.780 0.880 0.910 0.760
synth 11.000 89.000 0.890 0.830 0.900 0.940 0.840
upbeat 4 96.000 0.960 0.780 0.950 0.960 0.910
slow 44.000 56.000 0.560 0.790 0.600 0.760 0.660
deep 2 98.000 0.980 0.800 0.980 0.970 0.910
fiddle 1 99.000 0.990 0.730 0.990 0.980 0.920
orchestral 2 98.000 0.980 0.700 0.980 0.980 0.710
mansinging 2 98.000 0.980 0.800 0.980 0.950 0.960
wind 1 99.000 0.990 0.810 0.980 0.980 0.830
piano 9 91.000 0.910 0.800 0.820 0.650 0.890
femalesinger 4 96.000 0.960 0.810 0.940 0.940 0.940
singing 13.000 87.000 0.870 0.890 0.870 0.870 0.900
quiet 16.000 84.000 0.840 0.780 0.850 0.860 0.820
tribal 1 99.000 0.990 0.900 0.990 0.980 1.000
noguitar 2 98.000 0.980 0.490 0.980 0.970 0.890
femalevocal 13.000 87.000 0.870 0.850 0.830 0.910 0.890
fastbeat 1 99.000 0.990 0.850 0.990 0.980 0.920
instrumental 4 96.000 0.960 0.470 0.950 0.930 0.860
chorus 3 97.000 0.970 0.950 0.970 0.940 0.960
silence 1 99.000 0.960 0.920 0.990 0.980 0.960
sax 2 98.000 0.970 0.890 0.970 0.980 0.930
nobeat 1 99.000 0.990 0.570 0.990 0.980 0.880
nopiano 3 97.000 0.910 0.430 0.970 0.960 0.840
novocals 15.000 85.000 0.850 0.560 0.850 0.790 0.790
low 5 95.000 0.950 0.770 0.950 0.940 0.830
weird 3 97.000 0.970 0.710 0.970 0.920 0.930
dance 4 96.000 0.960 0.950 0.940 0.950 0.940
harp 3 97.000 0.970 0.900 0.970 0.950 0.950
horns 3 97.000 0.970 0.850 0.970 0.950 0.950
funky 1 99.000 0.990 0.850 0.990 0.970 0.970
hardrock 4 96.000 0.960 0.930 0.960 0.950 0.890
bells 2 98.000 0.980 0.770 0.990 0.970 0.980
punk 4 96.000 0.900 0.950 0.990 0.960 0.920
techno 12.000 88.000 0.880 0.890 0.920 0.880 0.870
modern 5 95.000 0.950 0.680 0.960 0.940 0.930
violins 25.000 75.000 0.730 0.790 0.770 0.780 0.760
opera 6 94.000 0.940 0.940 0.910 0.860 0.940
cello 21.000 79.000 0.790 0.870 0.810 0.770 0.840
sitar 1 99.000 0.990 0.910 0.950 0.930 0.980
man 4 96.000 0.960 0.800 0.970 0.940 0.950
femalevocals 9 91.000 0.910 0.860 0.900 0.890 0.900
beat 13.000 87.000 0.870 0.810 0.870 0.900 0.880
vocal 22.000 78.000 0.780 0.860 0.760 0.890 0.810
jazz 3 97.000 0.970 0.920 0.960 0.930 0.920
male 10.000 90.000 0.900 0.820 0.870 0.900 0.880
drums 14.000 86.000 0.860 0.830 0.860 0.860 0.870
electronic 16.000 84.000 0.840 0.850 0.870 0.910 0.860
violin 44.000 56.000 0.560 0.900 0.840 0.850 0.770
bass 5 95.000 0.950 0.710 0.950 0.940 0.930
string 12.000 88.000 0.880 0.640 0.880 0.850 0.790
womansinging 3 97.000 0.970 0.810 0.940 0.950 0.960
guitar 15.000 85.000 0.850 0.840 0.900 0.630 0.870
medieval 5 95.000 0.950 0.780 0.950 0.950 0.830
old 1 99.000 0.990 0.620 0.990 0.990 0.690
middleeastern 1 99.000 0.950 0.590 0.990 0.990 0.950
baroque 7 93.000 0.830 0.790 0.930 0.900 0.680
oriental 1 99.000 0.990 0.680 0.970 0.990 0.990
trumpet 2 98.000 0.940 0.820 0.980 0.970 0.980
irish 1 99.000 0.990 0.800 0.980 0.980 0.930
ambient 14.000 86.000 0.860 0.880 0.860 0.890 0.720
funk 2 98.000 0.980 0.950 0.980 0.980 0.980
metal 5 95.000 0.790 0.960 0.960 0.900 0.900
woman 14.000 86.000 0.860 0.910 0.860 0.900 0.870
dark 1 99.000 0.990 0.760 0.990 0.990 0.790
acoustic 3 97.000 0.960 0.900 0.960 0.980 0.980
light 1 99.000 0.990 0.520 0.990 0.980 0.980
trance 4 96.000 0.960 0.870 0.960 0.950 0.930
celtic 1 99.000 0.990 0.670 0.990 0.980 0.920
electric 1 99.000 0.990 0.750 0.960 0.980 0.940
malevocals 6 94.000 0.890 0.780 0.940 0.890 0.860
heavy 4 96.000 0.930 0.880 0.970 0.940 0.890
jazzy 3 97.000 0.970 0.890 0.980 0.930 0.930
country 2 98.000 0.980 0.880 0.940 0.930 0.960
beats 4 96.000 0.950 0.790 0.930 0.960 0.880
loud 11.000 89.000 0.890 0.850 0.890 0.930 0.880
classical 48.000 52.000 0.520 0.850 0.500 0.790 0.810
voices 2 98.000 0.980 0.880 0.980 0.970 0.970
choral 7 93.000 0.930 0.940 0.920 0.900 0.920
harpsichord 12.000 88.000 0.880 0.890 0.870 0.930 0.740
eastern 1 99.000 0.990 0.800 0.990 0.970 0.990
foreign 2 98.000 0.980 0.840 0.960 0.960 0.960
fast 17.000 83.000 0.830 0.780 0.820 0.850 0.820
english 1 99.000 0.990 0.780 0.980 0.970 0.970
spacey 2 98.000 0.980 0.850 0.980 0.990 0.810
electro 4 96.000 0.960 0.800 0.950 0.960 0.910
calm 6 94.000 0.930 0.580 0.920 0.940 0.920
voice 10.000 90.000 0.900 0.850 0.880 0.880 0.880
vocals 16.000 84.000 0.840 0.790 0.840 0.810 0.840
singer 1 99.000 0.990 0.730 0.980 0.960 0.960
strings 47.000 53.000 0.530 0.910 0.550 0.880 0.820
orchestra 5 95.000 0.950 0.750 0.940 0.910 0.690
chant 4 96.000 0.960 0.950 0.960 0.970 0.980
heavymetal 2 98.000 0.980 0.930 0.970 0.980 0.910
girl 2 98.000 0.960 0.780 0.980 0.960 0.960
flute 8 92.000 0.920 0.900 0.940 0.870 0.880
drum 3 97.000 0.970 0.790 0.970 0.980 0.950
classic 24.000 76.000 0.760 0.790 0.770 0.760 0.720
nosinging 3 97.000 0.970 0.420 0.970 0.970 0.920
chanting 2 98.000 0.980 0.930 0.980 0.970 0.960
folk 2 98.000 0.980 0.830 0.980 0.950 0.970
malesinger 1 99.000 0.990 0.760 0.980 0.950 0.920
mellow 5 95.000 0.940 0.720 0.950 0.940 0.920
indian 3 97.000 0.970 0.890 0.960 0.850 0.970
electronica 2 98.000 0.980 0.720 0.980 0.960 0.920
women 3 97.000 0.970 0.870 0.970 0.980 0.960
soft 21.000 79.000 0.790 0.670 0.770 0.730 0.750
malevoice 9 91.000 0.910 0.830 0.890 0.910 0.880
organ 1 99.000 0.990 0.730 0.920 0.990 0.830
female 19.000 81.000 0.810 0.940 0.850 0.900 0.850
classicalguitar 1 99.000 0.990 0.980 0.990 0.980 0.980
airy 4 96.000 0.950 0.770 0.960 0.970 0.870
malevocal 11.000 89.000 0.890 0.850 0.880 0.910 0.850
clapping 2 98.000 0.980 0.870 0.980 0.970 0.980
choir 7 93.000 0.930 0.950 0.940 0.910 0.940

download these results as csv

Positive Example Accuracy

Full dataset

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
nostrings 13.000 6486.000 0.000 0.615 0.000 0.000 0.000
chimes 22.000 6477.000 0.045 0.545 0.000 0.045 0.136
sad 18.000 6481.000 0.111 0.778 0.056 0.000 0.667
nodrums 48.000 6451.000 0.000 0.562 0.000 0.000 0.208
femalevoice 105.000 6394.000 0.000 0.762 0.105 0.295 0.333
horn 7 6492.000 0.000 0.286 0.000 0.000 0.143
pop 196.000 6303.000 0.000 0.827 0.459 0.444 0.505
rock 601.000 5898.000 0.000 0.847 0.426 0.907 0.839
house 22.000 6477.000 0.000 0.773 0.045 0.000 0.273
birds 7 6492.000 0.000 0.429 0.000 0.143 0.714
harpsicord 59.000 6440.000 0.000 0.780 0.169 0.525 0.780
strange 22.000 6477.000 0.000 0.909 0.000 0.091 0.136
noflute 35.000 6464.000 0.000 0.314 0.000 0.000 0.086
novocal 263.000 6236.000 0.000 0.616 0.004 0.186 0.202
solo 217.000 6282.000 0.000 0.641 0.055 0.369 0.484
notenglish 11.000 6488.000 0.000 0.909 0.000 0.091 0.455
novoice 146.000 6353.000 0.000 0.589 0.007 0.089 0.151
newage 157.000 6342.000 0.000 0.803 0.000 0.280 0.592
synth 294.000 6205.000 0.000 0.786 0.051 0.422 0.554
upbeat 52.000 6447.000 0.000 0.808 0.038 0.096 0.442
slow 1043.000 5456.000 0.000 0.705 0.199 0.730 0.516
deep 12.000 6487.000 0.000 0.583 0.083 0.000 0.250
fiddle 14.000 6485.000 0.000 0.786 0.000 0.071 0.214
orchestral 12.000 6487.000 0.000 0.500 0.000 0.000 0.750
notclassical 14.000 6485.000 0.000 0.500 0.000 0.000 0.214
mansinging 46.000 6453.000 0.000 0.783 0.022 0.087 0.239
wind 22.000 6477.000 0.045 0.636 0.045 0.000 0.591
piano 630.000 5869.000 0.000 0.767 0.629 0.805 0.657
spanish 65.000 6434.000 0.000 0.462 0.015 0.077 0.169
femalesinger 30.000 6469.000 0.000 0.700 0.067 0.233 0.267
singing 242.000 6257.000 0.000 0.777 0.103 0.529 0.264
quiet 263.000 6236.000 0.000 0.719 0.030 0.707 0.620
oboe 12.000 6487.000 0.000 0.167 0.167 0.000 0.083
tribal 40.000 6459.000 0.000 0.425 0.025 0.200 0.200
noguitar 46.000 6453.000 0.000 0.565 0.000 0.022 0.304
femalevocal 126.000 6373.000 0.000 0.786 0.135 0.452 0.333
fastbeat 33.000 6466.000 0.000 0.636 0.000 0.000 0.515
hiphop 32.000 6467.000 0.000 0.594 0.219 0.000 0.312
instrumental 102.000 6397.000 0.000 0.598 0.029 0.088 0.294
chorus 50.000 6449.000 0.000 0.760 0.400 0.000 0.700
silence 12.000 6487.000 0.000 0.833 0.167 0.000 0.333
duet 18.000 6481.000 0.000 0.278 0.000 0.000 0.167
sax 20.000 6479.000 0.000 0.200 0.000 0.050 0.000
nobeat 14.000 6485.000 0.000 0.571 0.000 0.000 0.429
nopiano 90.000 6409.000 0.033 0.533 0.000 0.011 0.089
novocals 326.000 6173.000 0.000 0.610 0.003 0.190 0.206
pianosolo 13.000 6486.000 0.000 0.692 0.462 0.000 0.923
low 35.000 6464.000 0.000 0.686 0.143 0.057 0.371
weird 120.000 6379.000 0.000 0.783 0.025 0.250 0.267
dance 184.000 6315.000 0.000 0.864 0.283 0.234 0.712
harp 137.000 6362.000 0.000 0.431 0.058 0.277 0.409
horns 12.000 6487.000 0.000 0.167 0.083 0.000 0.083
funky 66.000 6433.000 0.000 0.879 0.121 0.000 0.485
hardrock 80.000 6419.000 0.000 0.963 0.100 0.000 0.950
bells 36.000 6463.000 0.000 0.528 0.028 0.083 0.111
punk 42.000 6457.000 0.000 0.786 0.595 0.000 0.810
electricguitar 51.000 6448.000 0.000 0.588 0.059 0.176 0.588
techno 827.000 5672.000 0.000 0.926 0.336 0.900 0.790
modern 73.000 6426.000 0.000 0.699 0.055 0.082 0.260
violins 258.000 6241.000 0.000 0.841 0.155 0.275 0.636
noviolin 18.000 6481.000 0.000 0.611 0.056 0.056 0.056
opera 325.000 6174.000 0.000 0.926 0.649 0.905 0.766
india 22.000 6477.000 0.000 0.864 0.045 0.045 0.318
cello 145.000 6354.000 0.000 0.717 0.414 0.510 0.366
sitar 250.000 6249.000 0.000 0.868 0.432 0.624 0.676
hard 25.000 6474.000 0.000 0.960 0.040 0.000 0.920
banjo 15.000 6484.000 0.000 0.133 0.067 0.067 0.333
blues 42.000 6457.000 0.000 0.643 0.119 0.190 0.048
man 128.000 6371.000 0.000 0.805 0.023 0.375 0.258
water 12.000 6487.000 0.000 0.667 0.000 0.000 0.750
femalevocals 90.000 6409.000 0.000 0.700 0.100 0.267 0.211
beat 534.000 5965.000 0.000 0.876 0.202 0.723 0.695
vocal 346.000 6153.000 0.000 0.777 0.075 0.601 0.234
jazz 88.000 6411.000 0.000 0.761 0.182 0.284 0.466
male 316.000 6183.000 0.000 0.826 0.161 0.525 0.332
maleopera 18.000 6481.000 0.000 0.889 0.278 0.000 0.889
drums 663.000 5836.000 0.000 0.839 0.207 0.691 0.514
electronic 578.000 5921.000 0.000 0.846 0.102 0.713 0.640
talking 27.000 6472.000 0.000 0.704 0.037 0.000 0.037
violin 908.000 5591.000 0.000 0.882 0.744 0.885 0.699
bass 73.000 6426.000 0.000 0.753 0.041 0.219 0.438
notrock 19.000 6480.000 0.000 0.368 0.000 0.053 0.105
string 91.000 6408.000 0.000 0.736 0.033 0.121 0.275
womansinging 32.000 6467.000 0.000 0.938 0.031 0.250 0.219
guitar 1166.000 5333.000 0.000 0.701 0.359 0.886 0.522
medieval 39.000 6460.000 0.000 0.795 0.077 0.026 0.410
clarinet 16.000 6483.000 0.000 0.625 0.000 0.000 0.375
world 14.000 6485.000 0.000 0.643 0.071 0.000 0.286
old 14.000 6485.000 0.000 0.786 0.000 0.071 0.500
middleeastern 17.000 6482.000 0.000 0.529 0.059 0.118 0.118
baroque 81.000 6418.000 0.111 0.864 0.012 0.333 0.840
oriental 50.000 6449.000 0.000 0.700 0.100 0.100 0.320
trumpet 17.000 6482.000 0.000 0.471 0.059 0.000 0.000
irish 49.000 6450.000 0.000 0.714 0.020 0.163 0.184
ambient 419.000 6080.000 0.000 0.788 0.014 0.726 0.644
funk 32.000 6467.000 0.000 0.875 0.125 0.000 0.344
metal 159.000 6340.000 0.019 0.899 0.126 0.000 0.969
woman 186.000 6313.000 0.000 0.801 0.091 0.565 0.457
dark 36.000 6463.000 0.000 0.861 0.028 0.000 0.361
acoustic 66.000 6433.000 0.015 0.682 0.182 0.121 0.409
light 16.000 6483.000 0.000 0.750 0.000 0.125 0.062
repetitive 24.000 6475.000 0.000 0.417 0.000 0.000 0.000
trance 51.000 6448.000 0.000 0.804 0.020 0.098 0.510
celtic 27.000 6472.000 0.000 0.741 0.000 0.111 0.074
electric 44.000 6455.000 0.000 0.659 0.000 0.023 0.205
malevocals 123.000 6376.000 0.130 0.821 0.154 0.276 0.382
heavy 59.000 6440.000 0.000 0.864 0.169 0.000 0.932
jazzy 68.000 6431.000 0.000 0.824 0.324 0.191 0.485
country 122.000 6377.000 0.000 0.697 0.328 0.344 0.189
beats 157.000 6342.000 0.006 0.866 0.140 0.344 0.707
loud 313.000 6186.000 0.000 0.799 0.096 0.645 0.764
classical 1544.000 4955.000 0.000 0.852 0.158 0.994 0.720
voices 39.000 6460.000 0.000 0.615 0.000 0.077 0.333
flutes 54.000 6445.000 0.000 0.815 0.463 0.000 0.759
choral 104.000 6395.000 0.000 0.846 0.202 0.442 0.817
harpsichord 263.000 6236.000 0.000 0.768 0.684 0.821 0.867
eastern 80.000 6419.000 0.000 0.787 0.100 0.375 0.400
foreign 51.000 6448.000 0.000 0.725 0.039 0.275 0.216
fast 616.000 5883.000 0.000 0.701 0.094 0.646 0.433
english 11.000 6488.000 0.000 0.364 0.000 0.000 0.091
spacey 27.000 6472.000 0.000 0.852 0.000 0.111 0.444
electro 87.000 6412.000 0.000 0.805 0.011 0.138 0.471
calm 33.000 6466.000 0.000 0.545 0.000 0.061 0.182
lute 15.000 6484.000 0.000 0.867 0.200 0.000 0.733
arabic 10.000 6489.000 0.000 0.300 0.100 0.000 0.000
voice 111.000 6388.000 0.000 0.802 0.063 0.234 0.171
vocals 256.000 6243.000 0.000 0.695 0.109 0.367 0.234
rap 41.000 6458.000 0.000 0.659 0.488 0.000 0.488
singer 25.000 6474.000 0.000 0.800 0.080 0.040 0.000
strings 997.000 5502.000 0.000 0.822 0.085 0.825 0.640
orchestra 98.000 6401.000 0.000 0.704 0.102 0.245 0.571
guitars 25.000 6474.000 0.000 0.400 0.000 0.000 0.400
chant 51.000 6448.000 0.000 0.745 0.157 0.627 0.647
heavymetal 43.000 6456.000 0.000 0.860 0.163 0.000 0.860
girl 10.000 6489.000 0.000 0.900 0.000 0.000 0.400
percussion 26.000 6473.000 0.077 0.692 0.000 0.077 0.308
flute 455.000 6044.000 0.000 0.807 0.569 0.732 0.631
drum 89.000 6410.000 0.000 0.798 0.056 0.180 0.281
classic 235.000 6264.000 0.000 0.881 0.102 0.340 0.604
nosinging 51.000 6448.000 0.000 0.569 0.000 0.059 0.059
chanting 32.000 6467.000 0.000 0.406 0.031 0.156 0.438
folk 48.000 6451.000 0.000 0.500 0.021 0.083 0.146
malesinger 39.000 6460.000 0.000 0.718 0.282 0.154 0.256
mellow 29.000 6470.000 0.000 0.483 0.000 0.000 0.069
indian 313.000 6186.000 0.000 0.770 0.137 0.594 0.278
electronica 39.000 6460.000 0.000 0.897 0.000 0.103 0.308
women 22.000 6477.000 0.000 0.818 0.045 0.182 0.409
notopera 19.000 6480.000 0.000 0.316 0.000 0.000 0.053
noise 16.000 6483.000 0.000 0.688 0.062 0.000 0.438
soft 248.000 6251.000 0.000 0.641 0.056 0.395 0.472
femaleopera 27.000 6472.000 0.000 0.926 0.259 0.000 0.889
malevoice 155.000 6344.000 0.000 0.806 0.071 0.297 0.252
organ 17.000 6482.000 0.000 0.294 0.059 0.059 0.235
female 320.000 6179.000 0.000 0.822 0.341 0.697 0.447
classicalguitar 38.000 6461.000 0.000 0.789 0.263 0.000 0.868
operatic 17.000 6482.000 0.000 0.941 0.000 0.000 0.824
airy 12.000 6487.000 0.083 0.833 0.000 0.083 0.750
malevocal 271.000 6228.000 0.000 0.856 0.277 0.465 0.373
clapping 12.000 6487.000 0.000 0.417 0.000 0.000 0.000
choir 161.000 6338.000 0.000 0.876 0.404 0.745 0.770

download these results as csv

100 query subset used in Tagatune evaluation

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
sad 5 95.000 0.200 0.800 0.200 0.000 0.600
nodrums 4 96.000 0.000 0.750 0.000 0.000 0.500
femalevoice 6 94.000 0.000 0.833 0.000 0.333 0.167
pop 10.000 90.000 0.000 0.900 0.600 0.700 0.700
rock 14.000 86.000 0.000 0.786 0.214 1.000 0.786
birds 1 99.000 0.000 1.000 0.000 0.000 0.000
harpsicord 3 97.000 0.000 1.000 0.000 0.000 1.000
strange 2 98.000 0.000 1.000 0.000 0.000 0.000
novocal 12.000 88.000 0.000 0.833 0.083 0.083 0.083
solo 11.000 89.000 0.000 0.364 0.091 0.182 0.182
notenglish 1 99.000 0.000 0.000 0.000 0.000 0.000
novoice 4 96.000 0.000 0.750 0.000 0.000 0.000
newage 12.000 88.000 0.000 0.667 0.000 0.250 0.500
synth 11.000 89.000 0.000 0.818 0.091 0.545 0.364
upbeat 4 96.000 0.000 0.750 0.000 0.000 0.500
slow 44.000 56.000 0.000 0.795 0.159 0.750 0.364
deep 2 98.000 0.000 0.500 0.000 0.000 0.000
fiddle 1 99.000 0.000 1.000 0.000 0.000 0.000
orchestral 2 98.000 0.000 0.500 0.000 0.000 1.000
mansinging 2 98.000 0.000 1.000 0.000 0.500 0.500
wind 1 99.000 0.000 1.000 0.000 0.000 1.000
piano 9 91.000 0.000 0.444 0.111 0.778 0.222
femalesinger 4 96.000 0.000 0.750 0.000 0.000 0.000
singing 13.000 87.000 0.000 0.846 0.154 0.462 0.385
quiet 16.000 84.000 0.000 0.688 0.062 0.688 0.500
tribal 1 99.000 0.000 1.000 0.000 0.000 1.000
noguitar 2 98.000 0.000 1.000 0.000 0.000 0.000
femalevocal 13.000 87.000 0.000 0.692 0.154 0.538 0.308
fastbeat 1 99.000 0.000 0.000 0.000 0.000 0.000
instrumental 4 96.000 0.000 1.000 0.000 0.000 0.250
chorus 3 97.000 0.000 0.667 0.000 0.000 0.667
silence 1 99.000 0.000 1.000 0.000 0.000 1.000
sax 2 98.000 0.000 0.000 0.000 0.000 0.000
nobeat 1 99.000 0.000 1.000 0.000 0.000 0.000
nopiano 3 97.000 0.000 1.000 0.000 0.333 0.000
novocals 15.000 85.000 0.000 0.800 0.000 0.200 0.067
low 5 95.000 0.000 0.600 0.000 0.000 0.000
weird 3 97.000 0.000 0.667 0.000 0.000 0.000
dance 4 96.000 0.000 1.000 0.250 0.500 0.750
harp 3 97.000 0.000 0.000 0.000 0.000 0.333
horns 3 97.000 0.000 0.000 0.000 0.000 0.000
funky 1 99.000 0.000 1.000 0.000 0.000 0.000
hardrock 4 96.000 0.000 1.000 0.000 0.000 1.000
bells 2 98.000 0.000 0.500 0.500 0.000 0.000
punk 4 96.000 0.000 1.000 0.750 0.000 1.000
techno 12.000 88.000 0.000 0.750 0.333 0.667 0.500
modern 5 95.000 0.000 0.600 0.200 0.200 0.400
violins 25.000 75.000 0.000 0.800 0.160 0.240 0.560
opera 6 94.000 0.000 0.667 0.167 1.000 0.667
cello 21.000 79.000 0.000 0.810 0.238 0.333 0.333
sitar 1 99.000 0.000 0.000 0.000 0.000 1.000
man 4 96.000 0.000 0.500 0.250 0.500 0.000
femalevocals 9 91.000 0.000 0.667 0.000 0.333 0.222
beat 13.000 87.000 0.000 0.615 0.000 0.462 0.462
vocal 22.000 78.000 0.000 0.773 0.000 0.636 0.227
jazz 3 97.000 0.000 0.667 0.000 0.000 0.667
male 10.000 90.000 0.000 0.700 0.100 0.600 0.000
drums 14.000 86.000 0.000 0.786 0.143 0.571 0.429
electronic 16.000 84.000 0.000 0.625 0.188 0.625 0.500
violin 44.000 56.000 0.000 0.886 0.727 0.864 0.568
bass 5 95.000 0.000 0.600 0.000 0.200 0.000
string 12.000 88.000 0.000 0.750 0.000 0.000 0.167
womansinging 3 97.000 0.000 1.000 0.000 0.000 0.000
guitar 15.000 85.000 0.000 0.533 0.333 0.800 0.267
medieval 5 95.000 0.000 0.800 0.000 0.000 0.000
old 1 99.000 0.000 1.000 0.000 0.000 1.000
middleeastern 1 99.000 0.000 0.000 0.000 0.000 0.000
baroque 7 93.000 0.000 0.571 0.000 0.000 0.714
oriental 1 99.000 0.000 0.000 0.000 0.000 0.000
trumpet 2 98.000 0.000 0.000 0.000 0.000 0.000
irish 1 99.000 0.000 1.000 0.000 0.000 1.000
ambient 14.000 86.000 0.000 0.714 0.000 0.786 0.500
funk 2 98.000 0.000 1.000 0.000 0.000 0.500
metal 5 95.000 0.400 1.000 0.200 0.000 1.000
woman 14.000 86.000 0.000 0.786 0.000 0.500 0.286
dark 1 99.000 0.000 0.000 0.000 0.000 0.000
acoustic 3 97.000 0.000 1.000 0.000 0.333 0.333
light 1 99.000 0.000 1.000 0.000 0.000 1.000
trance 4 96.000 0.000 0.750 0.000 0.000 0.750
celtic 1 99.000 0.000 1.000 0.000 0.000 0.000
electric 1 99.000 0.000 1.000 0.000 0.000 0.000
malevocals 6 94.000 0.000 0.333 0.167 0.333 0.500
heavy 4 96.000 0.000 0.500 0.250 0.000 0.750
jazzy 3 97.000 0.000 0.667 1.000 0.000 1.000
country 2 98.000 0.000 1.000 0.000 0.500 0.000
beats 4 96.000 0.000 0.250 0.000 0.000 0.000
loud 11.000 89.000 0.000 0.818 0.091 0.455 0.727
classical 48.000 52.000 0.000 0.812 0.021 1.000 0.667
voices 2 98.000 0.000 1.000 0.000 0.000 0.500
choral 7 93.000 0.000 0.429 0.000 0.000 0.286
harpsichord 12.000 88.000 0.000 0.750 0.417 0.833 0.917
eastern 1 99.000 0.000 1.000 0.000 0.000 1.000
foreign 2 98.000 0.000 0.500 0.000 0.000 0.000
fast 17.000 83.000 0.000 0.412 0.000 0.529 0.294
english 1 99.000 0.000 0.000 0.000 0.000 0.000
spacey 2 98.000 0.000 1.000 0.000 0.500 0.500
electro 4 96.000 0.000 0.750 0.000 0.000 0.500
calm 6 94.000 0.000 0.333 0.000 0.000 0.000
voice 10.000 90.000 0.000 0.900 0.000 0.300 0.100
vocals 16.000 84.000 0.000 0.688 0.188 0.375 0.125
singer 1 99.000 0.000 1.000 1.000 0.000 0.000
strings 47.000 53.000 0.000 0.915 0.085 0.872 0.681
orchestra 5 95.000 0.000 0.400 0.000 0.000 0.000
chant 4 96.000 0.000 0.500 0.000 0.250 0.500
heavymetal 2 98.000 0.000 1.000 0.000 0.000 1.000
girl 2 98.000 0.000 0.500 0.000 0.000 0.500
flute 8 92.000 0.000 0.500 0.375 0.375 0.125
drum 3 97.000 0.000 0.667 0.000 0.333 0.000
classic 24.000 76.000 0.000 0.958 0.083 0.292 0.500
nosinging 3 97.000 0.000 1.000 0.000 0.000 0.000
chanting 2 98.000 0.000 0.500 0.000 0.000 0.000
folk 2 98.000 0.000 0.500 0.000 0.000 0.000
malesinger 1 99.000 0.000 1.000 1.000 1.000 0.000
mellow 5 95.000 0.000 0.600 0.000 0.000 0.000
indian 3 97.000 0.000 0.333 0.000 0.000 0.333
electronica 2 98.000 0.000 1.000 0.000 0.000 0.500
women 3 97.000 0.000 0.667 0.000 0.333 0.333
soft 21.000 79.000 0.000 0.429 0.000 0.286 0.190
malevoice 9 91.000 0.000 0.778 0.000 0.444 0.111
organ 1 99.000 0.000 0.000 0.000 0.000 0.000
female 19.000 81.000 0.000 0.842 0.421 0.579 0.316
classicalguitar 1 99.000 0.000 1.000 0.000 0.000 1.000
airy 4 96.000 0.000 1.000 0.000 0.250 1.000
malevocal 11.000 89.000 0.000 0.818 0.273 0.545 0.273
clapping 2 98.000 0.000 0.000 0.000 0.000 0.000
choir 7 93.000 0.000 0.571 0.143 0.286 0.429

download these results as csv

Negative Example Accuracy

Full dataset

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
nostrings 13.000 6486.000 1.000 0.496 1.000 0.994 0.988
chimes 22.000 6477.000 0.989 0.779 0.998 0.995 0.984
sad 18.000 6481.000 0.896 0.705 0.998 0.992 0.863
nodrums 48.000 6451.000 1.000 0.569 0.998 0.984 0.828
femalevoice 105.000 6394.000 1.000 0.852 0.976 0.955 0.960
horn 7 6492.000 0.997 0.802 0.998 0.992 0.979
pop 196.000 6303.000 1.000 0.746 0.890 0.936 0.848
rock 601.000 5898.000 1.000 0.881 0.972 0.859 0.861
house 22.000 6477.000 1.000 0.793 0.992 0.995 0.941
birds 7 6492.000 1.000 0.922 0.998 0.992 0.924
harpsicord 59.000 6440.000 1.000 0.930 0.986 0.968 0.865
strange 22.000 6477.000 1.000 0.587 0.999 0.989 0.963
noflute 35.000 6464.000 0.993 0.560 0.999 0.987 0.943
novocal 263.000 6236.000 1.000 0.545 0.992 0.899 0.842
solo 217.000 6282.000 0.999 0.837 0.992 0.908 0.874
notenglish 11.000 6488.000 1.000 0.839 0.999 0.992 0.973
novoice 146.000 6353.000 1.000 0.568 0.994 0.954 0.882
newage 157.000 6342.000 1.000 0.758 0.998 0.952 0.787
synth 294.000 6205.000 1.000 0.692 0.991 0.896 0.800
upbeat 52.000 6447.000 1.000 0.692 0.987 0.987 0.887
slow 1043.000 5456.000 1.000 0.709 0.932 0.713 0.786
deep 12.000 6487.000 1.000 0.831 0.997 0.993 0.955
fiddle 14.000 6485.000 1.000 0.817 0.993 0.991 0.952
orchestral 12.000 6487.000 1.000 0.779 0.997 0.990 0.908
notclassical 14.000 6485.000 1.000 0.661 0.998 0.994 0.947
mansinging 46.000 6453.000 1.000 0.748 0.978 0.988 0.960
wind 22.000 6477.000 0.997 0.835 0.990 0.987 0.878
piano 630.000 5869.000 1.000 0.891 0.919 0.753 0.914
spanish 65.000 6434.000 1.000 0.829 0.983 0.985 0.962
femalesinger 30.000 6469.000 1.000 0.870 0.997 0.988 0.977
singing 242.000 6257.000 1.000 0.803 0.974 0.903 0.956
quiet 263.000 6236.000 1.000 0.795 0.996 0.898 0.821
oboe 12.000 6487.000 1.000 0.936 0.979 0.979 0.919
tribal 40.000 6459.000 1.000 0.772 0.998 0.980 0.977
noguitar 46.000 6453.000 1.000 0.560 0.999 0.979 0.920
femalevocal 126.000 6373.000 1.000 0.871 0.952 0.943 0.961
fastbeat 33.000 6466.000 1.000 0.785 0.996 0.994 0.916
hiphop 32.000 6467.000 1.000 0.907 0.996 0.996 0.981
instrumental 102.000 6397.000 0.992 0.598 0.980 0.964 0.827
chorus 50.000 6449.000 1.000 0.941 0.987 0.978 0.967
silence 12.000 6487.000 0.985 0.901 0.994 0.993 0.961
duet 18.000 6481.000 1.000 0.892 0.989 0.991 0.940
sax 20.000 6479.000 0.977 0.898 0.990 0.991 0.963
nobeat 14.000 6485.000 0.996 0.677 0.999 0.992 0.944
nopiano 90.000 6409.000 0.960 0.572 0.997 0.969 0.905
novocals 326.000 6173.000 1.000 0.543 0.998 0.891 0.846
pianosolo 13.000 6486.000 1.000 0.855 0.987 0.993 0.932
low 35.000 6464.000 1.000 0.817 0.990 0.986 0.875
weird 120.000 6379.000 1.000 0.638 0.993 0.959 0.926
dance 184.000 6315.000 1.000 0.822 0.960 0.960 0.893
harp 137.000 6362.000 1.000 0.895 0.993 0.947 0.893
horns 12.000 6487.000 0.991 0.937 0.993 0.990 0.977
funky 66.000 6433.000 1.000 0.773 0.981 0.989 0.921
hardrock 80.000 6419.000 1.000 0.893 0.992 0.980 0.886
bells 36.000 6463.000 1.000 0.732 0.995 0.986 0.977
punk 42.000 6457.000 0.971 0.928 0.962 0.993 0.924
electricguitar 51.000 6448.000 1.000 0.822 0.995 0.985 0.870
techno 827.000 5672.000 1.000 0.819 0.973 0.846 0.890
modern 73.000 6426.000 1.000 0.591 0.985 0.975 0.903
violins 258.000 6241.000 0.978 0.818 0.965 0.949 0.858
noviolin 18.000 6481.000 1.000 0.519 0.999 0.992 0.968
opera 325.000 6174.000 1.000 0.955 0.971 0.844 0.965
india 22.000 6477.000 1.000 0.770 0.999 0.994 0.992
cello 145.000 6354.000 1.000 0.952 0.962 0.922 0.968
sitar 250.000 6249.000 1.000 0.891 0.981 0.940 0.898
hard 25.000 6474.000 0.995 0.884 0.998 0.994 0.894
banjo 15.000 6484.000 1.000 0.961 0.996 0.979 0.943
blues 42.000 6457.000 1.000 0.923 0.992 0.987 0.995
man 128.000 6371.000 1.000 0.791 0.983 0.962 0.972
water 12.000 6487.000 1.000 0.911 0.996 0.992 0.897
femalevocals 90.000 6409.000 1.000 0.870 0.974 0.966 0.975
beat 534.000 5965.000 1.000 0.744 0.973 0.908 0.881
vocal 346.000 6153.000 1.000 0.785 0.978 0.861 0.954
jazz 88.000 6411.000 1.000 0.812 0.951 0.967 0.911
male 316.000 6183.000 1.000 0.821 0.974 0.914 0.952
maleopera 18.000 6481.000 1.000 0.966 0.993 0.991 0.967
drums 663.000 5836.000 1.000 0.700 0.937 0.816 0.853
electronic 578.000 5921.000 1.000 0.726 0.980 0.828 0.834
talking 27.000 6472.000 1.000 0.835 0.999 0.992 0.989
violin 908.000 5591.000 1.000 0.875 0.893 0.793 0.889
bass 73.000 6426.000 1.000 0.697 0.987 0.973 0.907
notrock 19.000 6480.000 1.000 0.494 0.999 0.994 0.985
string 91.000 6408.000 1.000 0.718 0.985 0.971 0.854
womansinging 32.000 6467.000 1.000 0.853 0.995 0.985 0.986
guitar 1166.000 5333.000 1.000 0.847 0.958 0.649 0.913
medieval 39.000 6460.000 1.000 0.791 0.994 0.981 0.877
clarinet 16.000 6483.000 1.000 0.894 0.995 0.991 0.952
world 14.000 6485.000 1.000 0.605 0.998 0.994 0.985
old 14.000 6485.000 1.000 0.713 0.997 0.995 0.821
middleeastern 17.000 6482.000 0.971 0.680 0.991 0.989 0.956
baroque 81.000 6418.000 0.863 0.842 0.992 0.962 0.795
oriental 50.000 6449.000 1.000 0.726 0.980 0.987 0.941
trumpet 17.000 6482.000 0.961 0.848 0.999 0.993 0.995
irish 49.000 6450.000 1.000 0.859 0.991 0.982 0.917
ambient 419.000 6080.000 1.000 0.872 0.999 0.867 0.825
funk 32.000 6467.000 1.000 0.870 0.991 0.995 0.947
metal 159.000 6340.000 0.862 0.912 0.994 0.963 0.885
woman 186.000 6313.000 1.000 0.892 0.981 0.925 0.962
dark 36.000 6463.000 1.000 0.798 0.997 0.984 0.859
acoustic 66.000 6433.000 0.984 0.844 0.974 0.988 0.947
light 16.000 6483.000 1.000 0.607 0.998 0.991 0.956
repetitive 24.000 6475.000 0.952 0.756 1.000 0.995 0.993
trance 51.000 6448.000 1.000 0.757 0.993 0.982 0.884
celtic 27.000 6472.000 1.000 0.751 0.996 0.991 0.967
electric 44.000 6455.000 1.000 0.610 0.978 0.982 0.937
malevocals 123.000 6376.000 0.945 0.775 0.973 0.965 0.881
heavy 59.000 6440.000 0.936 0.873 0.991 0.984 0.875
jazzy 68.000 6431.000 1.000 0.805 0.945 0.976 0.924
country 122.000 6377.000 1.000 0.883 0.936 0.956 0.966
beats 157.000 6342.000 0.991 0.718 0.972 0.962 0.885
loud 313.000 6186.000 1.000 0.828 0.990 0.937 0.846
classical 1544.000 4955.000 1.000 0.846 0.957 0.437 0.810
voices 39.000 6460.000 1.000 0.843 0.996 0.982 0.969
flutes 54.000 6445.000 1.000 0.905 0.986 0.984 0.959
choral 104.000 6395.000 1.000 0.954 0.996 0.967 0.974
harpsichord 263.000 6236.000 1.000 0.912 0.917 0.897 0.826
eastern 80.000 6419.000 1.000 0.735 0.989 0.967 0.927
foreign 51.000 6448.000 1.000 0.825 0.981 0.975 0.986
fast 616.000 5883.000 1.000 0.725 0.987 0.820 0.838
english 11.000 6488.000 1.000 0.768 0.993 0.992 0.979
spacey 27.000 6472.000 1.000 0.857 0.998 0.987 0.894
electro 87.000 6412.000 0.998 0.684 0.993 0.968 0.879
calm 33.000 6466.000 0.989 0.653 0.994 0.989 0.925
lute 15.000 6484.000 1.000 0.922 0.992 0.987 0.937
arabic 10.000 6489.000 1.000 0.733 0.985 0.990 0.973
voice 111.000 6388.000 1.000 0.707 0.980 0.951 0.951
vocals 256.000 6243.000 1.000 0.745 0.939 0.907 0.948
rap 41.000 6458.000 1.000 0.935 0.981 0.992 0.989
singer 25.000 6474.000 1.000 0.736 0.983 0.989 0.989
strings 997.000 5502.000 1.000 0.790 0.978 0.744 0.794
orchestra 98.000 6401.000 1.000 0.843 0.988 0.955 0.865
guitars 25.000 6474.000 1.000 0.731 0.996 0.994 0.921
chant 51.000 6448.000 1.000 0.952 0.996 0.984 0.974
heavymetal 43.000 6456.000 1.000 0.905 0.988 0.993 0.919
girl 10.000 6489.000 0.978 0.755 0.999 0.994 0.972
percussion 26.000 6473.000 0.865 0.729 0.997 0.985 0.964
flute 455.000 6044.000 1.000 0.936 0.973 0.901 0.923
drum 89.000 6410.000 1.000 0.688 0.989 0.971 0.940
classic 235.000 6264.000 1.000 0.756 0.969 0.908 0.815
nosinging 51.000 6448.000 1.000 0.529 0.995 0.984 0.933
chanting 32.000 6467.000 1.000 0.948 0.999 0.988 0.972
folk 48.000 6451.000 1.000 0.805 0.991 0.980 0.974
malesinger 39.000 6460.000 1.000 0.748 0.931 0.985 0.946
mellow 29.000 6470.000 0.996 0.650 0.997 0.992 0.962
indian 313.000 6186.000 1.000 0.815 0.983 0.892 0.960
electronica 39.000 6460.000 1.000 0.627 0.993 0.984 0.925
women 22.000 6477.000 1.000 0.887 0.998 0.993 0.983
notopera 19.000 6480.000 1.000 0.559 1.000 0.994 0.992
noise 16.000 6483.000 1.000 0.793 0.998 0.984 0.902
soft 248.000 6251.000 1.000 0.758 0.983 0.908 0.837
femaleopera 27.000 6472.000 1.000 0.936 0.988 0.984 0.949
malevoice 155.000 6344.000 1.000 0.765 0.970 0.956 0.950
organ 17.000 6482.000 1.000 0.787 0.954 0.975 0.868
female 320.000 6179.000 1.000 0.909 0.925 0.864 0.961
classicalguitar 38.000 6461.000 1.000 0.959 0.978 0.980 0.932
operatic 17.000 6482.000 1.000 0.866 0.999 0.992 0.943
airy 12.000 6487.000 0.990 0.762 0.998 0.994 0.881
malevocal 271.000 6228.000 1.000 0.808 0.935 0.931 0.911
clapping 12.000 6487.000 1.000 0.816 0.999 0.994 0.998
choir 161.000 6338.000 1.000 0.960 0.998 0.956 0.979

download these results as csv

100 query subset used in Tagatune evaluation

Tag Positive Examples Negative Examples LabX Mandel Manzagol Marsyas Zhi
sad 5 95.000 0.895 0.589 1.000 1.000 0.789
nodrums 4 96.000 1.000 0.458 1.000 0.990 0.781
femalevoice 6 94.000 1.000 0.840 0.957 0.947 0.936
pop 10.000 90.000 1.000 0.778 0.922 0.967 0.878
rock 14.000 86.000 1.000 0.930 0.988 0.872 0.919
birds 1 99.000 1.000 0.919 1.000 1.000 0.838
harpsicord 3 97.000 1.000 0.907 0.979 1.000 0.753
strange 2 98.000 1.000 0.561 1.000 0.980 0.949
novocal 12.000 88.000 1.000 0.557 1.000 0.920 0.875
solo 11.000 89.000 1.000 0.843 0.989 0.955 0.933
notenglish 1 99.000 1.000 0.828 1.000 1.000 0.939
novoice 4 96.000 1.000 0.635 1.000 0.979 0.969
newage 12.000 88.000 1.000 0.795 1.000 1.000 0.795
synth 11.000 89.000 1.000 0.831 1.000 0.989 0.899
upbeat 4 96.000 1.000 0.781 0.990 1.000 0.927
slow 44.000 56.000 1.000 0.786 0.946 0.768 0.893
deep 2 98.000 1.000 0.806 1.000 0.990 0.929
fiddle 1 99.000 1.000 0.727 1.000 0.990 0.929
orchestral 2 98.000 1.000 0.704 1.000 1.000 0.704
mansinging 2 98.000 1.000 0.796 1.000 0.959 0.969
wind 1 99.000 1.000 0.808 0.990 0.990 0.828
piano 9 91.000 1.000 0.835 0.890 0.637 0.956
femalesinger 4 96.000 1.000 0.812 0.979 0.979 0.979
singing 13.000 87.000 1.000 0.897 0.977 0.931 0.977
quiet 16.000 84.000 1.000 0.798 1.000 0.893 0.881
tribal 1 99.000 1.000 0.899 1.000 0.990 1.000
noguitar 2 98.000 1.000 0.480 1.000 0.990 0.908
femalevocal 13.000 87.000 1.000 0.874 0.931 0.966 0.977
fastbeat 1 99.000 1.000 0.859 1.000 0.990 0.929
instrumental 4 96.000 1.000 0.448 0.990 0.969 0.885
chorus 3 97.000 1.000 0.959 1.000 0.969 0.969
silence 1 99.000 0.970 0.919 1.000 0.990 0.960
sax 2 98.000 0.990 0.908 0.990 1.000 0.949
nobeat 1 99.000 1.000 0.566 1.000 0.990 0.889
nopiano 3 97.000 0.938 0.412 1.000 0.979 0.866
novocals 15.000 85.000 1.000 0.518 1.000 0.894 0.918
low 5 95.000 1.000 0.779 1.000 0.989 0.874
weird 3 97.000 1.000 0.711 1.000 0.948 0.959
dance 4 96.000 1.000 0.948 0.969 0.969 0.948
harp 3 97.000 1.000 0.928 1.000 0.979 0.969
horns 3 97.000 1.000 0.876 1.000 0.979 0.979
funky 1 99.000 1.000 0.848 1.000 0.980 0.980
hardrock 4 96.000 1.000 0.927 1.000 0.990 0.885
bells 2 98.000 1.000 0.776 1.000 0.990 1.000
punk 4 96.000 0.938 0.948 1.000 1.000 0.917
techno 12.000 88.000 1.000 0.909 1.000 0.909 0.920
modern 5 95.000 1.000 0.684 1.000 0.979 0.958
violins 25.000 75.000 0.973 0.787 0.973 0.960 0.827
opera 6 94.000 1.000 0.957 0.957 0.851 0.957
cello 21.000 79.000 1.000 0.886 0.962 0.886 0.975
sitar 1 99.000 1.000 0.919 0.960 0.939 0.980
man 4 96.000 1.000 0.812 1.000 0.958 0.990
femalevocals 9 91.000 1.000 0.879 0.989 0.945 0.967
beat 13.000 87.000 1.000 0.839 1.000 0.966 0.943
vocal 22.000 78.000 1.000 0.885 0.974 0.962 0.974
jazz 3 97.000 1.000 0.928 0.990 0.959 0.928
male 10.000 90.000 1.000 0.833 0.956 0.933 0.978
drums 14.000 86.000 1.000 0.837 0.977 0.907 0.942
electronic 16.000 84.000 1.000 0.893 1.000 0.964 0.929
violin 44.000 56.000 1.000 0.911 0.929 0.839 0.929
bass 5 95.000 1.000 0.716 1.000 0.979 0.979
string 12.000 88.000 1.000 0.625 1.000 0.966 0.875
womansinging 3 97.000 1.000 0.804 0.969 0.979 0.990
guitar 15.000 85.000 1.000 0.894 1.000 0.600 0.976
medieval 5 95.000 1.000 0.779 1.000 1.000 0.874
old 1 99.000 1.000 0.616 1.000 1.000 0.687
middleeastern 1 99.000 0.960 0.596 1.000 1.000 0.960
baroque 7 93.000 0.892 0.806 1.000 0.968 0.677
oriental 1 99.000 1.000 0.687 0.980 1.000 1.000
trumpet 2 98.000 0.959 0.837 1.000 0.990 1.000
irish 1 99.000 1.000 0.798 0.990 0.990 0.929
ambient 14.000 86.000 1.000 0.907 1.000 0.907 0.756
funk 2 98.000 1.000 0.949 1.000 1.000 0.990
metal 5 95.000 0.811 0.958 1.000 0.947 0.895
woman 14.000 86.000 1.000 0.930 1.000 0.965 0.965
dark 1 99.000 1.000 0.768 1.000 1.000 0.798
acoustic 3 97.000 0.990 0.897 0.990 1.000 1.000
light 1 99.000 1.000 0.515 1.000 0.990 0.980
trance 4 96.000 1.000 0.875 1.000 0.990 0.938
celtic 1 99.000 1.000 0.667 1.000 0.990 0.929
electric 1 99.000 1.000 0.747 0.970 0.990 0.949
malevocals 6 94.000 0.947 0.809 0.989 0.926 0.883
heavy 4 96.000 0.969 0.896 1.000 0.979 0.896
jazzy 3 97.000 1.000 0.897 0.979 0.959 0.928
country 2 98.000 1.000 0.878 0.959 0.939 0.980
beats 4 96.000 0.990 0.812 0.969 1.000 0.917
loud 11.000 89.000 1.000 0.854 0.989 0.989 0.899
classical 48.000 52.000 1.000 0.885 0.942 0.596 0.942
voices 2 98.000 1.000 0.878 1.000 0.990 0.980
choral 7 93.000 1.000 0.978 0.989 0.968 0.968
harpsichord 12.000 88.000 1.000 0.909 0.932 0.943 0.716
eastern 1 99.000 1.000 0.798 1.000 0.980 0.990
foreign 2 98.000 1.000 0.847 0.980 0.980 0.980
fast 17.000 83.000 1.000 0.855 0.988 0.916 0.928
english 1 99.000 1.000 0.788 0.990 0.980 0.980
spacey 2 98.000 1.000 0.847 1.000 1.000 0.816
electro 4 96.000 1.000 0.802 0.990 1.000 0.927
calm 6 94.000 0.989 0.596 0.979 1.000 0.979
voice 10.000 90.000 1.000 0.844 0.978 0.944 0.967
vocals 16.000 84.000 1.000 0.810 0.964 0.893 0.976
singer 1 99.000 1.000 0.727 0.980 0.970 0.970
strings 47.000 53.000 1.000 0.906 0.962 0.887 0.943
orchestra 5 95.000 1.000 0.768 0.989 0.958 0.726
chant 4 96.000 1.000 0.969 1.000 1.000 1.000
heavymetal 2 98.000 1.000 0.929 0.990 1.000 0.908
girl 2 98.000 0.980 0.786 1.000 0.980 0.969
flute 8 92.000 1.000 0.935 0.989 0.913 0.946
drum 3 97.000 1.000 0.794 1.000 1.000 0.979
classic 24.000 76.000 1.000 0.737 0.987 0.908 0.789
nosinging 3 97.000 1.000 0.402 1.000 1.000 0.948
chanting 2 98.000 1.000 0.939 1.000 0.990 0.980
folk 2 98.000 1.000 0.837 1.000 0.969 0.990
malesinger 1 99.000 1.000 0.758 0.980 0.949 0.929
mellow 5 95.000 0.989 0.726 1.000 0.989 0.968
indian 3 97.000 1.000 0.907 0.990 0.876 0.990
electronica 2 98.000 1.000 0.714 1.000 0.980 0.929
women 3 97.000 1.000 0.876 1.000 1.000 0.979
soft 21.000 79.000 1.000 0.734 0.975 0.848 0.899
malevoice 9 91.000 1.000 0.835 0.978 0.956 0.956
organ 1 99.000 1.000 0.737 0.929 1.000 0.838
female 19.000 81.000 1.000 0.963 0.951 0.975 0.975
classicalguitar 1 99.000 1.000 0.980 1.000 0.990 0.980
airy 4 96.000 0.990 0.760 1.000 1.000 0.865
malevocal 11.000 89.000 1.000 0.854 0.955 0.955 0.921
clapping 2 98.000 1.000 0.888 1.000 0.990 1.000
choir 7 93.000 1.000 0.978 1.000 0.957 0.978

download these results as csv


Overall Summary Results (MIREX Statistical evaluation - Affinity)

Full dataset

Measure Mandel Manzagol Marsyas Zhi
Average AUC-ROC Tag 0.821 0.750 0.831 0.673
Average AUC-ROC Clip 0.886 0.810 0.933 0.748
Precision at 3 0.323 0.255 0.440 0.224
Precision at 6 0.245 0.194 0.314 0.192
Precision at 9 0.197 0.159 0.244 0.168
Precision at 12 0.167 0.136 0.201 0.146
Precision at 15 0.145 0.119 0.172 0.127

download these results as csv

100 query subset used in Tagatune evaluation

Measure Mandel Manzagol Marsyas Zhi
Average AUC-ROC Tag 0.646 0.566 0.636 0.499
Average AUC-ROC Clip 0.873 0.766 0.916 0.689
Precision at 3 0.613 0.477 0.743 0.363
Precision at 6 0.508 0.383 0.590 0.338
Precision at 9 0.434 0.322 0.489 0.310
Precision at 12 0.388 0.283 0.431 0.278
Precision at 15 0.339 0.252 0.383 0.248

download these results as csv


AUC-ROC Tag

Full dataset

Tag Mandel Manzagol Marsyas Zhi
nostrings 0.552 0.599 0.663 0.494
chimes 0.732 0.749 0.822 0.561
sad 0.842 0.662 0.816 0.785
nodrums 0.600 0.549 0.583 0.517
femalevoice 0.894 0.731 0.786 0.648
horn 0.435 0.683 0.760 0.560
pop 0.872 0.819 0.891 0.687
rock 0.947 0.926 0.957 0.879
house 0.870 0.651 0.949 0.607
birds 0.819 0.653 0.820 0.823
harpsicord 0.946 0.944 0.961 0.855
strange 0.801 0.736 0.744 0.550
noflute 0.454 0.542 0.532 0.516
novocal 0.617 0.538 0.626 0.519
solo 0.843 0.746 0.805 0.686
notenglish 0.861 0.587 0.822 0.717
novoice 0.623 0.538 0.578 0.517
newage 0.856 0.706 0.811 0.697
synth 0.816 0.728 0.778 0.690
upbeat 0.814 0.702 0.841 0.674
slow 0.778 0.733 0.797 0.662
deep 0.873 0.835 0.904 0.602
fiddle 0.895 0.825 0.844 0.585
orchestral 0.782 0.718 0.796 0.841
notclassical 0.605 0.498 0.752 0.578
mansinging 0.858 0.698 0.886 0.600
wind 0.855 0.779 0.866 0.752
piano 0.913 0.861 0.875 0.797
spanish 0.762 0.699 0.759 0.566
femalesinger 0.864 0.668 0.831 0.623
singing 0.866 0.720 0.847 0.611
quiet 0.854 0.777 0.906 0.736
oboe 0.806 0.869 0.863 0.504
tribal 0.732 0.737 0.761 0.588
noguitar 0.611 0.534 0.579 0.615
femalevocal 0.896 0.739 0.860 0.648
fastbeat 0.827 0.855 0.908 0.728
hiphop 0.901 0.856 0.912 0.647
instrumental 0.630 0.561 0.704 0.563
chorus 0.938 0.915 0.932 0.836
silence 0.927 0.936 0.988 0.648
duet 0.707 0.681 0.726 0.553
sax 0.683 0.666 0.663 0.482
nobeat 0.687 0.549 0.651 0.684
nopiano 0.559 0.480 0.542 0.496
novocals 0.613 0.538 0.640 0.526
pianosolo 0.906 0.970 0.959 0.937
low 0.830 0.814 0.887 0.629
weird 0.788 0.573 0.790 0.599
dance 0.910 0.850 0.925 0.813
harp 0.814 0.846 0.824 0.650
horns 0.637 0.659 0.524 0.530
funky 0.904 0.860 0.897 0.703
hardrock 0.976 0.940 0.969 0.948
bells 0.654 0.678 0.799 0.544
punk 0.939 0.958 0.958 0.879
electricguitar 0.834 0.790 0.836 0.734
techno 0.942 0.899 0.940 0.856
modern 0.725 0.642 0.772 0.582
violins 0.890 0.848 0.873 0.753
noviolin 0.521 0.637 0.697 0.512
opera 0.972 0.962 0.957 0.870
india 0.907 0.837 0.836 0.655
cello 0.930 0.882 0.900 0.669
sitar 0.940 0.902 0.896 0.793
hard 0.966 0.910 0.960 0.932
banjo 0.676 0.689 0.799 0.638
blues 0.909 0.894 0.894 0.521
man 0.886 0.739 0.883 0.616
water 0.896 0.634 0.956 0.851
femalevocals 0.891 0.780 0.777 0.593
beat 0.890 0.873 0.915 0.799
vocal 0.855 0.729 0.825 0.595
jazz 0.878 0.782 0.847 0.691
male 0.897 0.781 0.882 0.644
maleopera 0.982 0.923 0.992 0.935
drums 0.836 0.801 0.841 0.686
electronic 0.860 0.797 0.847 0.750
talking 0.875 0.718 0.933 0.513
violin 0.943 0.899 0.910 0.807
bass 0.804 0.779 0.827 0.677
notrock 0.510 0.568 0.603 0.546
string 0.795 0.682 0.768 0.567
womansinging 0.941 0.716 0.900 0.603
guitar 0.870 0.838 0.879 0.721
medieval 0.845 0.771 0.734 0.642
clarinet 0.886 0.753 0.920 0.666
world 0.600 0.565 0.786 0.636
old 0.744 0.644 0.720 0.679
middleeastern 0.668 0.604 0.693 0.536
baroque 0.925 0.847 0.877 0.840
oriental 0.792 0.798 0.826 0.628
trumpet 0.785 0.661 0.688 0.498
irish 0.861 0.792 0.783 0.551
ambient 0.916 0.761 0.873 0.761
funk 0.933 0.888 0.939 0.643
metal 0.971 0.954 0.974 0.962
woman 0.927 0.815 0.872 0.710
dark 0.897 0.794 0.874 0.613
acoustic 0.839 0.834 0.908 0.678
light 0.737 0.654 0.793 0.510
repetitive 0.655 0.497 0.865 0.496
trance 0.861 0.699 0.877 0.702
celtic 0.826 0.758 0.776 0.520
electric 0.709 0.527 0.753 0.572
malevocals 0.869 0.774 0.862 0.635
heavy 0.955 0.911 0.969 0.932
jazzy 0.881 0.801 0.849 0.709
country 0.876 0.771 0.883 0.577
beats 0.867 0.814 0.909 0.805
loud 0.893 0.817 0.927 0.831
classical 0.920 0.849 0.885 0.778
voices 0.803 0.691 0.806 0.651
flutes 0.943 0.906 0.939 0.865
choral 0.966 0.935 0.957 0.900
harpsichord 0.940 0.911 0.940 0.891
eastern 0.845 0.810 0.842 0.667
foreign 0.845 0.644 0.873 0.600
fast 0.802 0.758 0.815 0.642
english 0.664 0.519 0.811 0.536
spacey 0.907 0.759 0.886 0.680
electro 0.821 0.695 0.838 0.675
calm 0.668 0.601 0.820 0.554
lute 0.955 0.936 0.934 0.843
arabic 0.534 0.656 0.682 0.487
voice 0.819 0.620 0.780 0.562
vocals 0.812 0.676 0.791 0.591
rap 0.934 0.925 0.946 0.739
singer 0.811 0.642 0.838 0.494
strings 0.879 0.832 0.857 0.732
orchestra 0.883 0.797 0.867 0.725
guitars 0.659 0.690 0.817 0.664
chant 0.912 0.851 0.952 0.815
heavymetal 0.953 0.931 0.972 0.904
girl 0.919 0.701 0.806 0.685
percussion 0.778 0.850 0.850 0.638
flute 0.953 0.907 0.907 0.789
drum 0.814 0.735 0.821 0.611
classic 0.872 0.797 0.840 0.711
nosinging 0.566 0.526 0.657 0.495
chanting 0.891 0.657 0.878 0.706
folk 0.758 0.797 0.774 0.560
malesinger 0.835 0.742 0.883 0.603
mellow 0.647 0.632 0.787 0.515
indian 0.886 0.794 0.831 0.621
electronica 0.802 0.685 0.823 0.621
women 0.947 0.812 0.878 0.697
notopera 0.510 0.482 0.550 0.522
noise 0.817 0.737 0.759 0.676
soft 0.773 0.718 0.809 0.662
femaleopera 0.977 0.960 0.934 0.926
malevoice 0.870 0.660 0.869 0.602
organ 0.640 0.457 0.597 0.557
female 0.928 0.813 0.851 0.706
classicalguitar 0.963 0.949 0.926 0.909
operatic 0.948 0.862 0.913 0.892
airy 0.885 0.696 0.911 0.843
malevocal 0.895 0.761 0.892 0.645
clapping 0.798 0.575 0.628 0.499
choir 0.979 0.962 0.962 0.880

download these results as csv

100 query subset used in Tagatune evaluation

Tag Mandel Manzagol Marsyas Zhi
nostrings 0.000 0.000 0.000 0.000
chimes 0.000 0.000 0.000 0.000
sad 0.762 0.760 0.672 0.703
nodrums 0.609 0.320 0.622 0.535
femalevoice 0.885 0.704 0.910 0.556
horn 0.000 0.000 0.000 0.000
pop 0.931 0.892 0.969 0.752
rock 0.966 0.934 0.968 0.872
house 0.000 0.000 0.000 0.000
birds 0.939 0.980 0.606 0.424
harpsicord 0.935 0.887 0.914 0.880
strange 0.791 0.449 0.816 0.480
noflute 0.000 0.000 0.000 0.000
novocal 0.696 0.606 0.634 0.481
solo 0.796 0.612 0.776 0.568
notenglish 0.000 0.313 0.596 0.475
novoice 0.784 0.604 0.703 0.490
newage 0.813 0.665 0.863 0.633
synth 0.921 0.774 0.860 0.645
upbeat 0.888 0.789 0.828 0.724
slow 0.856 0.679 0.848 0.637
deep 0.755 0.959 0.852 0.469
fiddle 0.818 0.768 0.818 0.470
orchestral 0.709 0.796 0.806 0.908
notclassical 0.000 0.000 0.000 0.000
mansinging 0.980 0.816 0.969 0.745
wind 0.939 0.747 0.980 0.970
piano 0.819 0.568 0.683 0.531
spanish 0.000 0.000 0.000 0.000
femalesinger 0.794 0.581 0.839 0.495
singing 0.939 0.688 0.846 0.641
quiet 0.844 0.735 0.924 0.659
oboe 0.000 0.000 0.000 0.000
tribal 0.980 1.000 0.980 0.500
noguitar 0.699 0.480 0.189 0.459
femalevocal 0.836 0.753 0.925 0.648
fastbeat 0.737 0.485 0.838 0.470
hiphop 0.000 0.000 0.000 0.000
instrumental 0.846 0.622 0.643 0.578
chorus 0.900 0.852 0.969 0.820
silence 0.949 0.960 0.929 0.970
duet 0.000 0.000 0.000 0.000
sax 0.122 0.592 0.122 0.480
nobeat 0.778 0.141 0.828 0.449
nopiano 0.773 0.368 0.560 0.438
novocals 0.632 0.589 0.707 0.498
pianosolo 0.000 0.000 0.000 0.000
low 0.737 0.667 0.737 0.442
weird 0.746 0.333 0.773 0.485
dance 0.974 0.906 0.979 0.852
harp 0.663 0.746 0.928 0.653
horns 0.588 0.529 0.282 0.495
funky 1.000 0.980 0.939 0.495
hardrock 1.000 0.943 1.000 0.992
bells 0.536 0.857 0.393 0.500
punk 0.987 0.997 0.992 0.982
electricguitar 0.000 0.000 0.000 0.000
techno 0.922 0.852 0.891 0.713
modern 0.796 0.931 0.813 0.684
violins 0.890 0.864 0.890 0.676
noviolin 0.000 0.000 0.000 0.000
opera 0.846 0.911 0.943 0.821
india 0.000 0.000 0.000 0.000
cello 0.914 0.838 0.866 0.662
sitar 0.859 0.586 0.818 1.000
hard 0.000 0.000 0.000 0.000
banjo 0.000 0.000 0.000 0.000
blues 0.000 0.000 0.000 0.000
man 0.862 0.815 0.880 0.500
water 0.000 0.000 0.000 0.000
femalevocals 0.834 0.803 0.823 0.599
beat 0.877 0.887 0.947 0.660
vocal 0.904 0.722 0.859 0.578
jazz 0.907 0.790 0.883 0.799
male 0.909 0.723 0.926 0.494
maleopera 0.000 0.000 0.000 0.000
drums 0.894 0.868 0.866 0.686
electronic 0.890 0.794 0.846 0.689
talking 0.000 0.000 0.000 0.000
violin 0.958 0.903 0.932 0.740
bass 0.758 0.731 0.792 0.495
notrock 0.000 0.000 0.000 0.000
string 0.809 0.668 0.747 0.526
womansinging 0.911 0.897 0.914 0.500
guitar 0.864 0.800 0.807 0.587
medieval 0.745 0.752 0.501 0.442
clarinet 0.000 0.000 0.000 0.000
world 0.000 0.000 0.000 0.000
old 0.667 0.586 0.475 0.869
middleeastern 0.152 0.101 0.162 0.485
baroque 0.829 0.631 0.747 0.682
oriental 0.545 0.232 0.556 0.500
trumpet 0.643 0.648 0.372 0.500
irish 0.949 0.838 0.697 0.939
ambient 0.898 0.649 0.895 0.678
funk 0.995 0.939 0.974 0.495
metal 0.998 1.000 0.996 0.983
woman 0.857 0.841 0.856 0.588
dark 0.657 0.354 0.596 0.404
acoustic 0.948 0.983 0.997 0.500
light 0.929 0.677 0.697 1.000
repetitive 0.000 0.000 0.000 0.000
trance 0.909 0.867 0.990 0.845
celtic 1.000 0.919 0.909 0.470
electric 0.899 1.000 0.798 0.480
malevocals 0.814 0.652 0.881 0.695
heavy 0.943 0.932 0.943 0.855
jazzy 0.948 0.983 0.904 0.797
country 0.923 0.556 0.918 0.495
beats 0.792 0.698 0.776 0.464
loud 0.944 0.876 0.941 0.833
classical 0.945 0.802 0.898 0.787
voices 0.964 0.842 0.959 0.747
flutes 0.000 0.000 0.000 0.000
choral 0.862 0.647 0.846 0.631
harpsichord 0.926 0.747 0.929 0.897
eastern 0.909 1.000 0.687 1.000
foreign 0.449 0.347 0.796 0.495
fast 0.780 0.670 0.778 0.617
english 0.364 0.798 0.838 0.495
spacey 0.959 0.724 0.990 0.681
electro 0.844 0.651 0.927 0.590
calm 0.441 0.651 0.784 0.495
lute 0.000 0.000 0.000 0.000
arabic 0.000 0.000 0.000 0.000
voice 0.933 0.700 0.802 0.540
vocals 0.846 0.649 0.775 0.519
rap 0.000 0.000 0.000 0.000
singer 0.909 1.000 0.768 0.490
strings 0.972 0.902 0.933 0.814
orchestra 0.813 0.823 0.764 0.368
guitars 0.000 0.000 0.000 0.000
chant 0.815 0.721 0.974 0.625
heavymetal 0.995 0.867 0.990 0.995
girl 0.872 0.699 0.923 0.735
percussion 0.000 0.000 0.000 0.000
flute 0.807 0.802 0.837 0.542
drum 0.869 0.869 0.955 0.495
classic 0.880 0.776 0.815 0.635
nosinging 0.729 0.347 0.677 0.479
chanting 0.719 0.444 0.969 0.495
folk 0.673 0.867 0.536 0.500
malesinger 0.778 0.949 0.990 0.470
mellow 0.629 0.707 0.693 0.489
indian 0.801 0.419 0.481 0.667
electronica 0.852 0.526 0.913 0.735
women 0.928 0.739 0.845 0.660
notopera 0.000 0.000 0.000 0.000
noise 0.000 0.000 0.000 0.000
soft 0.661 0.667 0.761 0.548
femaleopera 0.000 0.000 0.000 0.000
malevoice 0.906 0.618 0.882 0.478
organ 0.707 0.101 0.525 0.424
female 0.910 0.712 0.827 0.620
classicalguitar 1.000 0.970 1.000 0.990
operatic 0.000 0.000 0.000 0.000
airy 0.995 0.693 0.982 0.992
malevocal 0.883 0.700 0.916 0.600
clapping 0.867 0.270 0.949 0.500
choir 0.892 0.688 0.848 0.710

download these results as csv


Select Friedman's Test Results

Tag F-measure (Binary) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the F-measure for each tag in the test, averaged over all folds.

Full dataset

TeamID TeamID Lowerbound Mean Upperbound Significance
Zhi Mandel -0.196 0.275 0.746 FALSE
Zhi Marsyas -0.102 0.369 0.840 FALSE
Zhi Manzagol 0.657 1.128 1.599 TRUE
Zhi LabX 2.007 2.478 2.949 TRUE
Mandel Marsyas -0.377 0.094 0.565 FALSE
Mandel Manzagol 0.382 0.853 1.324 TRUE
Mandel LabX 1.732 2.203 2.674 TRUE
Marsyas Manzagol 0.288 0.759 1.230 TRUE
Marsyas LabX 1.639 2.109 2.580 TRUE
Manzagol LabX 0.879 1.350 1.821 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/binary_FMeasure.friedman.tukeyKramerHSD.png


100 query subset used in Tagatune evaluation

TeamID TeamID Lowerbound Mean Upperbound Significance
Mandel Zhi 0.383 0.853 1.323 TRUE
Mandel Marsyas 0.609 1.079 1.550 TRUE
Mandel Manzagol 1.284 1.754 2.224 TRUE
Mandel LabX 1.895 2.365 2.835 TRUE
Zhi Marsyas -0.244 0.226 0.696 FALSE
Zhi Manzagol 0.431 0.901 1.371 TRUE
Zhi LabX 1.042 1.512 1.982 TRUE
Marsyas Manzagol 0.204 0.675 1.145 TRUE
Marsyas LabX 0.816 1.286 1.756 TRUE
Manzagol LabX 0.141 0.611 1.081 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/binary_FMeasure.friedman.tukeyKramerHSD.png


Per Track F-measure (Binary) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the F-measure for each track in the test, averaged over all folds.

Full dataset

TeamID TeamID Lowerbound Mean Upperbound Significance
Marsyas Manzagol 1.052 1.124 1.197 TRUE
Marsyas Zhi 0.829 0.901 0.973 TRUE
Marsyas Mandel 1.293 1.365 1.438 TRUE
Marsyas LabX 2.795 2.867 2.939 TRUE
Manzagol Zhi -0.296 -0.223 -0.151 TRUE
Manzagol Mandel 0.168 0.241 0.313 TRUE
Manzagol LabX 1.670 1.743 1.815 TRUE
Zhi Mandel 0.392 0.464 0.537 TRUE
Zhi LabX 1.894 1.966 2.038 TRUE
Mandel LabX 1.429 1.502 1.574 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/binary_FMeasure_per_track.friedman.tukeyKramerHSD.png


100 query subset used in Tagatune evaluation

TeamID TeamID Lowerbound Mean Upperbound Significance
Marsyas Zhi 0.381 0.985 1.589 TRUE
Marsyas Mandel 0.656 1.260 1.864 TRUE
Marsyas Manzagol 1.261 1.865 2.469 TRUE
Marsyas LabX 2.736 3.340 3.944 TRUE
Zhi Mandel -0.329 0.275 0.879 FALSE
Zhi Manzagol 0.276 0.880 1.484 TRUE
Zhi LabX 1.751 2.355 2.959 TRUE
Mandel Manzagol 0.001 0.605 1.209 TRUE
Mandel LabX 1.476 2.080 2.684 TRUE
Manzagol LabX 0.871 1.475 2.079 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/binary_FMeasure_per_track.friedman.tukeyKramerHSD.png


Tag AUC-ROC (Affinity) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Area Under the ROC curve (AUC-ROC) for each tag in the test, averaged over all folds.

Full dataset

TeamID TeamID Lowerbound Mean Upperbound Significance
Marsyas Mandel -0.283 0.087 0.458 FALSE
Marsyas Manzagol 0.935 1.306 1.677 TRUE
Marsyas Zhi 1.735 2.106 2.477 TRUE
Mandel Manzagol 0.848 1.219 1.590 TRUE
Mandel Zhi 1.648 2.019 2.390 TRUE
Manzagol Zhi 0.429 0.800 1.171 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png


100 query subset used in Tagatune evaluation

TeamID TeamID Lowerbound Mean Upperbound Significance
Mandel Marsyas -0.206 0.122 0.450 FALSE
Mandel Manzagol 0.522 0.850 1.178 TRUE
Mandel Zhi 1.025 1.353 1.681 TRUE
Marsyas Manzagol 0.400 0.728 1.056 TRUE
Marsyas Zhi 0.903 1.231 1.559 TRUE
Manzagol Zhi 0.175 0.503 0.831 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png


Per Track AUC-ROC (Affinity) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Area Under the ROC curve (AUC-ROC) for each track/clip in the test, averaged over all folds.

Full dataset

TeamID TeamID Lowerbound Mean Upperbound Significance
Marsyas Mandel 0.523 0.580 0.638 TRUE
Marsyas Manzagol 1.184 1.242 1.299 TRUE
Marsyas Zhi 1.611 1.668 1.726 TRUE
Mandel Manzagol 0.604 0.661 0.719 TRUE
Mandel Zhi 1.030 1.088 1.145 TRUE
Manzagol Zhi 0.369 0.426 0.484 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T/affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png


100 query subset used in Tagatune evaluation

TeamID TeamID Lowerbound Mean Upperbound Significance
Marsyas Mandel 0.071 0.540 1.009 TRUE
Marsyas Manzagol 1.191 1.660 2.129 TRUE
Marsyas Zhi 1.731 2.200 2.669 TRUE
Mandel Manzagol 0.651 1.120 1.589 TRUE
Mandel Zhi 1.191 1.660 2.129 TRUE
Manzagol Zhi 0.071 0.540 1.009 TRUE

download these results as csv

https://music-ir.org/mirex/results/2009/tagatune/eval_T_on_T_subset/affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png


Assorted Results Files for Download

MIREX Statistical Evaluation Results

Full dataset

affinity_tag_fold_AUC_ROC.csv
affinity_clip_AUC_ROC.csv
binary_per_fold_Accuracy.csv
binary_per_fold_Fmeasure.csv
binary_per_fold_negative_example_Accuracy.csv
binary_per_fold_per_track_Accuracy.csv
binary_per_fold_per_track_Fmeasure.csv
binary_per_fold_per_track_negative_example_Accuracy.csv
binary_per_fold_per_track_positive_example_Accuracy.csv
binary_per_fold_positive_example_Accuracy.csv
affinity_clip_Precision_at_3.csv
affinity_clip_Precision_at_6.csv
affinity_clip_Precision_at_9.csv
affinity_clip_Precision_at_12.csv
affinity_clip_Precision_at_15.csv

100 query subset used in Tagatune evaluation

affinity_tag_fold_AUC_ROC.csv
affinity_clip_AUC_ROC.csv
binary_per_fold_Accuracy.csv
binary_per_fold_Fmeasure.csv
binary_per_fold_negative_example_Accuracy.csv
binary_per_fold_per_track_Accuracy.csv
binary_per_fold_per_track_Fmeasure.csv
binary_per_fold_per_track_negative_example_Accuracy.csv
binary_per_fold_per_track_positive_example_Accuracy.csv
binary_per_fold_positive_example_Accuracy.csv
affinity_clip_Precision_at_3.csv
affinity_clip_Precision_at_6.csv
affinity_clip_Precision_at_9.csv
affinity_clip_Precision_at_12.csv
affinity_clip_Precision_at_15.csv

Friedman's Tests Results

Full dataset

affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png
affinity.PrecisionAt3.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt3.friedman.tukeyKramerHSD.png
affinity.PrecisionAt6.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt6.friedman.tukeyKramerHSD.png
affinity.PrecisionAt9.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt9.friedman.tukeyKramerHSD.png
affinity.PrecisionAt12.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt12.friedman.tukeyKramerHSD.png
affinity.PrecisionAt15.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt15.friedman.tukeyKramerHSD.png
binary_Accuracy.friedman.tukeyKramerHSD.csv
binary_Accuracy.friedman.tukeyKramerHSD.png
binary_FMeasure.friedman.tukeyKramerHSD.csv
binary_FMeasure.friedman.tukeyKramerHSD.png
binary_FMeasure_per_track.friedman.tukeyKramerHSD.csv
binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

100 query subset used in Tagatune evaluation

affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.csv
affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png
affinity.PrecisionAt3.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt3.friedman.tukeyKramerHSD.png
affinity.PrecisionAt6.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt6.friedman.tukeyKramerHSD.png
affinity.PrecisionAt9.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt9.friedman.tukeyKramerHSD.png
affinity.PrecisionAt12.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt12.friedman.tukeyKramerHSD.png
affinity.PrecisionAt15.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt15.friedman.tukeyKramerHSD.png
binary_Accuracy.friedman.tukeyKramerHSD.csv
binary_Accuracy.friedman.tukeyKramerHSD.png
binary_FMeasure.friedman.tukeyKramerHSD.csv
binary_FMeasure.friedman.tukeyKramerHSD.png
binary_FMeasure_per_track.friedman.tukeyKramerHSD.csv
binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

Results By Algorithm

(.tgz format)

Full dataset

LabX = Anonymous
Mandel = Michael Mandel
Manzagol = Pierre-Antoine Manzagol
Marsyas = George Tzanetakis
Zhi = Zhi-Sheng Chen


100 query subset used in Tagatune evaluation

LabX = Anonymous
Mandel = Michael Mandel
Manzagol = Pierre-Antoine Manzagol
Marsyas = George Tzanetakis
Zhi = Zhi-Sheng Chen