Difference between revisions of "2009:Audio Tag Classification (MajorMiner) Set Results"

From MIREX Wiki
(Team ID)
(Team ID)
Line 145: Line 145:
 
'''CC4''' =  [https://music-ir.org/mirex/2009/results/abs/CC.pdf Chuan Cao, Ming Li]<br />
 
'''CC4''' =  [https://music-ir.org/mirex/2009/results/abs/CC.pdf Chuan Cao, Ming Li]<br />
 
'''GP'''  =  [https://music-ir.org/mirex/2009/results/abs/Peeters_2009_MIREX_classification.pdf Geoffroy Peeters]<br />
 
'''GP'''  =  [https://music-ir.org/mirex/2009/results/abs/Peeters_2009_MIREX_classification.pdf Geoffroy Peeters]<br />
'''GT1''' = [https://music-ir.org/mirex/2009/results/abs/GT.pdf George Tzanetakis]<br />
+
'''GT1''' = [https://music-ir.org/mirex/2009/results/abs/GTfinal.pdf George Tzanetakis]<br />
'''GT2''' = [https://music-ir.org/mirex/2009/results/abs/GT.pdf George Tzanetakis]<br />
+
'''GT2''' = [https://music-ir.org/mirex/2009/results/abs/GTfinal.pdf George Tzanetakis]<br />
 
'''HBC''' = [https://music-ir.org/mirex/2009/results/abs/HBC.pdf Matthew D.Hoffman, David M. Blei, Perry R.Cook]<br />
 
'''HBC''' = [https://music-ir.org/mirex/2009/results/abs/HBC.pdf Matthew D.Hoffman, David M. Blei, Perry R.Cook]<br />
 
'''LWW1''' = [https://music-ir.org/mirex/2009/results/abs/LWW.pdf Hung-Yi Lo, Ju-Chiang Wang, Hsin-Min Wang]<br />
 
'''LWW1''' = [https://music-ir.org/mirex/2009/results/abs/LWW.pdf Hung-Yi Lo, Ju-Chiang Wang, Hsin-Min Wang]<br />

Revision as of 21:38, 26 October 2009

Introduction

This task compares various algorithms' abilities to associate tags with 10-second audio clips of songs. The tags come from the MajorMiner game. This task is very much related to the other audio classification tasks, however, instead of one N-way classification per clip, this task requires N binary classifications per clip.

Two outputs are produced by each algorithm:

  • a set of binary classifications indicating which tags are relevant to each example,
  • a set of 'affinity' scores which indicate the degree to which each tag applies to each track.

These different outputs allow the algorithms to be evaluated both on tag 'classification' and tag 'ranking' (where the tags may be ranked for each track and tracks ranked for each tag).

Data

All of the data is browsable via the MajorMiner search page.

The music consists of 2300 clips selected at random from 3900 tracks. Each clip is 10 seconds long. The 2300 clips represent a total of 1400 different tracks on 800 different albums by 500 different artists. To give a sense for the music collection, the following genre tags have been applied to these artists, albums, and tracks on Last.fm: electronica, rock, indie, alternative, pop, britpop, idm, new wave, hip-hop, singer-songwriter, trip-hop, post-punk, ambient, jazz.

Tags

The MajorMiner game has collected a total of about 73000 taggings, 12000 of which have been verified by at least two users. In these verified taggings, there are 43 tags that have been verified at least 35 times, for a total of about 9000 verified uses. These are the tags we will be using in this task.

Note that these data do not include strict negative labels. While many clips are tagged rock, none are tagged not rock. Frequently, however, a clip will be tagged many times without being tagged rock. We take this as an indication that rock does not apply to that clip. More specifically, a negative example of a particular tag is a clip on which another tag has been verified, but the tag in question has not.

Here is a list of the top 50 tags along with an approximate number of times each has been verified, how many times it's been used in total, and how many different users have ever used it:

Tag Verified Total Users
drums 962 3223 127
guitar 845 3204 181
male 724 2452 95
rock 658 2619 198
synth 498 1889 105
electronic 490 1878 131
pop 479 1761 151
bass 417 1632 99
vocal 355 1378 99
female 342 1387 100
dance 322 1244 115
techno 246 943 104
piano 179 826 120
electronica 168 686 67
hip hop 166 701 126
voice 160 790 55
slow 157 727 90
beat 154 708 90
rap 151 723 129
jazz 136 735 154
80s 130 601 94
fast 109 494 70
instrumental 103 539 62
drum machine 89 427 35
british 81 383 60
country 74 360 105
distortion 73 366 55
saxophone 70 316 86
house 65 298 66
ambient 61 335 78
soft 61 351 58
silence 57 200 35
r&b 57 242 59
strings 55 252 62
quiet 54 261 57
solo 53 268 56
keyboard 53 424 41
punk 51 242 76
horns 48 204 38
drum and bass 48 191 50
noise 46 249 61
funk 46 266 90
acoustic 40 193 58
trumpet 39 174 68
end 38 178 36
loud 37 218 62
organ 35 169 46
metal 35 178 64
folk 33 195 58
trance 33 226 49

Evaluation

Participating algorithms were evaluated with 3-fold cross validation. Artist filtering was used in the production of the test and training splits, I.e. training and test sets contained different artists.

Binary Evaluation

Algorithms are evaluated on their performance at tag classification using F-measure. Results are also reported for simple accuracy, however, as this statistic is dominated by the negative example accuracy it is not a reliable indicator of performance (as a system that returns no tags for any example will achieve a high score on this statistic). However, the accuracies are also reported for positive and negative examples separately as these can help elucidate the behaviour of an algorithm (for example demonstrating if the system is under of over predicting).

Affinity (ranking) Evaluation

Algorithms are evaluated on their performance at tag ranking using the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The affinity scores for each tag to be applied to a track are sorted prior to the computation of the AUC-ROC statistic, which gives higher scores to ranked tag sets where the correct tags appear towards the top of the set.


General Legend

Team ID

BP1 = Juan José Burred, Geoffroy Peeters
BP2 = Juan José Burred, Geoffroy Peeters
CC1 = Chuan Cao, Ming Li
CC2 = Chuan Cao, Ming Li
CC3 = Chuan Cao, Ming Li
CC4 = Chuan Cao, Ming Li
GP = Geoffroy Peeters
GT1 = George Tzanetakis
GT2 = George Tzanetakis
HBC = Matthew D.Hoffman, David M. Blei, Perry R.Cook
LWW1 = Hung-Yi Lo, Ju-Chiang Wang, Hsin-Min Wang
LWW2 = Hung-Yi Lo, Ju-Chiang Wang, Hsin-Min Wang

Overall Summary Results (Binary)

file /nema-raid/www/mirex/results/tag/MajorMiner/summary_binary.csv not found


Summary Binary Relevance F-Measure (Average Across All Folds)

file /nema-raid/www/mirex/results/tag/MajorMiner/binary_avg_Fmeasure.csv not found

Summary Binary Accuracy (Average Across All Folds)

file /nema-raid/www/mirex/results/tag/MajorMiner/binary_avg_Accuracy.csv not found

Summary Positive Example Accuracy (Average Across All Folds)

file /nema-raid/www/mirex/results/tag/MajorMiner/binary_avg_positive_example_Accuracy.csv not found

Summary Negative Example Accuracy (Average Across All Folds)

file /nema-raid/www/mirex/results/tag/MajorMiner/binary_avg_negative_example_Accuracy.csv not found

Overall Summary Results (Affinity)

file /nema-raid/www/mirex/results/tag/MajorMiner/summary_affinity.csv not found

Summary AUC-ROC Tag (Average Across All Folds)

file /nema-raid/www/mirex/results/tag/MajorMiner/affinity_tag_AUC_ROC.csv not found

Select Friedman's Test Results

Tag F-measure (Binary) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the F-measure for each tag in the test, averaged over all folds.

file /nema-raid/www/mirex/results/tag/MajorMiner/binary_FMeasure.friedman.tukeyKramerHSD.csv not found


https://music-ir.org/mirex/2009/results/tag/MajorMiner/small.binary_FMeasure.friedman.tukeyKramerHSD.png

Per Track F-measure (Binary) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the F-measure for each track in the test, averaged over all folds. file /nema-raid/www/mirex/results/tag/MajorMiner/binary_FMeasure_per_track.friedman.tukeyKramerHSD.csv not found


https://music-ir.org/mirex/2009/results/tag/MajorMiner/small.binary_FMeasure_per_track.friedman.tukeyKramerHSD.png

Tag AUC-ROC (Affinity) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Area Under the ROC curve (AUC-ROC) for each tag in the test, averaged over all folds.

file /nema-raid/www/mirex/results/tag/MajorMiner/affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.csv not found

https://music-ir.org/mirex/2009/results/tag/MajorMiner/small.affinity.AUC_ROC_TAG.friedman.tukeyKramerHSD.png

Per Track AUC-ROC (Affinity) Friedman Test

The following table and plot show the results of Friedman's ANOVA with Tukey-Kramer multiple comparisons computed over the Area Under the ROC curve (AUC-ROC) for each track/clip in the test, averaged over all folds.

file /nema-raid/www/mirex/results/tag/MajorMiner/affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.csv not found

https://music-ir.org/mirex/2009/results/tag/MajorMiner/small.affinity.AUC_ROC_TRACK.friedman.tukeyKramerHSD.png

Assorted Results Files for Download

General Results

affinity_tag_fold_AUC_ROC.csv
affinity_clip_AUC_ROC.csv
binary_per_fold_Accuracy.csv
binary_per_fold_Fmeasure.csv
binary_per_fold_negative_example_Accuracy.csv
binary_per_fold_per_track_Accuracy.csv
binary_per_fold_per_track_Fmeasure.csv
binary_per_fold_per_track_negative_example_Accuracy.csv
binary_per_fold_per_track_positive_example_Accuracy.csv
binary_per_fold_positive_example_Accuracy.csv

Friedman's Tests Results

affinity.PrecisionAt3.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt3.friedman.tukeyKramerHSD.png
affinity.PrecisionAt6.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt6.friedman.tukeyKramerHSD.png
affinity.PrecisionAt9.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt9.friedman.tukeyKramerHSD.png
affinity.PrecisionAt12.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt12.friedman.tukeyKramerHSD.png
affinity.PrecisionAt15.friedman.tukeyKramerHSD.csv
affinity.PrecisionAt15.friedman.tukeyKramerHSD.png
binary_Accuracy.friedman.tukeyKramerHSD.csv
binary_Accuracy.friedman.tukeyKramerHSD.png

Results By Algorithm

(.tgz format)

BP1 = Juan José Burred, Geoffroy Peeters
BP2 = Juan José Burred, Geoffroy Peeters
CC1 = Chuan Cao, Ming Li
CC2 = Chuan Cao, Ming Li
CC3 = Chuan Cao, Ming Li
CC4 = Chuan Cao, Ming Li
GP = Geoffroy Peeters
GT1 = George Tzanetakis
GT2 = George Tzanetakis
LWW1 = Hung-Yi Lo, Ju-Chiang Wang, Hsin-Min Wang
LWW2 = Hung-Yi Lo, Ju-Chiang Wang, Hsin-Min Wang
HBC = Matthew D.Hoffman, David M. Blei, Perry R.Cook