2008:Audio Cover Song Identification Results

Placeholder, please stand by (data below are simply copied from 2007 to give us a consistent structure) JSD Sept 7 2008.

Introduction

These are the results for the 2008 running of the Audio Cover Song Identification task. For background information about this task set please refer to the Audio Cover Song Identification page.

Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, was the new metric introduced last year.

General Legend

Team ID

EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper H├╕jvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, S├╕ren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez

Overall Summary Results

file /nema-raid/www/mirex/results/cover.overall.stats.csv not found

Runtimes

Where algorithms have been multi-threaded, the longest runtime is reported.

Where runtimes were not properly reported, file timestamps have been used to approximate a runtime.

file /nema-raid/www/mirex/results/cover.runtimes.csv not found

Number of Correct Covers at Rank X Returned in Top Ten

file /nema-raid/www/mirex/results/cover.rank.distributions.csv not found

Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

file /nema-raid/www/mirex/results/cover.friedmans.summary.csv not found

file /nema-raid/www/mirex/results/cover.friedmans.details.csv not found

Average Performance per Query Group

These are the arithmetic means of the average precisions within each of the 30 query groups.

file /nema-raid/www/mirex/results/cover.avg.prec.by.groups.csv not found

Individual Results Files

Average Precision Scores for Each Query

EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper H├╕jvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, S├╕ren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez

Ranks of the Ten Cover Songs Returned for Each Query

EC = Daniel P. W. Ellis, Courtenay V. Cotton
IM = IMIRSEL M2K
JB = Juan Bello
JEC = Jesper H├╕jvang Jensen, Daniel P. W. Ellis, Mads G. Christensen, S├╕ren Holdt
KL1 = Kyogu Lee 1
KL2 = Kyogu Lee 2
KP = Youngmoo E. Kim, Daniel Perelstein
SG = Joan Serrà, Emilia Gómez

test = Just, DocD