2006:Audio Cover Song Identification Results

From MIREX Wiki
Revision as of 19:32, 13 May 2010 by IMIRSELBot (talk | contribs) (Robot: Automated text replacement (-<csv([^>]*)> +<csv\1>2006/))

Introduction

These are the results for the 2006 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2006:Audio Cover Song page.

Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Special note: Some systems (e.g., KWL, KWT, LR and TP) participated as a by-product of the 2006:Audio Music Similarity and Retrieval task, that is, these systems were not specifically written to detect cover song variants. Other systems (e.g., CS, DE, KL1 and KL2) were specifically written to detect cover song variants.


General Legend

Team ID

CS = Christian Sailer and Karin Dressler(1)
DE = Daniel P. W. Ellis (1)
KL1 = Kyogu Lee 1 (1)
KL2 = Kyogu Lee 2 (1)
KWL = Kris West (Likely) (2)
KWT = Kris West (Trans) (2)
LR = Thomas Lidy and Andreas Rauber (2)
TP = Tim Pohle (2)

(1) Denotes submissions specifically designed to detect cover song variants.
(2) Denotes submissions not-specifically designed to detect cover song variants (See 2006:Audio Music Similarity and Retrieval Results).

Calculating Summary Measures

MRR = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each cover song group as well as overall.

Overall Summary Results

Measure Range CS DE KL1 KL2 KWL KWT LR TP
Total number of covers identified [0-3300] 211 761 365 314 117 102 149 116
Mean number of covers identified (average performance) [0-10] 0.64 2.31 1.11 0.95 0.35 0.31 0.45 0.35
Mean of maxima (average of best-case performance) [0-10] 2.13 4.53 2.50 2.27 1.30 1.23 1.57 1.50
Mean reciprocal rank of first correctly identified cover [0-1] 0.21 0.49 0.22 0.22 0.10 0.10 0.12 0.09

download these results as csv

Audio Cover Song Identification Runtime Data

Team ID Machine Run-time(seconds)
CS feature beer 8 28219
CS distmat beer 8 1707
DE feature beer 6 26814
DE distmat beer 6 60511
KL1 feature beer 1 15600
KL1 distmat beer 6 140360
KL2 feature beer 0 15623
KL2 distmat beer 0 136379

download these results as csv

Friedman Test with Multiple Comparisons Results (p=0.05)

The Friedman test was run in MATLAB against the MRR summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Friedman's ANOVA Table
Source SS df MS Chi-sq Prob>Chi-sq
Columns 313.1 7 44.7286 52.5798 4.49E-09
Error 937.4 203 4.6177
Total 1250.5 239

download these results as csv

TeamID TeamID Lowerbound Mean Upperbound Signficance
CS DE -3.9763 -2.0667 -0.157 TRUE
CS KL1 -1.043 0.8667 2.7763 FALSE
CS KL2 -1.193 0.7167 2.6263 FALSE
CS KWL -0.3597 1.55 3.4597 FALSE
CS KWT -0.2597 1.65 3.5597 FALSE
CS LR -1.2263 0.6833 2.593 FALSE
CS TP -0.3763 1.5333 3.443 FALSE
DE KL1 1.0237 2.9333 4.843 TRUE
DE KL2 0.8737 2.7833 4.693 TRUE
DE KWL 1.707 3.6167 5.5263 TRUE
DE KWT 1.807 3.7167 5.6263 TRUE
DE LR 0.8403 2.75 4.6597 TRUE
DE TP 1.6903 3.6 5.5097 TRUE
KL1 KL2 -2.0597 -0.15 1.7597 FALSE
KL1 KWL -1.2263 0.6833 2.593 FALSE
KL1 KWT -1.1263 0.7833 2.693 FALSE
KL1 LR -2.093 -0.1833 1.7263 FALSE
KL1 TP -1.243 0.6667 2.5763 FALSE
KL2 KWL -1.0763 0.8333 2.743 FALSE
KL2 KWT -0.9763 0.9333 2.843 FALSE
KL2 LR -1.943 -0.0333 1.8763 FALSE
KL2 TP -1.093 0.8167 2.7263 FALSE
KWL KWT -1.8097 0.1 2.0097 FALSE
KWL LR -2.7763 -0.8667 1.043 FALSE
KWL TP -1.9263 -0.0167 1.893 FALSE
KWT LR -2.8763 -0.9667 0.943 FALSE
KWT TP -2.0263 -0.1167 1.793 FALSE
LR TP -1.0597 0.85 2.7597 FALSE

download these results as csv

Mean Reciprocal Rank (MRR) Results

Song ID Query CS DE KL1 KL2 KWL KWT LR TP
1 c000095 0 0 0 0 0 0 0 0
1 c000142 0 0 1/7 0 0 0 0 0
1 c000219 0 0 0 0 0 0 0 0
1 c000275 1/6 0 0 0 1/7 0 0 0
1 c000337 0 0 0 0 0 0 0 1
1 c000353 0 0 0 0 0 0 1/6 0
1 c000457 1/8 0 0 0 0 0 1/9 0
1 c000561 0 0 0 0 0 0 0 0
1 c000604 0 0 0 0 0 0 1/2 0
1 c000606 0 0 0 0 0 0 0 1
1 c000707 0 0 0 0 0 0 0 0
1 MRR 0.03 0.00 0.01 0.00 0.01 0.00 0.07 0.18
2 c000047 1/2 1 0 0 0 0 1/7 0
2 c000088 1 1 0 0 0 0 0 0
2 c000144 0 1 0 0 0 0 0 0
2 c000172 0 1 0 0 0 0 0 0
2 c000186 1/6 1/3 0 0 0 1/5 1/5 1/5
2 c000323 0 1/5 0 0 0 0 1/4 0
2 c000538 0 0 0 0 0 0 0 0
2 c000748 0 1/3 0 1/9 1/9 1/2 0 1/7
2 c000862 0 1/2 1/2 1/2 0 0 1 1/5
2 c000956 0 1 0 0 0 0 0 0
2 c000981 0 0 0 0 0 0 0 0
2 MRR 0.16 0.58 0.05 0.06 0.01 0.07 0.14 0.05
3 c000001 0 1 0 0 0 1/9 1/9 0
3 c000198 1/4 1/3 1/2 1/4 0 0 1/6 1
3 c000212 0 0 0 0 0 1 1 1/3
3 c000288 0 1 1/3 1/2 0 0 0 0
3 c000366 0 0 0 0 0 0 0 0
3 c000395 1 1/2 1/5 1/4 0 0 0 0
3 c000483 1/3 1/4 1 1 0 1/7 0 0
3 c000491 0 1/5 1/5 1/3 0 0 0 0
3 c000551 0 1/5 0 0 0 1/6 1/5 1/6
3 c000570 1/6 1/3 0 0 1/5 0 0 0
3 c000599 1/2 1 0 0 1/8 0 1/6 1/4
3 MRR 0.20 0.44 0.21 0.21 0.03 0.15 0.15 0.16
4 c000055 1/3 1 0 0 0 0 0 0
4 c000057 0 1 0 0 0 0 0 0
4 c000228 0 0 0 0 1/2 0 1/4 1/5
4 c000333 0 1 0 0 0 0 0 0
4 c000347 0 1 0 0 1/2 0 0 0
4 c000461 0 1/2 0 0 0 0 0 0
4 c000542 0 1 0 0 0 0 0 0
4 c000548 0 0 0 0 0 0 1/3 0
4 c000639 0 1/6 0 0 0 0 0 0
4 c000689 0 0 0 0 0 0 0 0
4 c000948 0 0 0 1/6 1/7 0 0 0
4 MRR 0.03 0.52 0.00 0.02 0.10 0.00 0.05 0.02
5 c000004 1/2 1 1 1 1/4 1/7 0 1/3
5 c000101 0 1 1 0 0 1/6 1/2 1/6
5 c000189 0 1 1/2 1/2 0 1/7 1/4 1/2
5 c000224 0 1 1 1 1/7 1/3 1/3 1/9
5 c000245 1/8 1 1 1 0 0 0 0
5 c000316 0 1 1 1 1/7 1/8 0 0
5 c000385 0 1 0 1 0 1/7 0 0
5 c000603 0 1 1 1 1/7 1/7 1 1/2
5 c000734 0 0 1 0 1 1/6 0 1/8
5 c000765 0 1 1 1 0 1/3 0 1
5 c000832 0 1 1 1 1/4 1/5 1/2 1
5 MRR 0.06 0.91 0.86 0.77 0.18 0.17 0.23 0.35
6 c000005 1 1/2 0 0 1/5 0 1 0
6 c000043 1/2 0 0 1/5 1/2 0 1/4 0
6 c000082 1 1/6 0 0 0 0 0 0
6 c000126 1/3 1/2 0 0 0 0 0 0
6 c000204 0 1/9 0 0 1/5 0 0 0
6 c000256 0 1/7 0 0 0 0 0 0
6 c000289 1/3 0 1/4 1 0 0 0 1/6
6 c000377 1/4 0 0 0 0 0 0 0
6 c000399 1/5 1/4 0 0 0 0 0 0
6 c000768 0 1/8 1/7 0 0 0 0 0
6 c000928 1 0 1/5 0 0 0 0 1/8
6 MRR 0.42 0.16 0.05 0.12 0.08 0.00 0.12 0.03
7 c000112 0 0 1/7 0 0 0 1/8 0
7 c000145 0 0 0 0 0 0 0 0
7 c000314 1 1 0 0 0 0 0 1/6
7 c000345 0 1/5 0 0 0 0 0 0
7 c000348 0 1 0 0 0 0 0 0
7 c000396 0 1/4 0 0 0 0 0 1/7
7 c000539 1 1 0 0 0 0 0 1/5
7 c000578 1/2 1 0 0 0 0 0 0
7 c000654 0 0 0 0 1/8 0 0 1/5
7 c000890 1 1/5 0 0 0 0 0 0
7 c000952 0 1 0 0 0 1 1/2 0
7 MRR 0.32 0.51 0.01 0.00 0.01 0.10 0.07 0.06
8 c000015 1 0 0 0 1 0 0 0
8 c000155 0 0 0 0 0 0 0 0
8 c000182 1 0 0 0 0 1/9 1/6 1/6
8 c000610 1/4 1/9 0 0 1/3 1/9 0 0
8 c000690 1/3 1/5 0 0 1/2 1/2 0 0
8 c000706 1 0 0 0 0 0 0 1/5
8 c000842 0 0 0 0 0 1/6 0 0
8 c000846 1 0 0 0 0 0 0 0
8 c000897 1 0 0 0 0 0 1/4 0
8 c000975 0 1/8 0 0 0 0 0 0
8 c000998 0 0 0 0 1/3 1/7 1/3 1/9
8 MRR 0.51 0.04 0.00 0.00 0.20 0.09 0.07 0.06
9 c000042 0 1/3 0 0 0 0 0 0
9 c000136 0 0 0 0 0 0 0 0
9 c000157 0 1/8 1/2 0 0 0 0 0
9 c000210 0 1/2 0 0 0 1/3 1/5 1
9 c000221 1/9 1 1 1 0 1/3 1/2 0
9 c000418 0 1 0 0 0 1/6 0 0
9 c000801 1/5 1/2 0 0 0 1/8 0 0
9 c000834 0 1 0 0 1/8 1/6 1/6 1
9 c000885 0 1 0 0 0 0 0 0
9 c000908 0 1 0 0 0 0 0 0
9 c000919 0 1 1 1 1/8 0 0 0
9 MRR 0.03 0.68 0.23 0.18 0.03 0.11 0.08 0.18
10 c000007 0 0 0 0 0 0 0 1/9
10 c000021 0 0 0 0 0 1/2 0 0
10 c000050 0 0 0 0 0 0 0 0
10 c000073 1/2 0 0 0 1/6 0 0 0
10 c000218 0 0 0 0 0 0 1/7 0
10 c000241 0 0 0 0 0 1/2 1/6 1/2
10 c000679 0 0 0 0 0 0 0 0
10 c000730 0 0 0 0 0 1/8 1/6 1/9
10 c000804 0 0 0 0 0 0 0 0
10 c000867 0 0 0 0 1/6 1 0 0
10 c000922 0 0 0 1/9 0 1 1 1/2
10 MRR 0.05 0.00 0.00 0.01 0.03 0.28 0.15 0.11
11 c000059 1/5 1 1 1 0 1/4 1 1/3
11 c000197 0 1 1 1 1/4 0 1/8 0
11 c000220 0 0 1 1 0 0 0 0
11 c000263 1/8 1 1/2 1/3 1/3 1/4 1/9 1
11 c000375 1 1/5 1/4 0 0 1/8 0 0
11 c000429 1 0 1 1 1/4 0 0 0
11 c000431 1 1/4 1/2 0 1/2 0 1 1/3
11 c000529 1 1 0 1 1/3 1/6 1 1/3
11 c000568 1 0 0 1 1/3 1/2 1/4 1/3
11 c000685 1/2 1 1 1 0 0 0 1
11 c000692 0 0 0 0 0 0 0 0
11 MRR 0.53 0.50 0.57 0.67 0.18 0.12 0.32 0.30
12 c000179 0 1 1 1 0 0 0 0
12 c000181 1/8 1 1 1 1 0 0 0
12 c000260 0 1 1 1 1/7 1/5 0 1/6
12 c000284 0 1 1 1 1/2 0 1/3 1/5
12 c000329 0 1 0 0 0 0 0 0
12 c000367 1/5 1 0 1/2 1/5 0 1/6 0
12 c000521 0 1 1 1 0 0 0 0
12 c000800 0 1 0 1/5 0 0 0 0
12 c000865 1/2 1 1 1 0 0 0 0
12 c000962 0 1 1 1 1/2 0 0 0
12 c000971 0 1/2 1/3 1 0 0 0 0
12 MRR 0.08 0.95 0.67 0.79 0.21 0.02 0.05 0.03
13 c000013 1/9 0 0 0 0 0 0 0
13 c000303 0 0 0 0 0 0 0 0
13 c000364 0 0 0 0 0 0 0 0
13 c000412 0 0 0 0 1/3 0 1/7 0
13 c000567 0 0 0 1/3 1 0 0 0
13 c000713 0 0 1/7 0 0 0 0 0
13 c000761 0 1/8 0 0 1/4 0 1/3 0
13 c000781 0 1/6 0 0 0 1 0 0
13 c000868 0 1 0 0 1 1 0 0
13 c000909 0 1/3 0 0 0 1/6 0 1
13 c000994 1/2 0 0 0 0 0 1/2 0
13 MRR 0.06 0.15 0.01 0.03 0.23 0.20 0.09 0.09
14 c000063 1 1 1 1 0 0 0 0
14 c000110 0 1/5 0 0 0 0 0 0
14 c000116 0 1 1/6 0 0 0 0 0
14 c000146 0 1 1 1 0 0 0 0
14 c000336 0 1 1 1/5 0 1/5 0 0
14 c000378 1/4 1 1 1 0 0 0 0
14 c000579 1/4 0 1 1 0 1/4 0 0
14 c000823 0 1 1/8 0 0 0 0 1/8
14 c000869 1 1 1 1/7 0 1/2 0 0
14 c000972 0 1 0 0 0 0 1/9 0
14 c000997 0 0 0 0 0 0 0 0
14 MRR 0.23 0.75 0.57 0.39 0.00 0.10 0.01 0.01
15 c000048 1 1 0 0 0 0 0 0
15 c000125 0 1 0 0 0 0 0 0
15 c000191 0 1 0 0 0 0 1/7 0
15 c000451 1 1 0 0 0 0 0 0
15 c000475 0 1 0 0 0 0 0 0
15 c000585 0 1 0 0 1 0 0 0
15 c000614 1/4 0 0 0 0 0 0 0
15 c000658 1/3 1 0 0 0 0 0 1/8
15 c000659 0 0 0 0 0 0 0 0
15 c000724 1/5 1 0 0 1 0 0 0
15 c000915 0 1 0 0 1/2 0 0 0
15 MRR 0.26 0.83 0.00 0.01 0.23 0.01 0.01 0.01
16 c000038 0 1/4 0 1/2 0 1/3 1/6 1/2
16 c000080 0 0 0 0 0 0 0 0
16 c000085 0 0 0 0 0 0 1/3 1
16 c000153 1/4 1/6 0 0 0 0 0 0
16 c000162 0 1/3 0 0 0 1 1/5 0
16 c000257 0 1 0 0 0 1/3 1/3 1/4
16 c000324 0 0 0 0 0 0 0 1/6
16 c000416 0 1/2 0 0 0 0 0 0
16 c000512 0 0 0 0 0 0 0 0
16 c000793 1/8 0 0 1/3 0 0 0 0
16 c000859 0 0 0 0 0 0 0 0
16 MRR 0.03 0.20 0.00 0.08 0.00 0.16 0.09 0.17
17 c000006 0 1 0 0 0 0 1/2 0
17 c000019 1/3 1 0 0 0 0 0 0
17 c000149 0 1 1 0 0 0 0 1/2
17 c000195 0 1 0 0 0 0 0 0
17 c000266 1 1 1/5 1 0 1/4 1/5 0
17 c000372 0 1 0 0 0 0 0 0
17 c000425 0 1 0 0 1/5 1/3 0 1/2
17 c000649 1 1 0 0 0 0 0 0
17 c000775 0 0 0 0 1/5 0 1/2 1/3
17 c000888 0 1 1/4 0 0 0 0 0
17 c000955 0 1 0 1 0 0 0 0
17 MRR 0.21 0.91 0.13 0.19 0.04 0.05 0.11 0.12
18 c000074 0 1 1 1 0 0 0 0
18 c000081 1 1 1 1 0 0 1/8 1/7
18 c000128 0 1 1/4 0 0 0 0 0
18 c000167 1/2 1 1 1 0 0 0 0
18 c000386 1/2 1/2 0 1/6 1/7 0 1/5 0
18 c000400 0 1 1 1 0 0 1/9 0
18 c000525 0 1 0 0 0 0 0 0
18 c000637 0 1 0 1/8 0 0 0 0
18 c000684 1/2 1/5 0 0 0 0 0 0
18 c000731 0 1/3 0 1/9 0 0 0 0
18 c000773 0 1 1 1 1/4 0 1/6 0
18 MRR 0.24 0.82 0.49 0.49 0.04 0.00 0.05 0.01
19 c000016 0 0 0 0 0 0 0 0
19 c000199 0 1/2 0 0 0 0 1/4 0
19 c000265 0 0 0 0 1/2 0 1/6 0
19 c000279 0 0 0 0 0 0 1/6 0
19 c000326 0 1/5 0 0 0 0 0 0
19 c000423 0 1/8 0 0 0 0 0 0
19 c000437 0 1/2 0 0 1/7 1 1/6 0
19 c000788 0 1/2 0 0 0 0 0 1/6
19 c000851 1/4 0 0 0 0 0 0 0
19 c000924 0 0 0 0 0 0 0 0
19 c000934 0 1/3 0 0 0 0 1/5 0
19 MRR 0.02 0.21 0.00 0.00 0.06 0.09 0.09 0.02
20 c000304 0 0 0 0 0 0 0 0
20 c000330 0 0 0 0 1/9 0 1/3 1/3
20 c000447 0 0 0 0 0 0 0 0
20 c000508 0 0 0 0 0 0 0 0
20 c000625 0 0 0 0 0 0 1/6 0
20 c000674 0 1 0 0 0 0 1 1/9
20 c000716 0 0 0 0 0 0 0 0
20 c000854 0 1/2 0 0 0 0 0 0
20 c000870 0 0 0 0 0 1/7 1/3 0
20 c000882 0 1/8 0 0 0 0 0 0
20 c000925 0 1/2 0 0 0 0 0 0
20 MRR 0.00 0.19 0.00 0.00 0.01 0.01 0.18 0.04
21 c000023 0 0 0 1/8 0 0 0 0
21 c000120 1/5 0 1/2 1 1/7 1/6 1/9 0
21 c000230 0 0 0 0 0 0 0 0
21 c000276 0 0 0 0 0 0 0 0
21 c000281 1/6 1 0 0 0 0 1/2 1/6
21 c000342 0 1/7 0 0 0 0 0 0
21 c000359 0 1/3 1/4 1/3 0 0 0 0
21 c000467 0 1/2 0 0 1/2 1/8 0 1/6
21 c000666 1/3 0 0 0 1/6 0 1/5 0
21 c000688 0 0 0 0 0 0 1 0
21 c000769 1 0 1/5 1/6 0 1/8 1 0
21 MRR 0.15 0.18 0.09 0.15 0.07 0.04 0.26 0.03
22 c000097 1/9 1/2 0 0 0 1/4 0 0
22 c000129 0 1 0 0 1/9 1 0 1/6
22 c000234 0 1/5 0 0 0 1/9 0 1/4
22 c000319 1/8 1 0 1/8 0 0 0 0
22 c000446 1/4 0 1 1/4 0 0 0 0
22 c000473 0 0 0 0 0 0 0 0
22 c000635 1 0 0 0 0 0 0 0
22 c000737 0 1/2 0 0 0 0 0 0
22 c000778 0 1 0 0 0 0 0 0
22 c000876 1 0 0 0 0 0 0 0
22 c000929 1 1 1/6 0 1/7 0 0 0
22 MRR 0.32 0.47 0.11 0.03 0.02 0.12 0.00 0.05
23 c000058 1/4 1 1/8 1/4 0 0 0 0
23 c000096 1/8 1/5 1/2 1 0 1/4 0 1/2
23 c000203 1/4 0 0 0 1/5 0 0 0
23 c000229 1 1 1 1 1/9 1/2 1 1
23 c000274 0 1 1 0 0 1/8 0 0
23 c000322 1/3 0 1/2 1 1/5 1/3 1/2 0
23 c000495 1/3 1 1/4 1/2 0 1 1 0
23 c000555 1 0 0 0 0 0 0 0
23 c000586 1/4 1 1 1 1 1/9 1 1
23 c000602 1/2 0 0 0 0 0 0 0
23 c000873 1/3 1 0 0 0 1 1 1/7
23 MRR 0.40 0.56 0.40 0.43 0.14 0.30 0.43 0.24
24 c000207 0 0 0 0 0 0 1/2 0
24 c000302 0 0 0 0 0 0 1/4 0
24 c000341 1/5 0 0 0 1/7 0 0 0
24 c000371 1/3 1 0 0 0 1/7 0 1/2
24 c000430 0 1 0 0 0 0 1/3 0
24 c000574 1 1/3 0 0 0 0 0 0
24 c000708 1/8 0 0 0 0 0 0 0
24 c000739 0 0 0 0 0 1/4 0 1/3
24 c000762 1/2 0 0 0 0 0 0 0
24 c000875 1 1 0 0 0 1/7 1/6 0
24 c000903 1/6 0 0 0 0 0 0 0
24 MRR 0.30 0.30 0.00 0.00 0.01 0.05 0.11 0.08
25 c000022 0 0 0 0 0 0 0 0
25 c000067 1/3 1 0 0 1 0 1/4 0
25 c000098 0 0 1/2 0 0 0 1/7 0
25 c000174 1/6 1 1/4 1/5 1/8 0 0 0
25 c000223 0 1 1/2 1/6 0 0 0 1/4
25 c000268 0 1/2 1/5 0 0 0 0 1/2
25 c000273 0 1/3 0 0 0 0 0 0
25 c000296 0 0 0 0 0 0 0 0
25 c000519 0 0 1/4 0 0 0 0 0
25 c000580 1 1/2 1/2 1/2 1/9 1/5 0 0
25 c000794 0 0 0 0 1 0 0 0
25 MRR 0.14 0.39 0.22 0.09 0.20 0.02 0.04 0.07
26 c000099 0 1 0 0 0 0 1/6 0
26 c000271 1/3 0 0 0 0 0 0 1/3
26 c000338 1/3 1 0 0 0 1/6 0 0
26 c000349 1 1 0 0 0 0 1/9 0
26 c000370 1/8 1/2 1 1/3 1/4 1/2 1/2 1/3
26 c000424 0 0 0 1 0 0 0 0
26 c000573 0 1 1/5 1/2 0 1/8 0 0
26 c000663 0 1/6 0 0 0 0 0 0
26 c000701 0 0 0 0 0 0 0 0
26 c000771 0 1 0 0 0 0 0 0
26 c000959 1 0 0 0 0 1/6 0 1/8
26 MRR 0.25 0.52 0.11 0.17 0.02 0.09 0.07 0.07
27 c000454 0 0 1 1 0 0 0 0
27 c000571 1/4 1 0 0 0 1/5 0 0
27 c000627 0 0 0 1/6 0 0 0 0
27 c000633 0 1/2 0 0 0 0 0 0
27 c000643 0 1 1/5 1/8 0 1 0 0
27 c000749 0 0 1 1 0 0 0 0
27 c000780 0 1 1 1 1/8 1/5 0 1/4
27 c000815 0 1 1 1 0 0 1/5 1/4
27 c000824 0 0 0 0 0 0 0 0
27 c000840 0 0 1/8 1 1/2 0 1/2 0
27 c000855 0 1 0 0 1 1 0 0
27 MRR 0.02 0.50 0.39 0.48 0.15 0.22 0.06 0.05
28 c000027 1 1 0 0 1/8 1/2 0 1/7
28 c000041 1/5 1 1 1 0 0 1/2 0
28 c000106 0 1/4 0 1/7 1/4 1/5 1/2 0
28 c000298 0 1 0 0 0 0 0 0
28 c000358 1/3 1 0 0 0 0 1/9 0
28 c000362 1/2 1 1 0 1/2 0 1/8 0
28 c000449 1/3 1 0 1/5 0 1/7 0 1/4
28 c000478 1/2 1 1 1/4 1 1/7 0 0
28 c000511 1/2 1 1 0 0 0 0 1/5
28 c000770 1/2 1 0 0 0 0 0 1/6
28 c000816 1/2 1 0 1/2 1 1/2 1/2 0
28 MRR 0.40 0.93 0.36 0.19 0.26 0.14 0.16 0.08
29 c000009 0 1 0 1/4 0 0 0 0
29 c000046 1 1 1 1 1/2 0 1/6 0
29 c000049 0 1 1 1 0 0 1/8 0
29 c000202 0 1 1 1 0 1/2 1/4 1/7
29 c000238 1 0 1 1 0 0 1/6 1/5
29 c000293 0 1 0 0 0 0 1/7 0
29 c000409 0 1 1/4 1 0 0 0 0
29 c000484 0 1 1 1 1/2 1/6 1/2 1/3
29 c000618 0 1 0 1/3 0 0 1/6 1/2
29 c000792 0 1 0 0 1/4 0 0 1/4
29 c000983 0 1 1 1 1 1/2 1/2 0
29 MRR 0.18 0.91 0.57 0.69 0.20 0.11 0.18 0.14
30 c000090 1 1 1 0 0 1/7 0 1/8
30 c000240 1 1 1 1 1/2 0 1/7 0
30 c000513 0 0 0 0 1/7 0 0 0
30 c000572 1 1 1/5 1/5 1/4 0 0 0
30 c000652 1 1/3 1 1 0 1/7 0 1/4
30 c000757 0 1 0 0 1/2 0 0 0
30 c000817 0 1 1/8 1/9 1/9 0 1/7 0
30 c000848 1/3 1 1/2 0 1 0 0 0
30 c000967 1 1/4 1/6 1/5 1/3 0 0 0
30 c000989 1 0 0 0 0 1/6 0 0
30 c000996 1/6 1/5 0 0 1/7 0 1/5 0
30 MRR 0.59 0.62 0.36 0.23 0.27 0.04 0.04 0.03
Grand MRR 0.21 0.49 0.22 0.22 0.10 0.10 0.12 0.09

download these results as csv

Raw Results

The total number of correctly identified covers for each query, and the maximum number of covers identified for each cover song group.

Song ID Query CS DE KL1 KL2 KWT KWL LR TP
1 c000095 0 0 0 0 0 0 0 0
1 c000142 0 0 1 0 0 0 0 0
1 c000219 0 0 0 0 0 0 0 0
1 c000275 1 0 0 0 0 1 0 0
1 c000337 0 0 0 0 0 0 0 1
1 c000353 0 0 0 0 0 0 1 0
1 c000457 1 0 0 0 0 0 1 0
1 c000561 0 0 0 0 0 0 0 0
1 c000604 0 0 0 0 0 0 1 0
1 c000606 0 0 0 0 0 0 0 1
1 c000707 0 0 0 0 0 0 0 0
1 MAX 1 0 1 0 0 1 1 1
2 c000047 1 2 0 0 1 0 1 0
2 c000088 1 2 0 0 0 0 0 0
2 c000144 0 2 0 0 0 0 0 0
2 c000172 0 2 0 0 0 0 0 0
2 c000186 1 1 0 0 1 0 1 1
2 c000323 0 1 0 0 0 0 1 0
2 c000538 0 0 0 0 0 0 0 0
2 c000748 0 2 0 1 1 1 0 1
2 c000862 0 2 1 1 0 0 1 1
2 c000956 1 3 0 0 0 0 0 0
2 c000981 0 0 1 0 0 0 0 0
2 MAX 1 3 1 1 1 1 1 1
3 c000001 0 3 1 0 1 0 1 0
3 c000198 1 1 4 2 0 0 1 1
3 c000212 0 0 0 0 3 0 2 2
3 c000288 0 3 3 2 0 0 0 0
3 c000366 0 0 0 0 0 0 0 0
3 c000395 2 4 1 1 0 0 0 0
3 c000483 1 2 4 3 1 0 0 0
3 c000491 0 3 2 2 0 0 0 0
3 c000551 0 1 0 0 1 0 1 1
3 c000570 3 1 0 0 1 1 0 0
3 c000599 1 3 0 0 1 1 2 1
3 MAX 3 4 4 3 3 1 2 2
4 c000055 1 5 0 0 0 0 0 0
4 c000057 0 2 0 0 0 0 0 0
4 c000228 0 0 0 0 0 2 1 1
4 c000333 0 4 0 0 0 0 0 0
4 c000347 0 2 0 0 0 1 0 0
4 c000461 0 3 0 0 0 0 0 0
4 c000542 0 4 0 0 0 0 0 0
4 c000548 0 0 0 0 0 0 1 0
4 c000639 0 2 0 0 0 0 0 0
4 c000689 0 0 0 0 0 0 0 0
4 c000948 0 0 0 1 0 1 0 0
4 MAX 1 5 0 1 0 2 1 1
5 c000004 1 4 5 6 1 1 0 1
5 c000101 0 2 7 0 1 0 2 1
5 c000189 0 2 5 6 1 0 3 3
5 c000224 0 5 7 6 2 1 3 1
5 c000245 1 5 5 6 0 0 0 0
5 c000316 0 4 4 5 1 2 0 0
5 c000385 0 5 0 8 2 0 0 1
5 c000603 0 2 6 5 1 1 5 3
5 c000734 0 0 6 0 1 1 0 1
5 c000765 0 1 8 7 1 1 0 1
5 c000832 0 4 5 6 2 1 2 3
5 MAX 1 5 8 8 2 2 5 3
6 c000005 1 1 0 0 0 1 1 0
6 c000043 1 0 0 1 0 1 1 0
6 c000082 1 1 0 0 0 0 0 0
6 c000126 3 1 0 0 0 0 0 0
6 c000204 0 1 0 0 0 1 1 0
6 c000256 0 1 0 1 0 0 0 0
6 c000289 3 0 1 1 0 0 0 1
6 c000377 1 0 0 0 0 0 0 0
6 c000399 1 2 0 0 0 0 0 0
6 c000768 0 1 1 0 0 0 0 0
6 c000928 1 0 1 0 0 0 0 1
6 MAX 3 2 1 1 0 1 1 1
7 c000112 0 0 1 0 1 0 1 0
7 c000145 0 0 0 0 0 0 0 0
7 c000314 1 5 0 0 0 0 0 1
7 c000345 0 2 0 0 0 0 0 0
7 c000348 0 5 0 0 0 0 0 0
7 c000396 0 1 0 0 0 0 0 1
7 c000539 1 5 0 0 0 0 1 2
7 c000578 3 2 0 0 0 0 0 0
7 c000654 0 0 0 0 0 1 0 1
7 c000890 2 1 0 0 0 0 0 0
7 c000952 0 3 0 0 1 0 1 0
7 MAX 3 5 1 0 1 1 1 2
8 c000015 5 0 0 0 0 1 0 0
8 c000155 0 0 0 0 0 0 0 0
8 c000182 3 0 0 0 1 0 1 1
8 c000610 2 1 0 0 1 1 0 1
8 c000690 1 1 0 0 1 1 0 0
8 c000706 4 0 0 0 0 0 0 1
8 c000842 0 0 0 0 1 0 0 0
8 c000846 4 0 0 0 0 0 0 0
8 c000897 1 0 0 0 0 0 1 1
8 c000975 0 1 0 0 0 0 0 0
8 c000998 0 0 0 0 1 1 1 1
8 MAX 5 1 0 0 1 1 1 1
9 c000042 0 1 0 0 0 1 0 0
9 c000136 0 0 0 0 0 0 0 0
9 c000157 0 1 2 0 0 0 0 0
9 c000210 0 2 0 0 3 0 1 1
9 c000221 1 2 1 1 2 0 2 0
9 c000418 0 4 0 0 1 0 0 0
9 c000801 1 2 0 0 1 0 0 0
9 c000834 0 3 0 0 1 1 1 1
9 c000885 0 5 0 0 0 0 0 0
9 c000908 0 6 0 0 1 0 0 0
9 c000919 0 3 2 1 0 1 0 0
9 MAX 1 6 2 1 3 1 2 1
10 c000007 0 0 0 0 0 0 0 1
10 c000021 0 0 0 0 2 0 0 0
10 c000050 0 0 0 0 0 0 0 0
10 c000073 1 0 0 0 0 1 1 0
10 c000218 0 0 0 0 0 0 1 0
10 c000241 0 0 0 0 1 0 1 1
10 c000679 0 0 0 0 0 0 0 0
10 c000730 0 0 0 0 2 0 1 1
10 c000804 0 0 0 0 0 0 0 0
10 c000867 0 0 0 0 1 1 1 0
10 c000922 0 0 0 1 1 0 2 1
10 MAX 1 0 0 1 2 1 2 1
11 c000059 2 2 4 5 2 0 3 3
11 c000197 0 1 3 5 0 1 1 0
11 c000220 0 0 3 6 0 0 0 0
11 c000263 1 2 3 4 2 1 1 2
11 c000375 2 3 4 0 1 0 0 0
11 c000429 1 0 3 4 0 1 0 0
11 c000431 1 3 3 0 0 2 2 1
11 c000529 1 1 0 3 1 2 2 1
11 c000568 3 0 0 3 2 2 3 2
11 c000685 1 1 5 6 0 0 0 1
11 c000692 0 0 0 0 0 0 0 0
11 MAX 3 3 5 6 2 2 3 3
12 c000179 0 7 5 2 0 0 0 0
12 c000181 1 7 6 2 0 1 0 0
12 c000260 0 1 5 2 1 1 0 2
12 c000284 0 7 6 4 0 1 1 1
12 c000329 0 7 0 0 0 0 0 0
12 c000367 1 4 0 3 0 1 1 0
12 c000521 0 7 6 5 0 0 0 0
12 c000800 0 7 0 1 0 0 0 0
12 c000865 1 7 6 4 0 0 0 0
12 c000962 0 7 5 4 0 1 0 0
12 c000971 0 1 2 5 0 0 0 0
12 MAX 1 7 6 5 1 1 1 2
13 c000013 1 0 0 0 0 0 0 0
13 c000303 0 0 0 0 0 0 0 0
13 c000364 0 0 0 0 0 0 0 0
13 c000412 0 0 0 0 0 1 1 0
13 c000567 0 0 0 1 0 1 0 0
13 c000713 0 0 1 0 0 0 0 0
13 c000761 0 1 0 0 0 1 1 0
13 c000781 0 1 0 0 1 0 0 0
13 c000868 0 3 0 0 1 1 0 0
13 c000909 0 3 0 0 1 0 0 1
13 c000994 2 0 0 0 0 0 1 0
13 MAX 2 3 1 1 1 1 1 1
14 c000063 1 5 6 3 0 0 0 0
14 c000110 0 1 0 0 0 0 0 0
14 c000116 0 5 1 0 0 0 0 0
14 c000146 0 2 6 3 0 0 0 0
14 c000336 0 7 6 1 1 0 0 0
14 c000378 1 6 7 3 1 0 0 0
14 c000579 1 0 7 3 1 0 0 0
14 c000823 0 5 1 0 0 0 0 1
14 c000869 3 5 7 1 1 0 0 0
14 c000972 0 5 0 0 0 0 1 0
14 c000997 0 0 0 0 0 0 0 0
14 MAX 3 7 7 3 1 0 1 1
15 c000048 3 2 0 0 0 0 0 0
15 c000125 0 6 0 0 0 0 0 0
15 c000191 0 4 0 1 0 0 1 0
15 c000451 2 8 0 0 0 0 0 0
15 c000475 1 3 0 0 0 0 0 0
15 c000585 0 7 0 0 0 1 0 0
15 c000614 1 1 0 0 1 0 0 0
15 c000658 2 5 0 0 0 0 0 1
15 c000659 0 0 0 0 0 0 0 0
15 c000724 1 4 0 0 0 1 0 0
15 c000915 0 5 0 0 0 1 0 0
15 MAX 3 8 0 1 1 1 1 1
16 c000038 0 4 0 1 2 0 2 2
16 c000080 0 0 0 0 1 0 0 0
16 c000085 0 0 0 0 0 0 2 3
16 c000153 1 2 0 0 0 0 0 0
16 c000162 0 3 0 0 1 0 1 0
16 c000257 0 4 0 0 1 0 2 2
16 c000324 0 0 0 0 0 0 0 1
16 c000416 0 3 0 0 0 0 0 0
16 c000512 0 0 0 0 0 0 0 0
16 c000793 1 0 0 1 0 0 0 0
16 c000859 0 0 0 0 0 0 0 0
16 MAX 1 4 0 1 2 0 2 3
17 c000006 0 9 0 0 0 0 1 0
17 c000019 2 7 0 0 0 0 0 0
17 c000149 0 5 2 0 0 0 0 1
17 c000195 0 8 0 1 0 0 0 0
17 c000266 2 8 2 1 1 0 1 0
17 c000372 0 5 0 0 0 0 0 0
17 c000425 0 8 0 0 1 1 0 1
17 c000649 2 6 0 0 0 0 0 0
17 c000775 0 0 0 0 0 1 1 1
17 c000888 0 7 1 0 0 0 0 0
17 c000955 0 9 0 1 0 0 0 0
17 MAX 2 9 2 1 1 1 1 1
18 c000074 1 5 5 5 0 0 0 0
18 c000081 1 1 5 5 0 0 1 1
18 c000128 0 3 4 0 0 0 0 0
18 c000167 1 6 5 4 0 0 0 0
18 c000386 1 3 0 2 0 1 1 0
18 c000400 0 4 5 5 0 0 1 0
18 c000525 0 5 0 0 0 0 0 0
18 c000637 0 1 0 1 0 0 0 0
18 c000684 1 1 1 0 0 0 0 0
18 c000731 0 4 0 2 0 0 0 0
18 c000773 0 3 4 3 0 1 1 0
18 MAX 1 6 5 5 0 1 1 1
19 c000016 0 0 0 0 0 0 0 0
19 c000199 0 3 0 0 0 0 1 0
19 c000265 0 1 0 0 0 1 1 0
19 c000279 0 0 0 0 0 0 1 0
19 c000326 0 1 0 0 0 0 0 0
19 c000423 0 1 0 0 0 0 0 0
19 c000437 0 5 0 0 1 1 1 0
19 c000788 0 4 0 0 0 0 0 2
19 c000851 1 0 0 0 0 0 0 0
19 c000924 0 0 0 0 0 0 0 0
19 c000934 0 1 0 0 0 0 1 0
19 MAX 1 5 0 0 1 1 1 2
20 c000304 0 0 0 0 0 0 0 0
20 c000330 0 0 0 0 0 1 2 1
20 c000447 0 0 0 0 0 0 1 0
20 c000508 0 0 0 0 0 0 0 0
20 c000625 0 0 0 0 0 0 1 0
20 c000674 0 5 0 0 0 0 2 1
20 c000716 0 0 0 0 0 0 0 0
20 c000854 0 1 0 0 0 0 0 0
20 c000870 0 0 0 0 1 0 1 0
20 c000882 0 1 0 0 0 0 0 0
20 c000925 0 4 0 0 0 0 0 0
20 MAX 0 5 0 0 1 1 2 1
21 c000023 0 0 0 1 0 0 0 0
21 c000120 2 0 1 1 1 1 1 0
21 c000230 0 0 0 0 0 0 0 0
21 c000276 0 0 0 0 0 0 0 0
21 c000281 1 1 0 0 0 0 1 1
21 c000342 0 2 0 0 0 0 0 0
21 c000359 0 2 1 1 0 0 0 0
21 c000467 0 2 0 0 1 1 0 1
21 c000666 2 0 0 0 0 2 1 0
21 c000688 0 0 0 0 0 0 2 0
21 c000769 4 0 1 1 1 0 3 0
21 MAX 4 2 1 1 1 2 3 1
22 c000097 1 3 0 0 1 0 0 0
22 c000129 0 4 0 0 1 1 0 2
22 c000234 0 2 0 0 1 0 0 1
22 c000319 1 3 0 1 0 0 0 0
22 c000446 2 0 1 1 0 0 0 0
22 c000473 0 0 0 0 0 0 0 0
22 c000635 2 0 0 0 0 0 0 1
22 c000737 0 1 0 0 0 0 0 0
22 c000778 0 4 0 0 0 0 0 0
22 c000876 4 0 0 0 0 0 0 0
22 c000929 4 5 1 0 0 1 0 0
22 MAX 4 5 1 1 1 1 0 2
23 c000058 3 4 2 3 0 0 1 0
23 c000096 1 1 4 4 2 0 0 1
23 c000203 1 0 0 0 0 1 0 0
23 c000229 1 5 5 5 2 1 2 2
23 c000274 0 5 4 0 1 0 0 0
23 c000322 2 0 3 4 2 1 2 0
23 c000495 1 5 3 4 2 0 3 0
23 c000555 3 0 0 0 0 0 0 0
23 c000586 1 5 5 4 1 2 2 2
23 c000602 1 0 0 0 0 0 1 0
23 c000873 2 5 0 0 1 0 2 2
23 MAX 3 5 5 5 2 2 3 2
24 c000207 0 0 0 0 0 0 1 0
24 c000302 0 0 0 0 0 0 1 0
24 c000341 2 0 0 0 0 1 0 0
24 c000371 1 3 0 0 1 0 0 1
24 c000430 0 3 0 0 0 0 1 0
24 c000574 3 1 0 0 0 0 0 0
24 c000708 2 0 0 0 0 0 0 0
24 c000739 0 0 0 0 1 0 0 1
24 c000762 3 0 0 0 0 0 0 0
24 c000875 2 3 0 0 1 0 1 0
24 c000903 3 0 0 0 0 0 0 0
24 MAX 3 3 0 0 1 1 1 1
25 c000022 0 0 0 1 0 0 0 0
25 c000067 1 2 1 0 0 1 1 0
25 c000098 0 0 1 0 0 0 1 0
25 c000174 1 3 2 2 0 1 0 0
25 c000223 0 3 1 1 0 0 0 1
25 c000268 0 2 1 0 0 0 0 1
25 c000273 0 1 1 0 0 0 0 0
25 c000296 0 0 0 0 0 0 0 0
25 c000519 0 0 2 0 0 0 0 0
25 c000580 2 3 1 1 1 1 0 0
25 c000794 0 0 0 0 0 1 0 0
25 MAX 2 3 2 2 1 1 1 1
26 c000099 0 3 0 0 0 0 1 0
26 c000271 1 0 0 0 0 0 0 1
26 c000338 1 3 0 0 1 0 0 0
26 c000349 1 1 0 0 0 0 1 0
26 c000370 1 2 1 2 2 1 1 1
26 c000424 0 0 0 3 0 0 0 0
26 c000573 0 4 2 2 1 0 0 0
26 c000663 0 2 0 0 0 0 0 0
26 c000701 0 0 0 0 0 0 0 0
26 c000771 0 1 0 0 0 0 0 0
26 c000959 2 0 0 0 1 0 0 1
26 MAX 2 4 2 3 2 1 1 1
27 c000454 0 0 1 1 0 0 0 0
27 c000571 1 5 0 0 1 0 0 0
27 c000627 0 0 0 1 0 0 0 0
27 c000633 0 5 0 0 0 0 0 0
27 c000643 0 3 1 1 1 0 0 0
27 c000749 0 0 2 1 0 0 0 0
27 c000780 0 5 4 3 2 1 0 1
27 c000815 0 5 4 4 0 0 1 1
27 c000824 0 0 0 0 0 0 0 0
27 c000840 0 0 1 3 0 2 1 0
27 c000855 0 4 0 0 1 2 0 0
27 MAX 1 5 4 4 2 2 1 1
28 c000027 3 3 0 0 2 2 0 1
28 c000041 1 6 4 1 0 0 2 0
28 c000106 0 2 0 1 1 2 2 0
28 c000298 0 6 0 0 0 0 0 0
28 c000358 2 5 0 0 0 0 1 1
28 c000362 2 3 2 0 0 1 1 0
28 c000449 1 3 0 1 1 0 0 2
28 c000478 2 6 5 3 1 3 0 0
28 c000511 3 3 3 0 0 0 0 2
28 c000770 1 1 0 0 0 0 0 3
28 c000816 2 4 0 1 1 2 2 0
28 MAX 3 6 5 3 2 3 2 3
29 c000009 0 8 0 1 0 0 0 0
29 c000046 1 9 7 7 0 1 1 1
29 c000049 0 8 6 7 0 0 1 0
29 c000202 0 8 6 7 2 0 1 1
29 c000238 1 0 6 7 0 0 1 1
29 c000293 0 5 0 0 0 0 1 0
29 c000409 0 8 5 5 0 0 0 0
29 c000484 0 9 6 7 1 1 2 1
29 c000618 0 8 0 2 0 0 1 1
29 c000792 0 9 0 0 0 1 0 1
29 c000983 0 9 6 7 1 1 1 0
29 MAX 1 9 7 7 2 1 2 1
30 c000090 3 5 4 0 1 0 0 1
30 c000240 4 6 3 3 0 1 1 0
30 c000513 0 0 0 0 0 1 0 0
30 c000572 3 5 3 2 0 1 0 0
30 c000652 4 1 4 3 1 0 0 2
30 c000757 0 5 0 0 0 1 0 0
30 c000817 0 5 1 1 0 1 1 0
30 c000848 1 5 3 0 0 1 0 0
30 c000967 2 1 4 2 0 1 0 0
30 c000989 1 0 0 0 1 0 0 0
30 c000996 1 1 0 0 0 2 2 0
30 MAX 4 6 4 3 1 2 2 2
Sum of maxima 64 136 75 68 39 37 47 45
Grand Total 211 761 365 314 117 102 149 116

download these results as csv