Difference between revisions of "2008:Audio Cover Song Identification Results"
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-mirex/abs/ +mirex/abstracts/)) |
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-/mirex/2008/results/ +/mirex/results/2008/)) |
||
Line 46: | Line 46: | ||
==Individual Results Files== | ==Individual Results Files== | ||
===Average Precision Scores for Each Query=== | ===Average Precision Scores for Each Query=== | ||
− | '''CL1''' = [https://www.music-ir.org/mirex/2008 | + | '''CL1''' = [https://www.music-ir.org/mirex/results/2008/cover.cl1.eval.csv C. Cao, M. Li] <br /> |
− | '''CL2''' = [https://www.music-ir.org/mirex/2008 | + | '''CL2''' = [https://www.music-ir.org/mirex/results/2008/cover.cl2.eval.csv C. Cao, M. Li] <br /> |
− | '''EL1''' = [https://www.music-ir.org/mirex/2008 | + | '''EL1''' = [https://www.music-ir.org/mirex/results/2008/cover.el1.eval.csv A. Egorov, G. Linetsky] <br /> |
− | '''EL2''' = [https://www.music-ir.org/mirex/2008 | + | '''EL2''' = [https://www.music-ir.org/mirex/results/2008/cover.el2.eval.csv A. Egorov, G. Linetsky] <br /> |
− | '''EL3''' = [https://www.music-ir.org/mirex/2008 | + | '''EL3''' = [https://www.music-ir.org/mirex/results/2008/cover.el3.eval.csv A. Egorov, G. Linetsky] <br /> |
− | '''JCJ''' = [https://www.music-ir.org/mirex/2008 | + | '''JCJ''' = [https://www.music-ir.org/mirex/results/2008/cover.jcj.eval.csv J. H. Jensen, M. G. Christensen, S. H. Jensen] <br /> |
− | '''SGH1''' = [https://www.music-ir.org/mirex/2008 | + | '''SGH1''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh1.eval.csv J. Serr├á, E. G╧îmez, P. Herrera] <br /> |
− | '''SGH2''' = [https://www.music-ir.org/mirex/2008 | + | '''SGH2''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh2.eval.csv J. Serr├á, E. G╧îmez, P. Herrera] <br /> |
===Ranks of the Ten Cover Songs Returned for Each Query=== | ===Ranks of the Ten Cover Songs Returned for Each Query=== | ||
− | '''CL1''' = [https://www.music-ir.org/mirex/2008 | + | '''CL1''' = [https://www.music-ir.org/mirex/results/2008/cover.cl1.eval.debug.csv C. Cao, M. Li] <br /> |
− | '''CL2''' = [https://www.music-ir.org/mirex/2008 | + | '''CL2''' = [https://www.music-ir.org/mirex/results/2008/cover.cl2.eval.debug.csv C. Cao, M. Li] <br /> |
− | '''EL1''' = [https://www.music-ir.org/mirex/2008 | + | '''EL1''' = [https://www.music-ir.org/mirex/results/2008/cover.el1.eval.debug.csv A. Egorov, G. Linetsky] <br /> |
− | '''EL2''' = [https://www.music-ir.org/mirex/2008 | + | '''EL2''' = [https://www.music-ir.org/mirex/results/2008/cover.el2.eval.debug.csv A. Egorov, G. Linetsky] <br /> |
− | '''EL3''' = [https://www.music-ir.org/mirex/2008 | + | '''EL3''' = [https://www.music-ir.org/mirex/results/2008/cover.el3.eval.debug.csv A. Egorov, G. Linetsky] <br /> |
− | '''JCJ''' = [https://www.music-ir.org/mirex/2008 | + | '''JCJ''' = [https://www.music-ir.org/mirex/results/2008/cover.jcj.eval.debug.csv J. H. Jensen, M. G. Christensen, S. H. Jensen] <br /> |
− | '''SGH1''' = [https://www.music-ir.org/mirex/2008 | + | '''SGH1''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh1.eval.debug.csv J. Serr├á, E. G╧îmez, P. Herrera] <br /> |
− | '''SGH2''' = [https://www.music-ir.org/mirex/2008 | + | '''SGH2''' = [https://www.music-ir.org/mirex/results/2008/cover.sgh2.eval.csv J. Serr├á, E. G╧îmez, P. Herrera] <br /> |
===Runtimes=== | ===Runtimes=== |
Revision as of 13:56, 7 June 2010
Still missing runtimes JSD Sept. 11 2008.
Introduction
These are the results for the 2008 running of the Audio Cover Song Identification task. For background information about this task set please refer to the 2008:Audio Cover Song Identification page.
Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. Average precision, which looks at the entire per-query rank-ordered list of all songs in the collection, was the new metric introduced last year.
General Legend
Team ID
CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gόmez, P. Herrera
SGH2 = J. Serrà, E. Gόmez, P. Herrera
Overall Summary Results
Measure | Range | CL1 | CL2 | EL1 | EL2 | EL3 | JCJ | SGH1 | SGH2 |
---|---|---|---|---|---|---|---|---|---|
Total number of covers identified in top 10 | [0-3300] | 1056 | 1073 | 1762 | 1781 | 1778 | 763 | 2422 | 2116 |
Mean number of covers identified in top 10 (average performance) | [0-10] | 3.20 | 3.25 | 5.34 | 5.40 | 5.39 | 2.31 | 7.34 | 6.41 |
Mean (arithmetic) of Avg. Precisions | [0-1] | 0.34 | 0.34 | 0.55 | 0.56 | 0.56 | 0.23 | 0.75 | 0.66 |
Mean rank of first correctly identified cover | [1-1000] | 18.83 | 18.78 | 11.19 | 11.17 | 11.55 | 23.98 | 4.80 | 4.55 |
Number of Correct Covers at Rank X Returned in Top Ten
RankX | CL1 | CL2 | EL1 | EL2 | EL3 | JCJ | SGH1 | SGH2 |
---|---|---|---|---|---|---|---|---|
1 | 192 | 191 | 259 | 259 | 258 | 151 | 281 | 281 |
2 | 164 | 159 | 250 | 251 | 253 | 122 | 286 | 277 |
3 | 143 | 150 | 236 | 235 | 231 | 95 | 278 | 261 |
4 | 131 | 132 | 201 | 213 | 214 | 80 | 272 | 250 |
5 | 114 | 118 | 177 | 176 | 178 | 76 | 265 | 231 |
6 | 96 | 90 | 176 | 177 | 179 | 67 | 265 | 222 |
7 | 80 | 87 | 154 | 158 | 155 | 52 | 247 | 200 |
8 | 62 | 63 | 136 | 141 | 136 | 45 | 230 | 170 |
9 | 47 | 53 | 121 | 120 | 119 | 39 | 183 | 134 |
10 | 27 | 30 | 52 | 51 | 55 | 36 | 115 | 90 |
Total | 1056 | 1073 | 1762 | 1781 | 1778 | 763 | 2422 | 2116 |
Run Times
CL1 | CL2 | EL1 | EL2 | EL3 | JCJ | SGH1 | SGH2 | |
---|---|---|---|---|---|---|---|---|
Run Time (sec) | 281487 | 281352 | 136469 | 435240 | 434712 | 3648 | 373370 | 373371 |
CL1,CL2 ran on FAST2,FAST3. All others ran on ALE Nodes.
Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the Average Precision summary data over the 30 song groups.
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
---|---|---|---|---|---|
Columns | 854.92 | 7 | 122.131 | 144.03 | 0 |
Error | 391.58 | 203 | 1.929 | ||
Total | 1246.5 | 239 |
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CL1 | CL2 | -2.2066 | -0.3000 | 1.6066 | FALSE |
CL1 | EL1 | -4.4233 | -2.5167 | -0.6101 | TRUE |
CL1 | EL2 | -4.4399 | -2.5333 | -0.6267 | TRUE |
CL1 | EL3 | -4.3399 | -2.4333 | -0.5267 | TRUE |
CL1 | JCJ | -1.2899 | 0.6167 | 2.5233 | FALSE |
CL1 | SGH1 | -6.9399 | -5.0333 | -3.1267 | TRUE |
CL1 | SGH2 | -6.1066 | -4.2000 | -2.2934 | TRUE |
CL2 | EL1 | -4.1233 | -2.2167 | -0.3101 | TRUE |
CL2 | EL2 | -4.1399 | -2.2333 | -0.3267 | TRUE |
CL2 | EL3 | -4.0399 | -2.1333 | -0.2267 | TRUE |
CL2 | JCJ | -0.9899 | 0.9167 | 2.8233 | FALSE |
CL2 | SGH1 | -6.6399 | -4.7333 | -2.8267 | TRUE |
CL2 | SGH2 | -5.8066 | -3.9000 | -1.9934 | TRUE |
EL1 | EL2 | -1.9233 | -0.0167 | 1.8899 | FALSE |
EL1 | EL3 | -1.8233 | 0.0833 | 1.9899 | FALSE |
EL1 | JCJ | 1.2267 | 3.1333 | 5.0399 | TRUE |
EL1 | SGH1 | -4.4233 | -2.5167 | -0.6101 | TRUE |
EL1 | SGH2 | -3.5899 | -1.6833 | 0.2233 | FALSE |
EL2 | EL3 | -1.8066 | 0.1000 | 2.0066 | FALSE |
EL2 | JCJ | 1.2434 | 3.1500 | 5.0566 | TRUE |
EL2 | SGH1 | -4.4066 | -2.5000 | -0.5934 | TRUE |
EL2 | SGH2 | -3.5733 | -1.6667 | 0.2399 | FALSE |
EL3 | JCJ | 1.1434 | 3.0500 | 4.9566 | TRUE |
EL3 | SGH1 | -4.5066 | -2.6000 | -0.6934 | TRUE |
EL3 | SGH2 | -3.6733 | -1.7667 | 0.1399 | FALSE |
JCJ | SGH1 | -7.5566 | -5.6500 | -3.7434 | TRUE |
JCJ | SGH2 | -6.7233 | -4.8167 | -2.9101 | TRUE |
SGH1 | SGH2 | -1.0733 | 0.8333 | 2.7399 | FALSE |
Average Performance per Query Group
These are the arithmetic means of the average precisions within each of the 30 query groups.
QueryGroupID | CL1 | CL2 | EL1 | EL2 | EL3 | JCJ | SGH1 | SGH2 |
---|---|---|---|---|---|---|---|---|
1 | 0.411 | 0.412 | 0.792 | 0.819 | 0.819 | 0.128 | 1.000 | 0.770 |
2 | 0.163 | 0.152 | 0.383 | 0.381 | 0.375 | 0.165 | 0.690 | 0.562 |
3 | 0.024 | 0.023 | 0.338 | 0.320 | 0.332 | 0.038 | 0.696 | 0.534 |
4 | 0.730 | 0.752 | 0.991 | 0.991 | 0.991 | 0.501 | 0.995 | 0.995 |
5 | 0.819 | 0.838 | 0.844 | 0.843 | 0.843 | 0.475 | 0.879 | 0.847 |
6 | 0.033 | 0.030 | 0.136 | 0.128 | 0.130 | 0.025 | 0.211 | 0.212 |
7 | 0.406 | 0.346 | 0.923 | 0.920 | 0.929 | 0.268 | 0.998 | 0.981 |
8 | 0.054 | 0.032 | 0.503 | 0.562 | 0.578 | 0.203 | 0.671 | 0.536 |
9 | 0.306 | 0.314 | 0.302 | 0.377 | 0.371 | 0.259 | 0.894 | 0.734 |
10 | 0.507 | 0.506 | 0.720 | 0.713 | 0.717 | 0.285 | 1.000 | 0.916 |
11 | 0.585 | 0.595 | 0.749 | 0.749 | 0.751 | 0.207 | 0.847 | 0.851 |
12 | 0.355 | 0.388 | 0.398 | 0.463 | 0.386 | 0.174 | 0.985 | 0.529 |
13 | 0.316 | 0.323 | 0.426 | 0.427 | 0.423 | 0.300 | 0.474 | 0.482 |
14 | 0.416 | 0.379 | 0.998 | 0.998 | 0.996 | 0.589 | 1.000 | 1.000 |
15 | 0.511 | 0.498 | 0.816 | 0.814 | 0.814 | 0.160 | 0.832 | 0.832 |
16 | 0.041 | 0.037 | 0.567 | 0.492 | 0.547 | 0.132 | 0.899 | 0.766 |
17 | 0.849 | 0.852 | 0.936 | 0.935 | 0.936 | 0.280 | 0.977 | 0.977 |
18 | 0.472 | 0.511 | 0.739 | 0.743 | 0.725 | 0.298 | 0.951 | 0.930 |
19 | 0.093 | 0.104 | 0.093 | 0.104 | 0.103 | 0.079 | 0.118 | 0.100 |
20 | 0.427 | 0.454 | 0.357 | 0.352 | 0.353 | 0.021 | 0.608 | 0.487 |
21 | 0.037 | 0.034 | 0.460 | 0.459 | 0.460 | 0.084 | 0.717 | 0.516 |
22 | 0.054 | 0.058 | 0.078 | 0.078 | 0.078 | 0.056 | 0.027 | 0.035 |
23 | 0.333 | 0.338 | 0.704 | 0.734 | 0.737 | 0.663 | 0.820 | 0.800 |
24 | 0.019 | 0.020 | 0.148 | 0.130 | 0.135 | 0.034 | 0.567 | 0.444 |
25 | 0.025 | 0.038 | 0.553 | 0.556 | 0.545 | 0.121 | 0.843 | 0.683 |
26 | 0.832 | 0.831 | 0.821 | 0.821 | 0.770 | 0.672 | 1.000 | 1.000 |
27 | 0.221 | 0.200 | 0.336 | 0.362 | 0.372 | 0.119 | 0.722 | 0.474 |
28 | 0.163 | 0.161 | 0.489 | 0.510 | 0.476 | 0.329 | 0.474 | 0.457 |
29 | 0.085 | 0.091 | 0.244 | 0.297 | 0.272 | 0.102 | 0.648 | 0.520 |
30 | 0.831 | 0.835 | 0.787 | 0.787 | 0.787 | 0.281 | 0.966 | 0.950 |
Mean | 0.337 | 0.338 | 0.554 | 0.562 | 0.558 | 0.235 | 0.750 | 0.664 |
Individual Results Files
Average Precision Scores for Each Query
CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gόmez, P. Herrera
SGH2 = J. Serrà, E. Gόmez, P. Herrera
Ranks of the Ten Cover Songs Returned for Each Query
CL1 = C. Cao, M. Li
CL2 = C. Cao, M. Li
EL1 = A. Egorov, G. Linetsky
EL2 = A. Egorov, G. Linetsky
EL3 = A. Egorov, G. Linetsky
JCJ = J. H. Jensen, M. G. Christensen, S. H. Jensen
SGH1 = J. Serrà, E. Gόmez, P. Herrera
SGH2 = J. Serrà, E. Gόmez, P. Herrera
Runtimes
Where algorithms have been multi-threaded, the longest runtime is reported.
Where runtimes were not properly reported, file timestamps have been used to approximate a runtime.