Difference between revisions of "2007:Audio Artist Identification Results"

From MIREX Wiki
(New page: Category: Results NEEDS TO BE UPDATED: TEMPLATE ONLY ==Introduction== These are the results for the 2007 running of the Audio Cover Song Identification task. For background information...)
 
(→‎Team ID)
 
(22 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
[[Category: Results]]
 
−
NEEDS TO BE UPDATED: TEMPLATE ONLY
 
 
==Introduction==
 
==Introduction==
−
These are the results for the 2007 running of the Audio Cover Song Identification task. For background information about this task set please refer to the [[Audio Cover Song]] page.
+
These are the results for the 2007 running of the Audio Artist Identification task. For background information about this task set please refer to the [[2007:Audio Artist Identification]] page. The data set was 3060 30 second clips representing 102 different recording artists.
−
 
 
−
Each system was given a collection of 1000 songs which included of 30 different classes (sets) of cover songs where each class/set was represented by 11 different versions of a particular song. Each of the 330 cover songs were used as queries and the systems were required to return 10 results for each query. Systems were evaluated on the number of the songs from the same class/set as the query that were returned in the list of 10 results for each query. '''Special note''': Some systems (e.g., KWL, KWT, LR and TP) participated as a by-product of the [[Audio Music Similarity and Retrieval]] task, that is, these systems were not specifically written to detect cover song variants. Other systems (e.g., CS, DE, KL1 and KL2) were specifically written to detect cover song variants.
 
−
 
 
  
 
===General Legend===
 
===General Legend===
 
====Team ID====
 
====Team ID====
−
'''CS''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_sailer.pdf Christian Sailer and Karin Dressler<sup>(1)</sup>]<br />
+
'''ME''' = [https://www.music-ir.org/mirex/abstracts/2007/AI_CC_GC_MC_AS_mandel.pdf Michael I. Mandel, Daniel P. W. Ellis]<br />
−
'''DE''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_ellis.pdf Daniel P. W. Ellis <sup>(1)</sup>]<br />
+
'''TL''' = [https://www.music-ir.org/mirex/abstracts/2007/AI_CC_GC_MC_AS_lidy.pdf Thomas Lidy, Andreas Rauber, Antonio Pertusa, José Manuel Iñesta]<br />
−
'''KL1''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_lee.pdf Kyogu Lee 1 <sup>(1)</sup>]<br />
+
'''GT''' = [https://www.music-ir.org/mirex/abstracts/2007/AI_CC_GC_MC_AS_tzanetakis.pdf George Tzanetakis]<br />
−
'''KL2''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/CS_lee.pdf Kyogu Lee 2 <sup>(1)</sup>]<br />
+
'''KL''' = Kyogu Lee<br />
−
'''KWL''' = Kris West (Likely) <sup>(2)</sup><br />
+
'''IM''' = IMIRSEL M2K<br />
−
'''KWT''' = Kris West (Trans) <sup>(2)</sup><br />
 
−
'''LR''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_lidy.pdf Thomas Lidy and Andreas Rauber] <sup>(2)</sup><br />
 
−
'''TP''' = [https://www.music-ir.org/evaluation/MIREX/2006_abstracts/AS_pohle.pdf Tim Pohle] <sup>(2)</sup><br />
 
−
 
 
−
<sup>(1)</sup> Denotes submissions specifically designed to detect cover song variants.<br/>
 
−
<sup>(2)</sup> Denotes submissions not-specifically designed to detect cover song variants (See [[Audio Music Similarity and Retrieval Results]]).
 
−
 
 
−
===Calculating Summary Measures===
 
−
'''MRR''' = Mean Reciprocal Rank. Reciprocal rank is the reciprocal of the rank of the first correctly identified cover for each query (1/rank). These values are averaged for each cover song group as well as overall.<br />
 
  
 
==Overall Summary Results==
 
==Overall Summary Results==
−
<csv>coversong_overall.csv</csv>
+
===MIREX 2007 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds===
  
−
===Audio Cover Song Identification Runtime Data===
+
<csv>2007/artist.results.csv</csv>
  
−
<csv>ac06_runtime.csv</csv>
 
  
−
==Friedman Test with Multiple Comparisons Results (p=0.05)==
+
===MIREX 2007 Audio Artist Classification Evaluation Logs and Confusion Matrices===
−
The Friedman test was run in MATLAB against the MRR summary data over the 30 song groups.<br /> Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
+
[https://www.music-ir.org/mirex/results/2007/aa_imirsel_knn.eval.txt IM_knn]<br />
 +
[https://www.music-ir.org/mirex/results/2007/aa_imirsel_svm.eval.txt IM_svm]<br />
 +
[https://www.music-ir.org/mirex/results/2007/aa_lee.eval.txt KL]<br />
 +
[https://www.music-ir.org/mirex/results/2007/aa_lidy.eval.txt TL]<br />
 +
[https://www.music-ir.org/mirex/results/2007/aa_mandel_1.eval.txt ME]<br />
 +
[https://www.music-ir.org/mirex/results/2007/aa_mandel_2_spec.eval.txt ME_spec]<br />
 +
[https://www.music-ir.org/mirex/results/2007/aa_tzan.eval.txt GT]<br />
  
−
<csv>cover_song_sum_friedman.csv</csv>
+
===MIREX 2007 Audio Artist Classification Run Times===
−
<csv>audiocover_friedman_mrr.csv</csv>
 
  
−
==Mean Reciprocal Rank (MRR) Results==
+
<csv>2007/artist.runtime.csv</csv>
−
<csv>coversong_mrr.csv</csv>
 
  
−
==Raw Results==
+
[[Category: Results]]
−
The total number of correctly identified covers for each query, and the maximum number of covers identified for each cover song group.
 
−
<csv>coversong_results.csv</csv>
 

Latest revision as of 17:28, 23 July 2010

Introduction

These are the results for the 2007 running of the Audio Artist Identification task. For background information about this task set please refer to the 2007:Audio Artist Identification page. The data set was 3060 30 second clips representing 102 different recording artists.

General Legend

Team ID

ME = Michael I. Mandel, Daniel P. W. Ellis
TL = Thomas Lidy, Andreas Rauber, Antonio Pertusa, José Manuel Iñesta
GT = George Tzanetakis
KL = Kyogu Lee
IM = IMIRSEL M2K

Overall Summary Results

MIREX 2007 Audio Artist Classification Summary Results - Raw Classification Accuracy Averaged Over Three Train/Test Folds

Participant Avg. Raw Classification Accuracy
IM_knn 35.29%
IM_svm 48.14%
KL 9.71%
TL 38.76%
ME 40.46%
ME_spec 47.16%
GT 36.70%

download these results as csv


MIREX 2007 Audio Artist Classification Evaluation Logs and Confusion Matrices

IM_knn
IM_svm
KL
TL
ME
ME_spec
GT

MIREX 2007 Audio Artist Classification Run Times

Participant Runtime (sec) / Fold
IM-knn Feat Ex: 2760 Train/Classify: 245
IM-svm Feat Ex: 2760 Train/Classify: 188
KL Feat Ex: 1806 Train/Classify: 4664
TL Feat Ex: 23631 Train/Classify: 4764
ME Feat Ex: 3403 Train/Classify: 65
ME_spec Feat Ex: 3382 Train/Classify: 64
GT Feat Ex/Train/Classify: 632

download these results as csv

Retrieved from "https://www.music-ir.org/mirex/w/index.php?title=2007:Audio_Artist_Identification_Results&oldid=7394"