https://www.music-ir.org/mirex/w/api.php?action=feedcontributions&user=AndreasEhmann&feedformat=atomMIREX Wiki - User contributions [en]2024-03-29T08:55:22ZUser contributionsMediaWiki 1.31.1https://www.music-ir.org/mirex/w/index.php?title=2012:Symbolic_Melodic_Similarity_Results&diff=91122012:Symbolic Melodic Similarity Results2012-10-05T22:19:28Z<p>AndreasEhmann: /* General Legend */</p>
<hr />
<div>==Introduction== <br />
These are the results for the 2011 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2011:Symbolic Melodic Similarity]] page.<br />
<br />
Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:<br />
<br />
* 0. No errors<br />
* 1. One note deleted<br />
* 2. One note inserted<br />
* 3. One interval enlarged<br />
* 4. One interval compressed<br />
<br />
For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).<br />
<br />
====Evalutron 6000 Summary Data====<br />
'''Number of evaluators''' = 6 <br /><br />
'''Number of evaluations per query/candidate pair''' = 1 <br /><br />
'''Number of queries per grader''' = 1 <br /><br />
'''Total number of candidates returned''' = 3900 <br /><br />
'''Total number of unique query/candidate pairs graded''' = 895 <br /><br />
'''Average number of query/candidate pairs evaluated per grader: 149 <br /><br />
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30<br /><br />
<br />
== General Legend ==<br />
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"<br />
|- style="background: yellow"<br />
! width="80" | Sub code<br />
! width="200" | Submission name<br />
! width="80" style="text-align: center;" | Abstract<br />
! width="540" | Contributors<br />
|-<br />
<br />
! DB1<br />
| PPM-DJ || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/DB1.pdf PDF] ||<br />
[http://compmus.ime.usp.br Antonio de Carvalho<br />
Junior],[http://www.pet.di.ufpb.br Leonardo Batista]<br />
|-<br />
<br />
! ULMS1<br />
| ShapeH || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS1.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
<br />
! ULMS2<br />
| ShapeL || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS2.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
<br />
! ULMS3<br />
| ShapeG || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS3.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
! ULMS4<br />
| ShapeTime || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS4.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
! ULMS5<br />
| Time || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS5.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
|}<br />
<br />
====Broad Categories====<br />
'''NS''' = Not Similar<br /><br />
'''SS''' = Somewhat Similar<br /> <br />
'''VS''' = Very Similar<br /><br />
<br />
====Table Headings ====<br />
'''ADR''' = Average Dynamic Recall <br /><br />
'''NRGB''' = Normalize Recall at Group Boundaries <br /><br />
'''AP''' = Average Precision (non-interpolated) <br /><br />
'''PND''' = Precision at N Documents <br /><br />
<br />
===Calculating Summary Measures===<br />
'''Fine'''<sup>(1)</sup> = Sum of fine-grained human similarity decisions (0-100). <br /><br />
'''PSum'''<sup>(1)</sup> = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. <br /><br />
'''WCsum'''<sup>(1)</sup> = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). <br /><br />
'''SDsum'''<sup>(1)</sup> = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). <br /><br />
'''Greater0'''<sup>(1)</sup> = NS=0, SS=1, VS=1 (binary relevance judgment).<br /><br />
'''Greater1'''<sup>(1)</sup> = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar).<br /><br />
<br />
<sup>(1)</sup>Normalized to the range 0 to 1.<br />
<br />
==Summary Results==<br />
===Overall Scores (Includes Perfect and Error Candidates)===<br />
<csv>2012/sms/report_all.csv</csv><br />
<br />
===Scores by Query Error Types===<br />
====No Errors====<br />
<csv>2012/sms/No_Errors.csv</csv><br />
====Note Deletions====<br />
<csv>2012/sms/Deleted.csv</csv><br />
====Note Insertions====<br />
<csv>2012/sms/Inserted.csv</csv><br />
====Enlarged Intervals====<br />
<csv>2012/sms/Enlarged.csv</csv><br />
====Compressed Intervals====<br />
<csv>2012/sms/Compressed.csv</csv><br />
<br />
[[Category: Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Symbolic_Melodic_Similarity_Results&diff=91112012:Symbolic Melodic Similarity Results2012-10-05T22:14:28Z<p>AndreasEhmann: Created page with "==Introduction== These are the results for the 2011 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[201..."</p>
<hr />
<div>==Introduction== <br />
These are the results for the 2011 running of the Symbolic Melodic Similarity task set. For background information about this task set please refer to the [[2011:Symbolic Melodic Similarity]] page.<br />
<br />
Each system was given a query and returned the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each query, we made four classes of error-mutations, thus the set comprises the following query classes:<br />
<br />
* 0. No errors<br />
* 1. One note deleted<br />
* 2. One note inserted<br />
* 3. One interval enlarged<br />
* 4. One interval compressed<br />
<br />
For each query (and its 4 mutations), the returned results (candidates) from all systems were then grouped together (query set) for evaluation by the human graders. The graders were provide with only heard perfect version against which to evaluate the candidates and did not know whether the candidates came from a perfect or mutated query. Each query/candidate set was evaluated by 1 individual grader. Using the Evalutron 6000 system, the graders gave each query/candidate pair two types of scores. Graders were asked to provide 1 categorical score with 3 categories: NS,SS,VS as explained below, and one fine score (in the range from 0 to 100).<br />
<br />
====Evalutron 6000 Summary Data====<br />
'''Number of evaluators''' = 6 <br /><br />
'''Number of evaluations per query/candidate pair''' = 1 <br /><br />
'''Number of queries per grader''' = 1 <br /><br />
'''Total number of candidates returned''' = 3900 <br /><br />
'''Total number of unique query/candidate pairs graded''' = 895 <br /><br />
'''Average number of query/candidate pairs evaluated per grader: 149 <br /><br />
'''Number of queries''' = 6 (perfect) with each perfect query error-mutated 4 different ways = 30<br /><br />
<br />
== General Legend ==<br />
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"<br />
|- style="background: yellow"<br />
! width="80" | Sub code<br />
! width="200" | Submission name<br />
! width="80" style="text-align: center;" | Abstract<br />
! width="540" | Contributors<br />
|-<br />
<br />
! DB1<br />
| PPM-DJ || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/DB1.pdf PDF] ||<br />
[http://compmus.ime.usp.br Antonio de Carvalho<br />
Junior],[http://www.pet.di.ufpb.br Leonardo Batista]<br />
|-<br />
<br />
! ULMS1<br />
| ShapeH || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS1.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
<br />
! ULMS2<br />
| ShapeL || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS2.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
<br />
! ULMS3<br />
| ShapeG || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS3.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
! ULMS4<br />
| ShapeTime || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS4.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
! ULMS5<br />
| Time || style="text-align: center;" |<br />
[https://www.music-ir.org/mirex/abstracts/2012/ULMS5.pdf PDF] ||<br />
[http://julian-urbano.info Julián Urbano], [http://www.kr.inf.uc3m.es<br />
Juan Lloréns],[http://sites.google.com/site/jorgemorato/ Jorge<br />
Morato], [http://www.kr.inf.uc3m.es Sonia Sánchez-Cuadrado]<br />
|-<br />
<br />
|}<br />
====Broad Categories====<br />
'''NS''' = Not Similar<br /><br />
'''SS''' = Somewhat Similar<br /> <br />
'''VS''' = Very Similar<br /><br />
<br />
====Table Headings ====<br />
'''ADR''' = Average Dynamic Recall <br /><br />
'''NRGB''' = Normalize Recall at Group Boundaries <br /><br />
'''AP''' = Average Precision (non-interpolated) <br /><br />
'''PND''' = Precision at N Documents <br /><br />
<br />
===Calculating Summary Measures===<br />
'''Fine'''<sup>(1)</sup> = Sum of fine-grained human similarity decisions (0-100). <br /><br />
'''PSum'''<sup>(1)</sup> = Sum of human broad similarity decisions: NS=0, SS=1, VS=2. <br /><br />
'''WCsum'''<sup>(1)</sup> = 'World Cup' scoring: NS=0, SS=1, VS=3 (rewards Very Similar). <br /><br />
'''SDsum'''<sup>(1)</sup> = 'Stephen Downie' scoring: NS=0, SS=1, VS=4 (strongly rewards Very Similar). <br /><br />
'''Greater0'''<sup>(1)</sup> = NS=0, SS=1, VS=1 (binary relevance judgment).<br /><br />
'''Greater1'''<sup>(1)</sup> = NS=0, SS=0, VS=1 (binary relevance judgment using only Very Similar).<br /><br />
<br />
<sup>(1)</sup>Normalized to the range 0 to 1.<br />
<br />
==Summary Results==<br />
===Overall Scores (Includes Perfect and Error Candidates)===<br />
<csv>2012/sms/report_all.csv</csv><br />
<br />
===Scores by Query Error Types===<br />
====No Errors====<br />
<csv>2012/sms/No_Errors.csv</csv><br />
====Note Deletions====<br />
<csv>2012/sms/Deleted.csv</csv><br />
====Note Insertions====<br />
<csv>2012/sms/Inserted.csv</csv><br />
====Enlarged Intervals====<br />
<csv>2012/sms/Enlarged.csv</csv><br />
====Compressed Intervals====<br />
<csv>2012/sms/Compressed.csv</csv><br />
<br />
[[Category: Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=91102012:MIREX2012 Results2012-10-04T21:39:11Z<p>AndreasEhmann: /* Runtime for Submissions Run by NEMA DIY */</p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; <br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx05/ MIREX05 Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/ind08/ INDIAN08 Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx09_0db/ MIREX09 0dB Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset] &nbsp;<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* Query-by-Singing/Humming Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1a_jang/ Jang Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1b_thinkit/ ThinkIt Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1c_ioacas/ IOACAS Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset] &nbsp;<br />
* Query-by-Tapping Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task1_jang/ Jang Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task1_hsiao/ HSIAO Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task2_jang/ Subtask2 Dataset] &nbsp;<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=91092012:MIREX2012 Results2012-10-04T21:38:54Z<p>AndreasEhmann: /* Machine Specifications */</p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; <br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx05/ MIREX05 Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/ind08/ INDIAN08 Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx09_0db/ MIREX09 0dB Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx09_m5db/ MIREX09 -5dB Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mrx09_p5db/ MIREX09 +5dB Dataset] &nbsp;<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] <br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* Query-by-Singing/Humming Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1a_jang/ Jang Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1b_thinkit/ ThinkIt Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1c_ioacas/ IOACAS Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset] &nbsp;<br />
* Query-by-Tapping Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task1_jang/ Jang Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task1_hsiao/ HSIAO Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task2_jang/ Subtask2 Dataset] &nbsp;<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Music_Similarity_and_Retrieval_Results&diff=91082012:Audio Music Similarity and Retrieval Results2012-10-04T21:13:22Z<p>AndreasEhmann: /* Reports */</p>
<hr />
<div>== Introduction ==<br />
These are the results for the 2012 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.<br />
<br />
Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.<br />
<br />
The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage. <br />
<br />
<br />
=== General Legend ===<br />
<br />
<br />
==== Team ID ====<br />
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"<br />
|- style="background: yellow;"<br />
! width="80" | Sub code <br />
! width="200" | Submission name <br />
! width="80" style="text-align: center;" | Abstract <br />
! width="440" | Contributors<br />
|-<br />
! DM6<br />
| DM6 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/DM6.pdf PDF] || [http://www.iam.ecs.soton.ac.uk/ Franz de Leon], [http://www.iam.ecs.soton.ac.uk/ Kirk Martinez]<br />
|-<br />
! DM7<br />
| DM7 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/DM7.pdf PDF] || [http://www.iam.ecs.soton.ac.uk/ Franz de Leon], [http://www.iam.ecs.soton.ac.uk/ Kirk Martinez]<br />
|-<br />
! GT3<br />
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/GT3.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis] <br />
|- <br />
! JR2<br />
| modulationSim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/JR2.pdf PDF] || [http://mirlab.org/jmzen0921 Jia-Min Ren], [http://mirlab.org/jang Jyh-Shing Roger Jang]<br />
|- <br />
! NHHL1<br />
| AMSR_2012_1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/NHHL1.pdf PDF] || [http://11471178.net/ Byeong-jun Han], [http://marg.snu.ac.kr/people Kyogu Lee],[http://ccrma.stanford.edu/~juhan Juhan Nam],[http://ccrma.stanford.edu/~jorgeh/ Jorge Herrera]<br />
|- <br />
! NHHL2<br />
| AMSR_2012_2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/NHHL2.pdf PDF] || [http://11471178.net/ Byeong-jun Han], [http://marg.snu.ac.kr/people Kyogu Lee],[http://ccrma.stanford.edu/~juhan Juhan Nam],[http://ccrma.stanford.edu/~jorgeh/ Jorge Herrera]<br />
|-<br />
! PS1<br />
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/PS1.pdf PDF] || [http://www.ofai.at/~dominik.schnitzer Dominik Schnitzer], [http://www.cp.jku.at/ Tim Pohle]<br />
|-<br />
! RW4<br />
| modulationSimFrameUBM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/RW4.pdf PDF] || [http://mirlab.org/jmzen0921 Jia-Min Ren],[http://mirlab.org/new/ Ming-Ju Wu],[http://mirlab.org/jang Jyh-Shing Roger Jang]<br />
|-<br />
! SSKP1<br />
| cbmr_sim_2010 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSPK1.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Tim Pohle]<br />
|-<br />
! SSKS2<br />
| cbmr_sim_2011 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSKS2.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Reinhard Sonnleitner]<br />
|- <br />
|}<br />
<br />
====Broad Categories====<br />
'''NS''' = Not Similar<br /><br />
'''SS''' = Somewhat Similar<br /><br />
'''VS''' = Very Similar<br /><br />
<br />
=====Understanding Summary Measures=====<br />
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). <br /><br />
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. <br /><br />
<br />
==Human Evaluation==<br />
===Overall Summary Results===<br />
<br />
<csv p=3>2012/ams/summary_evalutron.csv</csv><br />
<br />
<br /><br />
'''Note:RZ1''' is the random result for comparing purpose.<br />
<br />
===Friedman's Tests===<br />
====Friedman's Test (FINE Scores)====<br />
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);<br />
<br />
<csv p=3>2012/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv><br />
<br />
[[File:evalutron.fine.friedman.tukeyKramerHSD.png|500px]]<br />
<br />
====Friedman's Test (BROAD Scores)====<br />
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);<br />
<br />
<csv p=3>2012/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv><br />
<br />
[[File:evalutron.cat.friedman.tukeyKramerHSD.png|500px]]<br />
<br />
===Summary Results by Query===<br />
====FINE Scores====<br />
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference. <br />
<br />
<csv p=1>2012/ams/fine_scores.csv</csv><br />
<br />
====BROAD Scores====<br />
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference. <br />
<br />
<csv p=1>2012/ams/cat_scores.csv</csv><br />
<br />
===Raw Scores===<br />
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2012:Audio Music Similarity and Retrieval Raw Data]] page.<br />
<br />
==Metadata and Distance Space Evaluation==<br />
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:<br />
* Neighbourhood clustering by artist, album and genre<br />
* Artist-filtered genre clustering<br />
* How often the triangular inequality holds<br />
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).<br />
<br />
=== Reports ===<br />
<br />
'''DM6''' = [https://music-ir.org/mirex/results/2012/ams/statistics/DM6/report.txt Franz de Leon, Kirk Martinez]<br /><br />
'''DM7''' = [https://music-ir.org/mirex/results/2012/ams/statistics/DM7/report.txt Franz de Leon, Kirk Martinez]<br /><br />
'''GT3''' = [https://music-ir.org/mirex/results/2012/ams/statistics/GT3/report.txt George Tzanetakis]<br /><br />
'''JR2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/JR2/report.txt Jia-Min Ren, Jyh-Shing Roger Jang]<br /><br />
'''NHHL1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/NHHL1/report.txt Byeong-jun Han, Kyogu Lee,Juhan Nam,Jorge Herrera]<br /><br />
'''NHHL2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/NHHL2/report.txt Byeong-jun Han, Kyogu Lee,Juhan Nam,Jorge Herrera]<br /><br />
'''PS1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/PS1/report.txt Dominik Schnitzer, Tim Pohle]<br /><br />
'''RW4''' = [https://music-ir.org/mirex/results/2012/ams/statistics/RW4/report.txt Jia-Min Ren,Ming-Ju Wu,Jyh-Shing Roger Jang]<br /><br />
'''SSKP1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKP1/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br /><br />
'''SSKP2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKS2/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Reinhard Sonnleitner]<br /></div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Music_Similarity_and_Retrieval_Results&diff=91072012:Audio Music Similarity and Retrieval Results2012-10-04T21:12:26Z<p>AndreasEhmann: /* Team ID */</p>
<hr />
<div>== Introduction ==<br />
These are the results for the 2012 running of the Audio Music Similarity and Retrieval task set. For background information about this task set please refer to the Audio Music Similarity and Retrieval page.<br />
<br />
Each system was given 7000 songs chosen from IMIRSEL's "uspop", "uscrap" and "american" "classical" and "sundry" collections. Each system then returned a 7000x7000 distance matrix. 100 songs were randomly selected from the 10 genre groups (10 per genre) as queries and the first 5 most highly ranked songs out of the 7000 were extracted for each query (after filtering out the query itself, returned results from the same artist were also omitted). Then, for each query, the returned results (candidates) from all participants were grouped and were evaluated by human graders using the Evalutron 6000 grading system. Each individual query/candidate set was evaluated by a single grader. For each query/candidate pair, graders provided two scores. Graders were asked to provide 1 categorical '''BROAD''' score with 3 categories: NS,SS,VS as explained below, and one '''FINE''' score (in the range from 0 to 100). A description and analysis is provided below.<br />
<br />
The systems read in 30 second audio clips as their raw data. The same 30 second clips were used in the grading stage. <br />
<br />
<br />
=== General Legend ===<br />
<br />
<br />
==== Team ID ====<br />
{| border="1" cellspacing="0" style="text-align: left; width: 800px;"<br />
|- style="background: yellow;"<br />
! width="80" | Sub code <br />
! width="200" | Submission name <br />
! width="80" style="text-align: center;" | Abstract <br />
! width="440" | Contributors<br />
|-<br />
! DM6<br />
| DM6 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/DM6.pdf PDF] || [http://www.iam.ecs.soton.ac.uk/ Franz de Leon], [http://www.iam.ecs.soton.ac.uk/ Kirk Martinez]<br />
|-<br />
! DM7<br />
| DM7 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/DM7.pdf PDF] || [http://www.iam.ecs.soton.ac.uk/ Franz de Leon], [http://www.iam.ecs.soton.ac.uk/ Kirk Martinez]<br />
|-<br />
! GT3<br />
| MarsyasSimilarity || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/GT3.pdf PDF] || [http://www.cs.uvic.ca/~gtzan George Tzanetakis] <br />
|- <br />
! JR2<br />
| modulationSim || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/JR2.pdf PDF] || [http://mirlab.org/jmzen0921 Jia-Min Ren], [http://mirlab.org/jang Jyh-Shing Roger Jang]<br />
|- <br />
! NHHL1<br />
| AMSR_2012_1 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/NHHL1.pdf PDF] || [http://11471178.net/ Byeong-jun Han], [http://marg.snu.ac.kr/people Kyogu Lee],[http://ccrma.stanford.edu/~juhan Juhan Nam],[http://ccrma.stanford.edu/~jorgeh/ Jorge Herrera]<br />
|- <br />
! NHHL2<br />
| AMSR_2012_2 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/NHHL2.pdf PDF] || [http://11471178.net/ Byeong-jun Han], [http://marg.snu.ac.kr/people Kyogu Lee],[http://ccrma.stanford.edu/~juhan Juhan Nam],[http://ccrma.stanford.edu/~jorgeh/ Jorge Herrera]<br />
|-<br />
! PS1<br />
| PS09 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2011/PS1.pdf PDF] || [http://www.ofai.at/~dominik.schnitzer Dominik Schnitzer], [http://www.cp.jku.at/ Tim Pohle]<br />
|-<br />
! RW4<br />
| modulationSimFrameUBM || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/RW4.pdf PDF] || [http://mirlab.org/jmzen0921 Jia-Min Ren],[http://mirlab.org/new/ Ming-Ju Wu],[http://mirlab.org/jang Jyh-Shing Roger Jang]<br />
|-<br />
! SSKP1<br />
| cbmr_sim_2010 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSPK1.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Tim Pohle]<br />
|-<br />
! SSKS2<br />
| cbmr_sim_2011 || style="text-align: center;" | [https://www.music-ir.org/mirex/abstracts/2012/SSKS2.pdf PDF] || [http://www.seyerlehner.info Klaus Seyerlehner], [http://www.cp.jku.at Markus Schedl], [http://www.cp.jku.at Peter Knees], [http://www.cp.jku.at/ Reinhard Sonnleitner]<br />
|- <br />
|}<br />
<br />
====Broad Categories====<br />
'''NS''' = Not Similar<br /><br />
'''SS''' = Somewhat Similar<br /><br />
'''VS''' = Very Similar<br /><br />
<br />
=====Understanding Summary Measures=====<br />
'''Fine''' = Has a range from 0 (failure) to 100 (perfection). <br /><br />
'''Broad''' = Has a range from 0 (failure) to 2 (perfection) as each query/candidate pair is scored with either NS=0, SS=1 or VS=2. <br /><br />
<br />
==Human Evaluation==<br />
===Overall Summary Results===<br />
<br />
<csv p=3>2012/ams/summary_evalutron.csv</csv><br />
<br />
<br /><br />
'''Note:RZ1''' is the random result for comparing purpose.<br />
<br />
===Friedman's Tests===<br />
====Friedman's Test (FINE Scores)====<br />
The Friedman test was run in MATLAB against the '''Fine''' summary data over the 100 queries.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);<br />
<br />
<csv p=3>2012/ams/evalutron.fine.friedman.tukeyKramerHSD.csv</csv><br />
<br />
[[File:evalutron.fine.friedman.tukeyKramerHSD.png|500px]]<br />
<br />
====Friedman's Test (BROAD Scores)====<br />
The Friedman test was run in MATLAB against the '''BROAD''' summary data over the 100 queries.<br /><br />
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);<br />
<br />
<csv p=3>2012/ams/evalutron.cat.friedman.tukeyKramerHSD.csv</csv><br />
<br />
[[File:evalutron.cat.friedman.tukeyKramerHSD.png|500px]]<br />
<br />
===Summary Results by Query===<br />
====FINE Scores====<br />
These are the mean FINE scores per query assigned by Evalutron graders. The FINE scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 and 100. A perfect score would be 100. Genre labels have been included for reference. <br />
<br />
<csv p=1>2012/ams/fine_scores.csv</csv><br />
<br />
====BROAD Scores====<br />
These are the mean BROAD scores per query assigned by Evalutron graders. The BROAD scores for the 5 candidates returned per algorithm, per query, have been averaged. Values are bounded between 0 (not similar) and 2 (very similar). A perfect score would be 2. Genre labels have been included for reference. <br />
<br />
<csv p=1>2012/ams/cat_scores.csv</csv><br />
<br />
===Raw Scores===<br />
The raw data derived from the Evalutron 6000 human evaluations are located on the [[2012:Audio Music Similarity and Retrieval Raw Data]] page.<br />
<br />
==Metadata and Distance Space Evaluation==<br />
The following reports provide evaluation statistics based on analysis of the distance space and metadata matches and include:<br />
* Neighbourhood clustering by artist, album and genre<br />
* Artist-filtered genre clustering<br />
* How often the triangular inequality holds<br />
* Statistics on 'hubs' (tracks similar to many tracks) and orphans (tracks that are not similar to any other tracks at N results).<br />
<br />
=== Reports ===<br />
<br />
'''DM6''' = [https://music-ir.org/mirex/results/2012/ams/statistics/DM6/report.txt Franz de Leon, Kirk Martinez]<br /><br />
'''DM7''' = [https://music-ir.org/mirex/results/2012/ams/statistics/DM7/report.txt Franz de Leon, Kirk Martinez]<br /><br />
'''GT3''' = [https://music-ir.org/mirex/results/2012/ams/statistics/GT3/report.txt George Tzanetakis]<br /><br />
'''JR2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/JR2/report.txt Jia-Min Ren, Jyh-Shing Roger Jang]<br /><br />
'''NHHL1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/NHHL1/report.txt Byeong-jun Han, Kyogu Lee,Juhan Nam,Jorge Herrera]<br /><br />
'''NHHL2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/NHHL2/report.txt Byeong-jun Han, Kyogu Lee,Juhan Nam,Jorge Herrera]<br /><br />
'''PS1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/PS1/report.txt Dominik Schnitzer, Tim Pohle]<br /><br />
'''RW4''' = [https://music-ir.org/mirex/results/2012/ams/statistics/RW4/report.txt Jia-Min Ren,Ming-Ju Wu,Jyh-Shing Roger Jang]<br /><br />
'''SSKP1''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKP1/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br /><br />
'''SSKP2''' = [https://music-ir.org/mirex/results/2012/ams/statistics/SSKS2/report.txt Klaus Seyerlehner, Markus Schedl, Peter Knees, Tim Pohle]<br /></div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90642012:MIREX2012 Results2012-10-04T17:37:45Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; <br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* Query-by-Singing/Humming Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1a_jang/ Jang Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1b_thinkit/ ThinkIt Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task1c_ioacas/ IOACAS Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbsh/qbsh_task2_jang/ Subtask2 Dataset] &nbsp;<br />
* Query-by-Tapping Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task1_jang/ Jang Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task1_hsiao/ HSIAO Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/qbt/qbt_task2_jang/ Subtask2 Dataset] &nbsp;<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90432012:MIREX2012 Results2012-10-03T01:56:51Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; <br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]]<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012_talk:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90202012:MIREX2012 Results2012-10-02T21:05:51Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; <br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; <br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012_talk:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90192012:MIREX2012 Results2012-10-02T16:14:26Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
'''NOTE: There was an issue in the evaluation codes' interactions with the submission system database, and some submissions may be pointing to incorrect abstracts, etc. This will be resolved shorty.'''<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012_talk:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]]<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90162012:MIREX2012 Results2012-10-02T06:08:26Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX10) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]] (COMING SOON)<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90152012:MIREX2012 Results2012-10-02T06:07:45Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero (MIREX2010) Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]] (COMING SOON)<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90142012:MIREX2012 Results2012-10-02T04:50:06Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]] (COMING SOON)<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90132012:MIREX2012 Results2012-10-02T02:03:34Z<p>AndreasEhmann: </p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mrx/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/mcg/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero Ground-truth] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;(COMING SOON)<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]] (COMING SOON)<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=MediaWiki:Sidebar&diff=9012MediaWiki:Sidebar2012-10-02T01:50:33Z<p>AndreasEhmann: </p>
<hr />
<div>* mirex by year<br />
** 2012:Main_Page|MIREX 2012<br />
** 2011:Main_Page|MIREX 2011<br />
** 2010:Main_Page|MIREX 2010<br />
** 2009:Main_Page|MIREX 2009<br />
** 2008:Main_Page|MIREX 2008<br />
** 2007:Main_Page|MIREX 2007<br />
** 2006:Main_Page|MIREX 2006<br />
** 2005:Main_Page|MIREX 2005<br />
<br />
*results by year<br />
**2012:MIREX2012_Results| MIREX 2012 Results<br />
**2011:MIREX2011_Results| MIREX 2011 Results<br />
**2010:MIREX2010_Results| MIREX 2010 Results<br />
**2009:MIREX2009_Results| MIREX 2009 Results <br />
**2008:MIREX2008_Results| MIREX 2008 Results <br />
**2007:MIREX2007_Results| MIREX 2007 Results <br />
**2006:MIREX2006_Results| MIREX 2006 Results <br />
**2005:MIREX2005_Results| MIREX 2005 Results <br />
<br />
*2011 account request<br />
**Special:RequestAccount | Account Request Form<br />
<br />
* SEARCH<br />
<br />
* navigation<br />
** mainpage|MIREX CENTRAL HOME<br />
** portal-url|portal<br />
** currentevents-url|currentevents<br />
** recentchanges-url|recentchanges<br />
** randompage-url|randompage<br />
** helppage|help<br />
<br />
* TOOLBOX<br />
* LANGUAGES</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=MIREX_HOME&diff=9011MIREX HOME2012-10-02T01:47:43Z<p>AndreasEhmann: </p>
<hr />
<div>__TOC__<br />
<br />
== Current MIREX Wiki (2012) ==<br />
You can view the current 2012 content here: [[2012:Main_Page]]<br />
<br />
== Introduction ==<br />
The Music Information Retrieval Evaluation eXchange (MIREX) is an annual evaluation campaign for Music Information Retrieval (MIR) algorithms, coupled to the [http://www.ismir.net International Society (and Conference) for Music Information Retrieval (ISMIR)]. MIREX is hosted by the [https://www.music-ir.org/evaluation/ International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL)] at the [http://www.lis.illinois.edu/ Graduate School of Library Information Sciences (GSLIS)] which is part of the [http://www.illinois.edu/ University of Illinois at Urbana-Champaign (UIUC)].<br />
<br />
<br />
Current and Future MIREXs are in part facilitated by the work of the [https://nema.lis.illinois.edu/ Networked Environment for Music Analysis (NEMA) project]. The NEMA projects aims to automate and expose the workings of MIREX and MIR experimentation/evaluation to the MIR community, helping to deal with issues of collection/data/code sharing within the community by handling issues relating to copyright and IP restrictions by allowing MIR researchers to work (remotely) with resources without having to obtain licenses to the content/code.<br />
<br />
<br />
MIR tasks evaluated at past MIREXs include:<br />
* [[Audio Train/Test Tasks]]<br />
** Audio Artist Identification<br />
** Audio Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* Symbolic Genre Classification<br />
* [[Audio Onset Detection]]<br />
* [[Audio Key Detection]]<br />
* Symbolic Key Detection<br />
* [[Audio Tag Classification]]<br />
* [[Audio Cover Song Identification]]<br />
* [[Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[Query by Singing/Humming]]<br />
* [[Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[Audio Chord Estimation]]<br />
* [[Audio Melody Extraction]]<br />
* [[Query by Tapping]]<br />
* [[Audio Beat Tracking]]<br />
* [[Audio Music Similarity and Retrieval]]<br />
* [[Symbolic Melodic Similarity]]<br />
* [[Structural Segmentation]]<br />
* [[Audio Drum Detection]]<br />
* [[Audio Tempo Extraction]]<br />
<br />
== Recent Changes ==<br />
We recently have merged all current and previous iterations of the MIREX wiki into a single wiki installation to make it easier to manage. All the pages, images, abstracts, and images have been migrated, but some links and images may still be broken. We're currently manually inspecting all pages, but would appreciate your help in correcting any errors you see.<br />
<br />
Content on the wiki is now organized into mediawiki namespaces, one for each year. You can view the current 2012 content here: [[2012:Main_Page]]<br />
<br />
Similarly for previous content.<br />
* [[2012:Main_Page]]<br />
* [[2011:Main_Page]]<br />
* [[2010:Main_Page]]<br />
* [[2009:Main_Page]]<br />
* [[2008:Main_Page]]<br />
* [[2007:Main_Page]]<br />
* [[2006:Main_Page]]<br />
* [[2005:Main_Page]]<br />
<br />
All links to older wiki content will be redirected to this new wiki, and should take you to the correct page on the new installation, but please update any bookmarks or links you may have which point into current or old wiki content.</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=MediaWiki:Sidebar&diff=9010MediaWiki:Sidebar2012-10-02T01:47:18Z<p>AndreasEhmann: </p>
<hr />
<div>* mirex by year<br />
** 2012:Main_Page|MIREX 2012<br />
** 2011:Main_Page|MIREX 2011<br />
** 2010:Main_Page|MIREX 2010<br />
** 2009:Main_Page|MIREX 2009<br />
** 2008:Main_Page|MIREX 2008<br />
** 2007:Main_Page|MIREX 2007<br />
** 2006:Main_Page|MIREX 2006<br />
** 2005:Main_Page|MIREX 2005<br />
<br />
*results by year<br />
**2012:MIREX2011_Results| MIREX 2012 Results<br />
**2011:MIREX2011_Results| MIREX 2011 Results<br />
**2010:MIREX2010_Results| MIREX 2010 Results<br />
**2009:MIREX2009_Results| MIREX 2009 Results <br />
**2008:MIREX2008_Results| MIREX 2008 Results <br />
**2007:MIREX2007_Results| MIREX 2007 Results <br />
**2006:MIREX2006_Results| MIREX 2006 Results <br />
**2005:MIREX2005_Results| MIREX 2005 Results <br />
<br />
*2011 account request<br />
**Special:RequestAccount | Account Request Form<br />
<br />
* SEARCH<br />
<br />
* navigation<br />
** mainpage|MIREX CENTRAL HOME<br />
** portal-url|portal<br />
** currentevents-url|currentevents<br />
** recentchanges-url|recentchanges<br />
** randompage-url|randompage<br />
** helppage|help<br />
<br />
* TOOLBOX<br />
* LANGUAGES</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX2012_Results&diff=90092012:MIREX2012 Results2012-10-02T01:27:14Z<p>AndreasEhmann: Created page with "==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->== [http://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Ov..."</p>
<hr />
<div>==OVERALL RESULTS POSTERS <!--(First Version: Will need updating as last runs are completed)-->==<br />
<br />
[https://www.music-ir.org/mirex/results/2012/mirex_2012_poster.pdf MIREX 2012 Overall Results Posters (PDF)]<br />
<br />
==Results by Task ==<br />
<br />
===Train-Test Task Set===<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/composer_report/ Audio Classical Composer Identification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/latin_report/ Audio Latin Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mood_report/index.html Audio Music Mood Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
* [https://www.music-ir.org/nema_out/mirex2012/results/act/mixed_report/ Audio Mixed Popular Genre Classification Results ]&nbsp;&nbsp; (COMING SOON)<br />
<br />
===Other Tasks===<br />
<br />
* Audio Beat Tracking Results <br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/mck/ MCK Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/maz/ MAZ Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/abt/smc/ SMC Dataset] &nbsp;<br />
* Audio Chord Detection Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/ MIREX Dataset] &nbsp;<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ace/ McGill Dataset] &nbsp;<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/akd/ Audio Key Detection Results] &nbsp;<br />
* Audio Melody Extraction Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/adc04/ ADC04 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex05/ MIREX05 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/indian08/ INDIAN08 Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_0dB/ MIREX09 0dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_m5dB/ MIREX09 -5dB Dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ame/mirex09_p5dB/ MIREX09 +5dB Dataset] &nbsp;(COMING SOON)<br />
* [[2012:Audio_Music_Similarity_and_Retrieval_Results | Audio Music Similarity and Retrieval Results]] (COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/aod/ Audio Onset Detection Results] &nbsp;<br />
* Audio Tag Classification Results<br />
** Major Miner Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask1_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
** Mood Tag dataset<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/bin/ Binary relevance (classification evaluation)] &nbsp;(COMING SOON)<br />
*** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/atg/subtask2_report/aff/ Affinity estimation evaluation] &nbsp;(COMING SOON)<br />
* [https://nema.lis.illinois.edu/nema_out/mirex2012/results/ate/ Audio Tempo Estimation Results] &nbsp;<br />
* [[2012:Multiple_Fundamental_Frequency_Estimation_&_Tracking_Results | Multiple Fundamental Frequency Estimation & Tracking Results]] (COMING SOON)<br />
* Music Structure Segmentation Results<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx09/ MIREX09 dataset] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_1/ RWC dataset - Quaero Ground-truth] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/mrx10_2/ RWC dataset - Original RWC Ground-truth] &nbsp;(COMING SOON)<br />
** [https://nema.lis.illinois.edu/nema_out/mirex2012/results/struct/sal/ SALAMI dataset] &nbsp;(COMING SOON)<br />
* [[2012:Query-by-Singing/Humming_Results | Query-by-Singing/Humming Results]] (COMING SOON)<br />
* [[2012:Query-by-Tapping_Results| Query-by-Tapping Results]] (COMING SOON)<br />
*[[2012:Real-time_Audio_to_Score_Alignment_(a.k.a._Score_Following)_Results | Real-time Audio to Score Alignment (a.k.a. Score Following) Results ]] (COMING SOON)<br />
* [[2012:Symbolic_Melodic_Similarity_Results | Symbolic Melodic Similarity Results]] (COMING SOON)<br />
<br />
== Machine Specifications ==<br />
TBA<br />
<br />
== Runtime for Submissions Run by NEMA DIY ==<br />
<br />
TBA<br />
<br />
[[Category:Results]]</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Classification_(Train/Test)_Tasks&diff=87452012:Audio Classification (Train/Test) Tasks2012-06-07T21:39:30Z<p>AndreasEhmann: /* Participation in previous years and Links to Results */</p>
<hr />
<div>= Audio Classification (Test/Train) tasks =<br />
<br />
== Description ==<br />
<br />
Many tasks in music classification can be characterized into a two-stage process: training classification models using labeled data and testing the models using new/unseen data. Therefore, we propose this "meta" task which includes various audio classification tasks that follow this Train/Test process. For MIREX 2012, five classification sub-tasks are included: <br />
<br />
* Audio Classical Composer Identification<br />
* Audio US Pop Music Genre Classification <br />
* Audio Latin Music Genre Classification <br />
* Audio Mood Classification<br />
<br />
All five classification tasks were conducted in previous MIREX runs (please see ). This page presents the evaluation of these tasks, including the datasets as well as the submission rules and formats.<br />
<br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
=== Audio Classical Composer Identification ===<br />
This dataset requires algorithms to classify music audio according to the composer of the track (drawn from a collection of performances of a variety of classical music genres). The collection used at MIREX 2009 will be re-used.<br />
<br />
Collection statistics: <br />
* 2772 30-second 22.05 kHz mono wav clips<br />
* 11 "classical" composers (252 clips per composer), including:<br />
** Bach<br />
** Beethoven<br />
** Brahms<br />
** Chopin<br />
** Dvorak<br />
** Handel<br />
** Haydn<br />
** Mendelssohn<br />
** Mozart<br />
** Schubert<br />
** Vivaldi<br />
<br />
<br />
=== Audio US Pop Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of US Pop music tracks). The MIREX 2007 Genre dataset will be re-used, which was drawn from the USPOP 2002 and USCRAP collections.<br />
<br />
Collection statistics: <br />
* 7000 30-second audio clips in 22.05kHz mono WAV format <br />
* 10 genres (700 clips from each genre), including:<br />
** Blues<br />
** Jazz<br />
** Country/Western<br />
** Baroque<br />
** Classical<br />
** Romantic<br />
** Electronica<br />
** Hip-Hop<br />
** Rock<br />
** HardRock/Metal<br />
<br />
<br />
=== Audio Latin Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of Latin popular and dance music, sourced from Brazil and hand labeled by music experts). Carlos Silla's (cns2 (at) kent (dot) ac (dot) uk) Latin popular and dance music dataset [http://ismir2008.ismir.net/papers/ISMIR2008_106.pdf] will be re-used. This collection is likely to contain a greater number of styles of music that will be differentiated by rhythmic characteristics than the MIREX 2007 dataset.<br />
<br />
Collection statistics: <br />
* 3,227 audio files in 22.05kHz mono WAV format <br />
* 10 Latin music genres, including:<br />
** Axe<br />
** Bachata<br />
** Bolero<br />
** Forro <br />
** Gaucha<br />
** Merengue<br />
** Pagode<br />
** Sertaneja<br />
** Tango<br />
<br />
<br />
=== Audio Mood Classification ===<br />
This dataset requires algorithms to classify music audio according to the mood of the track (drawn from a collection of production msuic sourced from the APM collection [http://www.apmmusic.com]). The MIREX 2007 Mood Classification dataset [http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf] will be re-used. <br />
<br />
Collection statistics: <br />
* 600 30 second audio clips in 22.05kHz mono WAV format selected from the APM collection [http://www.apmmusic.com], and labeled by human judges using the Evalutron6000 system. <br />
* 5 mood categories [http://ismir2007.ismir.net/proceedings/ISMIR2007_p067_hu.pdf] each of which contains 120 clips:<br />
**Cluster_1: passionate, rousing, confident,boisterous, rowdy <br />
**Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured <br />
**Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding <br />
**Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry <br />
**Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral<br />
<br />
== Audio Formats ==<br />
For all datasets, participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 22 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV<br />
<br />
<br />
== Evaluation ==<br />
This section first describes evaluation methods common to all the datasets, then specifies settings unique to each of the tasks. <br />
<br />
Participating algorithms will be evaluated with 3-fold cross validation. For '''Artist Identification''' and '''Classical Composer Classification''', album filtering will be used the test and training splits, i.e. training and test sets will contain tracks from different albums; for '''US Pop Genre Classification''' and '''Latin Genre Classification''', artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists. <br />
<br />
The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.<br />
<br />
Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance. <br />
<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
== Submission Format ==<br />
=== File I/O Format ===<br />
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below: <br />
<br />
<br />
==== Feature extraction list file ====<br />
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Training list file ====<br />
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.<br />
<br />
I.e. <br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav rock<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
==== Test (classification) list file ====<br />
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.<br />
<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Classification output file ====<br />
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.<br />
<br />
I.e.<br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav classical<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
=== Submission calling formats ===<br />
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.<br />
<br />
Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.<br />
<br />
Executables will have to accept the paths to the aforementioned list files as command line parameters.<br />
<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.<br />
<br />
<br />
==== Example submission calling formats ====<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt <br />
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt <br />
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations. <br />
<br />
extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
=== Packaging submissions ===<br />
<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2012 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
=== Time and hardware limits ===<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 24 hours will be imposed on feature extraction times.<br />
<br />
A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.<br />
<br />
=== Potential Participants ===<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Tempo_Estimation&diff=87442012:Audio Tempo Estimation2012-06-07T21:38:44Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
This task compares current methods for the extraction of tempo from musical audio. We distinguish between notated tempo and perceptual tempo and will test for the extraction of perceptual tempo. <br />
<br />
We differentiate between notated tempo and perceived tempo. If you have the notated tempo (e.g., from the score) it is straightforward attach a tempo annotation to an excerpt and run a contest for algorithms to predict the notated tempo. For excerpts for which we have no "official" tempo annotation, we can also annotate the *perceived* tempo. This is not a straightforward task and needs to be done carefully. If you ask a group of listeners (including skilled musicians) to annotate the tempo of music excerpts, they can give you different answers (they tap at different metrical levels) if they are unfamiliar with the piece. For some excerpts the perceived pulse or tempo is less ambiguous and everyone taps at the same metrical level, but for other excerpts the tempo can be quite ambiguous and you get a complete split across listeners.<br />
<br />
The annotation of perceptual tempo can take several forms: a probability density function as a function of tempo; a series of tempos, ranked by their respective perceptual salience; etc. These measures of perceptual tempo can be used as a ground truth on which to test algorithms for tempo extraction. The dominant perceived tempo is sometimes the same as the notated tempo but not always. A piece of music can "feel" faster or slower than it's notated tempo in that the dominant perceived pulse can be a metrical level higher or lower than the notated tempo.<br />
<br />
There are several reasons to examine the perceptual tempo, either in place of or in addition to the notated tempo. For many applications of automatic tempo extractors, the perceived tempo of the music is more relevant than the notated tempo. An automatic playlist generator or music navigator, for instance, might allow listeners to select or filter music by its (automatically extracted) tempo. In this case, the "feel", or perceptual tempo may be more relevant than the notated tempo. An automatic DJ apparatus might also perform better with a representation of perceived tempo rather than notated tempo.<br />
<br />
A more pragmatic reason for using perceptual tempo rather than notated tempo as a ground truth for our contest is that we simply do not have the notated tempo of our test set. If we notate it by having a panel of expert listeners tap along and label the excerpts, we are by default dealing with the perceived tempo. The handling of this data as ground truth must be done with care.<br />
<br />
<br />
== Data ==<br />
=== Collections ===<br />
MIREX 2006 Tempo dataset collected by Martin F. McKinney (Philips) and Dirk Moelants (IPEM, Ghent University). Composed of 160 30-second clips in WAV format with annotated tempos. <br />
<br />
<br />
=== Audio Formats ===<br />
The data are monophonic sound files, with the associated onset times and data about the annotation robustness.<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
* 30 second clips<br />
<br />
<br />
== Submission Format ==<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
<br />
=== Input data ===<br />
Individual audio files in WAV format (30-second clips drawn from the 140 unseen tracks in the dataset). The audio recordings were selected to provide a stable tempo value, a wide distribution of tempi values, and a large variety of instrumentation and musical styles. About 20% of the files contain non-binary meters, and a small number of examples contain changing meters.<br />
<br />
<br />
=== Output Data ===<br />
Submitted programs should output two tempi (a slower tempo, T1, and a faster tempo, T2) as well as the strength of T1 relative to T2 (0-1). The relative strength ST2 (not output) is simply 1 - ST1. The tempo estimates from each algorithm should be written to a text file in the following format:<br />
<br />
T1<tab>T2<tab>ST1<br />
<br />
E.g.<br />
60 180 0.7<br />
<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the tempo estimation detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as ''%input'' and the output file path and name as ''%output'', a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
or<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
<br />
=== README File ===<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
<br />
== Evaluation Procedures ==<br />
<br />
This section focuses on the mechanics of the method while we discuss the data (music excerpts and perceptual data) in the next section. There are two general steps to the method: 1) collection of perceptual tempo annotations; and 2) evaluation of tempo extraction algorithms.<br />
<br />
=== Perceptual tempo data collection ===<br />
<br />
The following procedure is described in more detail in McKinney and Moelants (2004) and Moelants and McKinney (2004). Listeners were asked to tap to the beat of a series of musical excerpts. Responses were collected and their perceived tempo was calculated. For each excerpt, a distribution of perceived tempo was generated. A relatively simple form of perceived tempo was proposed for this contest: The two highest peaks in the perceived tempo distribution for each excerpt were taken, along with their respective heights (normalized to sum to 1.0) as the two tempo candidates for that particular excerpt. The height of a peak in the distribution is assumed to represent the perceptual salience of that tempo. <br />
<br />
==== References ====<br />
* McKinney, M.F. and Moelants, D. (2004), Deviations from the resonance theory of tempo induction, Conference on Interdisciplinary Musicology, Graz. URL: http://www-gewi.uni-graz.at/staff/parncutt/cim04/CIM04_paper_pdf/McKinney_Moelants_CIM04_proceedings_t.pdf<br />
* Moelants, D. and McKinney, M.F. (2004), Tempo perception and musical content: What makes a piece slow, fast, or temporally ambiguous? International Conference on Music Perception & Cognition, Evanston, IL. URL: http://icmpc8.umn.edu/proceedings/ICMPC8/PDF/AUTHOR/MP040237.PDF <br />
<br />
=== Evaluation of tempo extraction algorithms ===<br />
Algorithms will process musical excerpts and return the following data: Two tempi in BPM (T1 and T2, where T1 is the slower of the two tempi). For a given algorithm, the performance, P, for each audio excerpt will be given by the following equation:<br />
<br />
P = ST1 * TT1 + (1 - ST1) * TT2<br />
<br />
where ST1 is the relative perceptual strength of T1 (given by groundtruth data, varies from 0 to 1.0), TT1 is the ability of the algorithm to identify T1 to within 8%, and TT2 is the ability of the algorithm to identify T2 to within 8%. No credit will be given for tempi other than T1 and T2.<br />
<br />
The algorithm with the best average P-score will achieve the highest rank in the task. <br />
<br />
<br />
== Relevant Test Collections ==<br />
We will use a collection of 160 musical exerpts for the evaluation procedure. 40 of the excerpts have been taken from one of McKinney/Moelants previous experiments (See McKinney/Moelants ICMPC paper above).<br />
<br />
Excerpts were selected to provide:<br />
<br />
* stable tempo within each excerpt<br />
* a good distribution of tempi across excerpts<br />
* a large variety of instrumentation and beat strengths (with and without percussion)<br />
* a variation of musical styles, including many non-western styles<br />
* the presence of non-binary meters (about 20% have a ternary element and there are a few examples with odd or changing meter). <br />
<br />
We will provide 20 excerpts with ground truth data for participants to try/tune their algorithms before submission. The remaining 140 excerpts will be novel to all participants.<br />
<br />
<br />
===Practice Data===<br />
You can find it here:<br />
<br />
https://www.music-ir.org/evaluation/MIREX/data/2006/beat/<br />
<br />
User: beattrack Password: b34trx<br />
<br />
https://www.music-ir.org/evaluation/MIREX/data/2006/tempo/<br />
<br />
User: tempo Password: t3mp0<br />
<br />
Data has been uploaded in both .tgz and .zip format.<br />
<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 8 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.<br />
<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Structural_Segmentation&diff=87432012:Structural Segmentation2012-06-07T21:38:26Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
<br />
The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one of the most important musical parameters. It is furthermore special because musical structure -- especially in popular music genres (e.g. verse, chorus, etc.) -- is accessible to everybody: it needs no particular musical knowledge. This task was first run in 2009.<br />
<br />
== Data == <br />
<br />
=== Collections ===<br />
* The MIREX 2009 Collection: 297 pieces, most of it derived from the work of the Beatles.<br />
<br />
* MIREX 2010 RWC collection. 100 pieces of popular music. There are two ground truths. The first is the one originally included with the RWC dataset. The explanation of the second set of annotations can be found at http://hal.inria.fr/docs/00/47/34/79/PDF/PI-1948.pdf. The second set of annotations contains no labels for segments, but rather provides an annotation of segment boundaries.<br />
<br />
* MIREX 2012 dataset. The new data set contains over 1,000 annotated pieces covering a range of musical styles. The majority of the pieces have been annotated by two independent annotators. <br />
<br />
=== Audio Formats ===<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
<br />
== Submission Format ==<br />
<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
=== Input Data ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44.1 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
=== Output Data ===<br />
<br />
The structural segmentation algorithms will return the segmentation in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.<br />
<br />
=== Output File Format (Structural Segmentation) ===<br />
<br />
The Structural Segmentation output file format is a tab-delimited ASCII text format. This is the same as Chris Harte's chord labelling files (.lab), and so is the same format as the ground truth as well. Onset and offset times are given in seconds, and the labels are simply letters: 'A', 'B', ... with segments referring to the same structural element having the same label.<br />
<br />
Three column text file of the format<br />
<br />
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n<br />
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n<br />
...<br />
<br />
where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:<br />
<br />
0.000 5.223 A<br />
5.223 15.101 B<br />
15.101 20.334 A<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the structural segmentation on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
=== README File ===<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:<br />
<br />
foobar -param1 .1 -i %input -o %output<br />
<br />
For a submission using MATLAB, the README file could look like:<br />
<br />
matlab -r "foobar(.1,'%input','%output');quit;"<br />
<br />
== Evaluation Procedures ==<br />
At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".<br />
<br />
=== Boundary retrieval ===<br />
'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.<br />
<br />
'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])<br />
<br />
=== Frame clustering ===<br />
Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets <math>P_E</math> (for the system result) and <math>P_A</math> (for the ground truth). The ''pairwise precision rate'' can be calculated by <math>P = \frac{|P_E \cap P_A|}{|P_E|}</math>, ''pairwise recall rate'' by <math>R = \frac{|P_E \cap P_A|}{|P_A|}</math>, and ''pairwise F-measure'' by <math>F=\frac{2 P R}{P + R}</math>. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])<br />
<br />
=== Normalised conditional entropies ===<br />
Over- and under segmentation based evaluation measures proposed in [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich ISMIR2008].<br />
Structure descriptions are represented as frame sequences with the associated cluster information (similar to the Frame clustering measure). Confusion matrix between the labels in ground truth and the result is calculated. The matrix C is of size |L_A| * |L_E|, i.e., number of unique labels in the ground truth times number of unique labels in the result. From the confusion matrix, the joint distribution is calculated by normalising the values with the total number of frames F:<br />
<br />
<math>p_{i,j} = C_{i,j} / F</math><br />
<br />
Similarly, the two marginals are calculated:<br />
<br />
<math>p_i^a = \sum_{j=1}^{|L_E|} C{i,j}/F</math>, and<br />
<br />
<math>p_j^e = \sum_{i=1}^{|L_A|} C{i,j}/F</math><br />
<br />
Conditional distributions:<br />
<br />
<math>p_{i,j}^{a|e} = C_{i,j} / \sum_{i=1}^{|L_A|} C{i,j}</math>, and<br />
<br />
<math>p_{i,j}^{e|a} = C_{i,j} / \sum_{j=1}^{|L_E|} C{i,j}</math><br />
<br />
The conditional entropies will then be<br />
<br />
<math>H(E|A) = - \sum_{i=1}^{|L_A|} p_i^a \sum_{j=1}^{|L_E|} p_{i,j}^{e|a} \log_2(p_{i,j}^{e|a})</math>, and<br />
<br />
<math>H(A|E) = - \sum_{j=1}^{|L_E|} p_j^e \sum_{i=1}^{|L_A|} p_{i,j}^{a|e} \log_2(p_{i,j}^{a|e})</math><br />
<br />
The final evaluation measures will then be the oversegmentation score<br />
<br />
<math>S_O = 1 - \frac{H(E|A)}{\log_2(|L_E|)}</math> , and the undersegmentation score<br />
<br />
<math>S_U = 1 - \frac{H(A|E)}{\log_2(|L_A|)}</math><br />
<br />
== Relevant Development Collections == <br />
*Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].<br />
<br />
*Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).<br />
<br />
These public corpora give a combined 220 songs.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 24 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.<br />
<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Beat_Tracking&diff=87422012:Audio Beat Tracking2012-06-07T21:38:02Z<p>AndreasEhmann: /* Collections */</p>
<hr />
<div>== Description ==<br />
<br />
The aim of the automatic beat tracking task is to track each beat locations in a collection of sound files. Unlike the Audio Tempo Extraction task, which aim is to detect tempi for each file, the beat tracking task aims at detecting all beat locations in recordings. The algorithms will be evaluated in terms of their accuracy in predicting beat locations annotated by a group of listeners. <br />
<br />
== Data ==<br />
=== Collections ===<br />
The original 2006 dataset contains 160 30-second excerpts (WAV format) used for the Audio Tempo and Beat contests in 2006. Beat locations have been annotated in each excerpt by 40 different listeners (39 listeners for a few excerpts. The length of each excerpt is 30 seconds. These audio recordings were selected to provide a stable tempo value, a wide distribution of tempi values, and a large variety of instrumentation and musical styles. About 20% of the files contain non-binary meters, and a small number of examples contain changing meters. One disadvantage of using this set for beat tracking is that the tempi are rather stable and this set will not test beat-tracking algorithms in their ability to track tempo changes.<br />
<br />
The second collection is comprised of 367 Chopin Mazurkas, represented as full audio tracks (WAV format). The Mazurka dataset contains tempo changes so it will evaluate the ability of algorithms to track these.<br />
<br />
The third collection was assembled and donated in 2012, and consists of what should be fairly challenging pieces for beat trackers.<br />
<br />
=== Audio Formats ===<br />
<br />
The data are monophonic sound files, with the associated onset times and data about the annotation robustness.<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
* file length between 2 and 36 seconds (total time: 14 minutes) <br />
<br />
<br />
== Submission Format ==<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
=== Input Data ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44.1 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
=== Output Data ===<br />
<br />
The beat tracking algorithms will return beat-times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.<br />
<br />
=== Output File Format (Audio Beat tracking) ===<br />
<br />
The Beat Tracking output file format is an ASCII text format. Each beat time is specified, in seconds, on its own line. Specifically, <br />
<br />
<beat time(in seconds)>\n<br />
<br />
where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:<br />
<br />
0.243<br />
0.486<br />
0.729<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
<br />
=== README File ===<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
For instance, to test the program foobar with different values for parameters param1, the README file would look like:<br />
<br />
foobar -param1 .1 -i %input -o %output<br />
foobar -param1 .15 -i %input -o %output<br />
foobar -param1 .2 -i %input -o %output<br />
foobar -param1 .25 -i %input -o %output<br />
foobar -param1 .3 -i %input -o %output<br />
...<br />
<br />
For a submission using MATLAB, the README file could look like:<br />
<br />
matlab -r "foobar(.1,'%input','%output');quit;"<br />
matlab -r "foobar(.15,'%input','%output');quit;"<br />
matlab -r "foobar(.2,'%input','%output');quit;" <br />
matlab -r "foobar(.25,'%input','%output');quit;"<br />
matlab -r "foobar(.3,'%input','%output');quit;"<br />
...<br />
<br />
The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.<br />
<br />
== Evaluation Procedures ==<br />
<br />
The evaluation methods are taken from the beat evaluation toolbox and<br />
are described in the following technical report: <br />
<br />
M. E. P. Davies, N. Degara and M. D. Plumbley. "Evaluation methods for musical audio beat tracking algorithms". [https://music-ir.org/mirex/results/2009/beat/techreport_beateval.pdf ''Technical Report C4DM-TR-09-06''].<br />
<br />
For further details on the specifics of the methods please refer to the<br />
paper. However, here is a brief summary with appropriate references:<br />
<br />
*'''F-measure''' - the standard calculation as used in onset evaluation but<br />
with a 70ms window. <br />
<br />
S. Dixon, "Onset detection revisited," in ''Proceedings of 9th<br />
International Conference on Digital Audio Effects (DAFx)'', Montreal,<br />
Canada, pp. 133-137, 2006.<br />
<br />
S. Dixon, "Evaluation of audio beat tracking system beatroot," ''Journal<br />
of New Music Research'', vol. 36, no. 1, pp. 39-51, 2007.<br />
<br />
*'''Cemgil''' - beat accuracy is calculated using a Gaussian error function<br />
with 40ms standard deviation.<br />
<br />
A. T. Cemgil, B. Kappen, P. Desain, and H. Honing, "On tempo tracking:<br />
Tempogram representation and Kalman filtering," ''Journal Of New Music<br />
Research'', vol. 28, no. 4, pp. 259-273, 2001<br />
<br />
*'''Goto''' - binary decision of correct or incorrect tracking based on<br />
statistical properties of a beat error sequence.<br />
<br />
M. Goto and Y. Muraoka, "Issues in evaluating beat tracking systems," in<br />
''Working Notes of the IJCAI-97 Workshop on Issues in AI and Music -<br />
Evaluation and Assessment'', 1997, pp. 9-16.<br />
<br />
*'''PScore''' - McKinney's impulse train cross-correlation method as used in<br />
2006.<br />
<br />
M. F. McKinney, D. Moelants, M. E. P. Davies, and A. Klapuri,<br />
"Evaluation of audio beat tracking and music tempo extraction<br />
algorithms," ''Journal of New Music Research'', vol. 36, no. 1, pp. 1-16,<br />
2007.<br />
<br />
*'''CMLc''', '''CMLt''', '''AMLc''', '''AMLt''' - continuity-based evaluation methods based on<br />
the longest continuously correctly tracked section. <br />
<br />
S. Hainsworth, "Techniques for the automated analysis of musical audio,"<br />
Ph.D. dissertation, Department of Engineering, Cambridge University,<br />
2004.<br />
<br />
A. P. Klapuri, A. Eronen, and J. Astola, "Analysis of the meter of<br />
acoustic musical signals," IEEE Transactions on Audio, Speech and<br />
Language Processing, vol. 14, no. 1, pp. 342-355, 2006.<br />
<br />
*'''D''', '''Dg''' - information based criteria based on analysis of a beat error<br />
histogram (note the results are measured in 'bits' and not percentages),<br />
see the technical report for a description.<br />
<br />
== Relevant Development Collections ==<br />
You can find it here:<br />
<br />
https://www.music-ir.org/evaluation/MIREX/data/2006/beat/<br />
<br />
User: beattrack Password: b34trx<br />
<br />
https://www.music-ir.org/evaluation/MIREX/data/2006/tempo/<br />
<br />
User: tempo Password: t3mp0<br />
<br />
Data has been uploaded in both .tgz and .zip format.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 12 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.<br />
<br />
<br />
<br />
== Submission opening date ==<br />
<br />
Friday August 5th 2012<br />
<br />
== Submission closing date ==<br />
Friday September 2nd 2012<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Query_by_Tapping&diff=87412012:Query by Tapping2012-06-07T21:34:58Z<p>AndreasEhmann: </p>
<hr />
<div>== Overview ==<br />
The text of this section is copied from the 2009 page. Please add your comments and discussions for 2012. <br />
<br />
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have two corpora for evaluation:<br />
<br />
* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).<br />
* Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard).<br />
<br />
== Task description ==<br />
<br />
=== Subtask 1: QBT with symbolic input ===<br />
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.<br />
* '''Query files''': About 800 text files of onset time to retrieve target MIDIs in MIR_QBT. These onset files can help participant concentrate on similarity matching instead of onset detection. All onset files cannot guarantee to have perfect detection result from original wav query files.<br />
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate).<br />
<br />
=== Subtask 2: QBT with wave input ===<br />
* '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT.<br />
* '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.<br />
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate).<br />
<br />
== Disucussions for 2012 == <br />
<br />
== Command formats ==<br />
<br />
=== Indexing the MIDIs collection ===<br />
Command format should look like this: <br />
<br />
indexing %dbMidi.list% %dir_workspace_root%<br />
<br />
where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example: <br />
<br />
QBT/database/00001.mid<br />
QBT/database/00002.mid<br />
QBT/database/00003.mid<br />
QBT/database/00004.mid<br />
...<br />
<br />
Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)<br />
<br />
=== Test the query files ===<br />
The command format should be like this:<br />
<br />
qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%<br />
<br />
You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:<br />
<br />
qbtQuery/query_00001.onset 00001.mid<br />
qbtQuery/query_00002.onset 00001.mid<br />
qbtQuery/query_00003.onset 00002.mid<br />
...<br />
<br />
(Pleae refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)<br />
<br />
If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:<br />
<br />
qbtQuery/query_00001.wav 00001.mid<br />
qbtQuery/query_00002.wav 00001.mid<br />
qbtQuery/query_00003.wav 00002.mid<br />
...<br />
<br />
The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:<br />
<br />
qbtQuery/query_00001.onset: 00025 01003 02200 ... <br />
qbtQuery/query_00002.onset: 01547 02313 07653 ... <br />
qbtQuery/query_00003.onset: 03142 00320 00973 ... <br />
...<br />
<br />
And for subtask 2:<br />
<br />
qbtQuery/query_00001.wav: 00025 01003 02200 ... <br />
qbtQuery/query_00002.wav: 01547 02313 07653 ... <br />
qbtQuery/query_00003.wav: 03142 00320 00973 ... <br />
...<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Chord_Estimation&diff=87402012:Audio Chord Estimation2012-06-07T21:34:33Z<p>AndreasEhmann: </p>
<hr />
<div>[[The Utrecht Agreement on Chord Evaluation]]<br />
<br />
===Evaluation of Chord Transcriptions===<br />
<br />
Before the final description of the chord evaluation goes live here, please see the discussion based on the [[The Utrecht Agreement on Chord Evaluation]].<br />
<br />
== Description ==<br />
This task requires participants to extract or transcribe a sequence of chords from an audio music recording. For many applications in music information retrieval, extracting the harmonic structure of an audio track is very desirable, for example for segmenting pieces into characteristic segments, for finding similar pieces, or for semantic analysis of music.<br />
<br />
The extraction of the harmonic structure requires the detection of as many chords as possible in a piece. That includes the characterisation of chords with a key and type as well as a chronological order with onset and duration of the chords.<br />
<br />
Although some publications are available on this topic [1,2,3,4,5], comparison of the results is difficult, because different measures are used to assess the performance. To overcome this problem an accurately defined methodology is needed. This includes a repertory of the findable chords, a defined test set along with ground truth and unambiguous calculation rules to measure the performance.<br />
<br />
<br />
== Data ==<br />
Two datasets are used to evaluate chord transcription accuracy:<br />
<br />
=== Beatles dataset ===<br />
Christopher Harte`s Beatles dataset consisting of annotations of 12 Beatles albums.<br />
<br />
The text annotation procedure of musical chords that was used to produce this dataset is presented in [6]. <br />
<br />
=== Queen and Zweieck dataset ===<br />
Matthias Mauch's Queen and Zweieck dataset consisting of 38 songs from Queen and Zweieck.<br />
<br />
===Example ground-truth file ===<br />
The ground-truth files take the form:<br />
<br />
...<br />
41.2631021 44.2456460 B<br />
44.2456460 45.7201230 E<br />
45.7201230 47.2061900 E:7/3<br />
47.2061900 48.6922670 A<br />
48.6922670 50.1551240 A:min/b3<br />
...<br />
<br />
<br />
== Evaluation ==<br />
<br />
<br />
<br />
=== Segmentation Score ===<br />
<br />
The segmentation score will be calculated using directional hamming distance as described in [8]. An over-segmentation value (m) and an under-segmentation value (f) will be calculated and the final segmentation score will be calculated using the worst case from these two i.e:<br />
<br />
segmentation score = 1 - max(m,f)<br />
<br />
m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.-- Chrish 17:05, 9 September 2009 (UTC)<br />
<br />
=== Frame-based recall ===<br />
<br />
<br />
For recall evaluation, we may define a different chord dictionary for each level of evaluation (dyads, triads, tetrads etc). Each dictionary is a text file containing chord shorthands / interval lists of the chords that will be considered in that evaluation. The following dictionaries are proposed:<br />
<br />
For dyad comparison of major/minor chords only:<br />
<br />
N<br><br />
X:maj<br><br />
X:min<br><br />
<br />
For comparison of standard triad chords:<br />
<br />
N<br><br />
X:maj<br><br />
X:min<br><br />
X:aug<br><br />
X:dim<br><br />
X:sus2<br><br />
X:sus4<br><br />
<br />
For comparison of tetrad (quad) chords:<br />
<br />
N <br><br />
X:maj <br><br />
X:min<br><br />
X:aug<br><br />
X:dim<br><br />
X:sus2<br><br />
X:sus4<br><br />
X:maj7<br><br />
X:7<br><br />
X:maj(9)<br><br />
X:aug(7) <br><br />
X:min(7)<br><br />
X:min7<br><br />
X:min(9)<br><br />
X:dim(7)<br><br />
X:hdim7 <br><br />
X:sus4(7)<br><br />
X:sus4(b7)<br><br />
X:dim7<br><br />
<br />
<br />
For each evaluation level, the ground truth annotation is compared against the dictionary. Any chord label not belonging to the current dictionary will be replaced with an "X" in a local copy of the annotation and will not be included in the recall calculation.<br />
<br />
Note that the level of comparison in terms of intervals can be varied. For example, in a triad evaluation we can consider the first three component intervals in the chord so that a major (1,3,5) and a major7 (1,3,5,7) will be considered the same chord. For a tetrad (quad) evaluation, we would consider the first 4 intervals so major and major7 would then be considered to be different chords.<br />
<br />
For the maj/min evaluation (using the first example dictionary), using an interval comparison of 2 (dyad) will compare only the first two intervals of each chord label. This would map augmented and diminished chords to major and minor respectively (and any other symbols that had a major 3rd or minor 3rd as their first interval). Using an interval comparison of 3 with the same dictionary would keep only those chords that have major and minor triads as their first 3 intervals so augmented and diminished chords would be removed from the evaluation.<br />
<br />
After the annotation has been "filtered" using a given dictionary, it can be compared against the machine generated estimates output by the algorithm under test. The chord sequences described in the annotation and estimate text files are sampled at a given frame rate (in this case 10ms per frame) to give two sequences of chord frames which may be compared directly with each other. For calculating a hit or a miss, the chord labels from the current frame in each sequence will be compared. Chord comparison is done by converting each chord label into an ordered list of pitch classes then comparing the two lists element by element. If the lists match to the required number of intervals then a hit is recorded, otherwise the estimate is considered a miss. It should be noted that, by converting to pitch classes in the comparison, this evaluation ignores enharmonic pitch and interval spellings so the following chords (slightly silly example just for illustration) will all evaluate as identical:<br />
<br />
C:maj = Dbb:maj = C#:(b1,b3,#4)<br />
<br />
<br />
Basic recall calculation algorithm:<br />
<br />
1) filter annotated transcription using chord dictionary for a defined number of intervals<br />
<br />
2) sample annotated transcription and machine estimated transcription at 10ms intervals to create a sequence of annotation frames and estimate frames<br />
<br />
3) start at the first frame<br />
<br />
4) get chord label for current annotation frame and estimate frame<br />
<br />
5) check annotation label:<br><br />
<br />
IF symbol is 'X' (i.e. non-dictionary) <br><br />
<br />
THEN ignore frame (record number of ignored frames)<br><br />
<br />
ELSE compare annotated/estimated chords for the predefined number of intervals <br><br />
increment hit count if chords match<br><br />
<br />
ENDIF<br />
<br />
6) increment frame count <br />
<br />
7) go back to 4 until final chord frame<br />
--[[User:Chrish|Chrish]] 17:05, 9 September 2009 (UTC)<br />
<br />
<br />
== Submission Format ==<br />
<br />
=== Audio Format ===<br />
Audio tracks will be encoded as 44.1 kHz 16bit mono WAV files.<br />
<br />
<br />
=== I/O Format ===<br />
The expected output chord transcription file for participating algorithms is that proposed by Christopher Harte [6]. <br />
<br />
Hence, algorithms should output text files with a similar format to that used in the ground truth transcriptions. That is to say, they should be flat text files with chord segment labels and times arranged thus:<br />
<br />
start_time end_time chord_label<br />
<br />
with elements separated by white spaces, times given in seconds, chord labels corresponding to the syntax described in [6] and one chord segment per line. <br />
<br />
The chord root is given as a natural (A|B|C|D|E|F|G) followed by optional sharp or flat modifiers (#|b). For the evaluation process we may assume enharmonic equivalence for chord roots. For a given chord type on root X, the chord labels can be given as a list of intervals or as a shorthand notation as shown in the following table:<br />
<br />
{|border="1" cellpadding="5" cellspacing="0" align="center"<br />
|-<br />
!NAME<br />
!INTERVALS<br />
!SHORTHAND<br />
|-<br />
|-*Triads: <br />
|-<br />
|-<br />
|major<br />
|X:(1,3,5)<br />
|X or X:maj <br />
|-<br />
|-<br />
|minor<br />
|X:(1,b3,5)<br />
|X:min <br />
|-<br />
|-<br />
|diminished<br />
|X:(1,b3,b5)<br />
|X:dim<br />
|-<br />
|-<br />
|augmented<br />
|X:(1,3,#5)<br />
|X:aug<br />
|-<br />
|-<br />
|suspended4<br />
|X:(1,4,5)<br />
|X:sus4<br />
|-<br />
|-<br />
|possible 6th triad: <br />
|<br />
| <br />
|-<br />
|-<br />
|suspended2<br />
|X:(1,2,5)<br />
|X:sus2<br />
|-<br />
|-<br />
|*Quads: <br />
|<br />
| <br />
|-<br />
|-<br />
|major-major7<br />
|X:(1,3,5,7)<br />
|X:maj7<br />
|-<br />
|-<br />
|major-minor7<br />
|X:(1,3,5,b7)<br />
|X:7<br />
|-<br />
|-<br />
|major-add9<br />
|X:(1,3,5,9)<br />
|X:maj(9)<br />
|-<br />
|-<br />
|major-major7-#5<br />
|X:(1,3,#5,7)<br />
|X:aug(7)<br />
|-<br />
|-<br />
|minor-major7<br />
|X:(1,b3,5,7)<br />
|X:min(7)<br />
|-<br />
|-<br />
|minor-minor7<br />
|X:(1,b3,5,b7)<br />
|X:min7<br />
|-<br />
|-<br />
|minor-add9<br />
|X:(1,b3,5,9)<br />
|X:min(9)<br />
|-<br />
|-<br />
|minor 7/b5 (ambiguous - could be either of the following) <br />
|<br />
|<br />
|-<br />
|-<br />
|minor-major7-b5<br />
|X:(1,b3,b5,7)<br />
|X:dim(7)<br />
|-<br />
|-<br />
|minor-minor7-b5 (a half diminished-7th)<br />
|X:(1,b3,b5,b7)<br />
|X:hdim7<br />
|-<br />
|-<br />
|sus4-major7<br />
|X:(1,4,5,7)<br />
|X:sus4(7)<br />
|-<br />
|-<br />
|sus4-minor7<br />
|X:(1,4,5,b7)<br />
|X:sus4(b7)<br />
|-<br />
|-<br />
|omitted from list on wiki:<br />
|<br />
|<br />
|-<br />
|-<br />
|diminished7<br />
|X:(1,b3,b5,bb7)<br />
|X:dim7<br />
|-<br />
|-<br />
|No Chord<br />
|N<br />
|<br />
|}<br />
<br />
<br />
Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--[[User:Chrish|Chrish]] 17:05, 9 September 2009 (UTC)<br />
<br />
We still accept participants who would only like to be evaluated on major/minor chords and want to use the number format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments. '''Please note that the format is still the same'''<br />
<br />
start_time end_time chord_number<br />
<br />
Systems are supposed to print out the onset-offset times as opposed to MIREX 2008 chord output format where only onset were used.<br />
<br />
=== Command line calling format ===<br />
<br />
Submissions have to conform to the specified format below:<br />
<br />
''extractFeaturesAndTrain "/path/to/trainFileList.txt" "/path/to/scratch/dir" '' <br />
<br />
Where fileList.txt has the paths to each wav file. The features extracted on this stage can be stored under "/path/to/scratch/dir" <br />
The ground truth files for the supervised learning will be in the same path with a ".txt" extension at the end. For example for "/path/to/trainFile1.wav", there will be a corresponding ground truth file called "/path/to/trainFile1.wav.txt" . <br />
<br />
For testing:<br />
<br />
''doChordID.sh "/path/to/testFileList.txt" "/path/to/scratch/dir" "/path/to/results/dir" '' <br />
<br />
If there is no training, you can ignore the second argument here. In the results directory, there should be one file for each testfile with same name as the test file + .txt . <br />
<br />
Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.<br />
<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed).<br />
<br />
All submissions should include a README file including the following information:<br />
<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 24 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.<br />
<br />
== Potential Participants ==<br />
name / email<br />
<br />
<br />
== Bibliography ==<br />
<br />
1. Harte,C.A. and Sandler,M.B.(2005). '''Automatic chord identification using a quantised chromagram.''' Proceedings of 118th Audio Engineering Society's Convention.<br />
<br />
2. Sailer,C. and Rosenbauer K.(2006). '''A bottom-up approach to chord detection.''' Proceedings of International Computer Music Conference 2006.<br />
<br />
3. Shenoy,A. and Wang,Y.(2005). '''Key, chord, and rythm tracking of popular music recordings.''' Computer Music Journal 29(3), 75-86.<br />
<br />
4. Sheh,A. and Ellis,D.P.W.(2003). '''Chord segmentation and recognition using em-trained hidden markov models.''' Proceedings of 4th International Conference on Music Information Retrieval.<br />
<br />
5. Yoshioka,T. et al.(2004). '''Automatic Chord Transcription with concurrent recognition of chord symbols and boundaries.''' Proceedings of 5th International Conference on Music Information Retrieval.<br />
<br />
6. Harte,C. and Sandler,M. and Abdallah,S. and G├│mez,E.(2005). '''Symbolic representation of musical chords: a proposed syntax for text annotations.''' Proceedings of 6th International Conference on Music Information Retrieval.<br />
<br />
7. Papadopoulos,H. and Peeters,G.(2007). '''Large-scale study of chord estimation algorithms based on chroma representation and HMM.''' Proceedings of 5th International Conference on Content-Based Multimedia Indexing.<br />
<br />
8. Samer Abdallah, Katy Noland, Mark Sandler, Michael Casey & Christophe Rhodes: '''Theory and Evaluation of a Bayesian Music Structure Extractor''' (pp. 420-425) Proc. 6th International Conference on Music Information Retrieval, ISMIR 2005.</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Melody_Extraction&diff=87392012:Audio Melody Extraction2012-06-07T21:34:04Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
<br />
The aim of the MIREX audio melody extraction evaluation is to identify the melody pitch contour from polyphonic musical audio. Pitch is expressed as the fundamental frequency of the main melodic voice, and is reported in a frame-based manner on an evenly-spaced time-grid.<br />
<br />
The task consists of two parts: <br />
* Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not),<br />
* pitch detection (deciding the most likely melody pitch for each time frame). <br />
<br />
We structure the submission to allow these parts to be done independently within a single output file. That is, it is possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced. Algorithms which don't perform a discrimination between melodic and non-melodic parts are also welcome!<br />
<br />
<br />
== Data == <br />
<br />
=== Collections ===<br />
* MIREX09 database : 374 Karaoke recordings of Chinese songs. Each recording is mixed at three different levels of Signal-to-Accompaniment Ratio {-5dB, 0dB, +5 dB} for a total of 1122 audio clips. Instruments: singing voice (male, female), synthetic accompaniment.<br />
* MIREX08 database : 4 excerpts of 1 min. from "north Indian classical vocal performances", instruments: singing voice (male, female), tanpura (Indian instrument, perpetual background drone), harmonium (secondary melodic instrument) and tablas (pitched percussions). There are two different mixtures of each of the 4 excerpts with differing amounts of accompaniment for a total of 8 audio clips.<br />
* MIREX05 database : 25 phrase excerpts of 10-40 sec from the following genres: Rock, R&B, Pop, Jazz, Solo classical piano.<br />
* ADC04 database : Dataset from the 2004 Audio Description Contest. 20 excerpts of about 20s each.<br />
* manually annotated reference data (10 ms time grid)<br />
<br />
=== Audio Formats ===<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
<br />
== Submission Format ==<br />
<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
=== Input Data ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44.1 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
=== Output Data ===<br />
<br />
The melody extraction algorithms will return the melody contour in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.<br />
<br />
=== Output File Format (Audio Melody Extraction) ===<br />
<br />
The Audio Melody Extraction output file format is a tab-delimited ASCII text format. Fundamental frequencies (in Hz) of the main melody are reported on a 10ms time-grid. If an algorithm estimates that there is no melody present within a given time frame it is to report a NEGATIVE frequency estimate. This allows the algorithm to still output a pitch estimate even if its voiced/unvoiced detection mechanism is incorrect. Therefore, pitch accuracy and segmentation performance can be evaluated separately. Estimating ZERO frequency is also acceptable. However, Pitch Accuracy performance will go down if the voiced/unvoiced detection of the algorithm is incorrect. If the algorithm performs no segmentation, it can report all positive fundamental frequencies (and the segmentation aspects of the evaluation ignored). If the time-stamp in the algorithm output is not on a 10ms time-grid, it will be resampled using 0th-order interpolation during evaluation. Therefore, we encourage the use of a 10ms frame hop-size. Each line of the output file should look like: <br />
<br />
<timestamp (seconds)>\t<frequency (Hz)>\n<br />
<br />
where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:<br />
<br />
0.00 -439.3<br />
0.01 -439.4<br />
0.02 440.2<br />
0.03 440.3<br />
0.04 440.2<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the melody extraction on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
=== README File ===<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:<br />
<br />
foobar -param1 .1 -i %input -o %output<br />
<br />
For a submission using MATLAB, the README file could look like:<br />
<br />
matlab -r "foobar(.1,'%input','%output');quit;"<br />
<br />
== Evaluation Procedures ==<br />
<br />
The task consists of two parts: Voicing detection (deciding whether a particular time frame contains a "melody pitch" or not), and pitch detection (deciding the most likely melody pitch for each time frame). We structured the submission to allow these parts to be done independently, i.e. it was possible (via a negative pitch value) to guess a pitch even for frames that were being judged unvoiced.<br />
So consider a matrix of the per-frame voiced (Ground Truth or Detected values != 0) and unvoiced (GT, Det == 0) results, where the counts are:<br />
Detected<br />
unvx vx sum<br />
---------------<br />
Ground unvoiced | TN | FP | GU<br />
Truth voiced | FN | TP | GV<br />
---------------<br />
sum DU DV TO<br />
<br />
TP ("true positives", frames where the voicing was correctly detected) further breaks down into pitch correct and pitch incorrect, say TP = TPC + TPI<br />
<br />
Similarly, the ability to record pitch guesses even for frames judged unvoiced breaks down FN ("false negatives", frames which were actually pitched but detected as unpitched) into pitch correct and pitch incorrect, say FN = FNC + FNI<br />
In both these cases, we can also count the number of times the chroma was correct, i.e. ignoring octave errors, say TP = TPCch + TPIch and FN = FNCch + FNIch.<br />
<br />
To assess the voicing detection portion, we use the standard tools of detection theory. <br />
<br />
*'''Voicing Detection''' is the probability that a frame which is truly voiced is labeled as voiced i.e. TP/GV (also known as "hit rate").<br />
*'''Voicing False Alarm''' is the probability that a frame which is not actually voiced is none the less labeled as voiced i.e. FP/GU.<br />
*'''Voicing d-prime''' is a measure of the sensitivity of the detector that attempts to factor out the overall bias towards labeling any frame as voiced (which can move both hit rate and false alarm rate up and down in tandem). It converts the hit rate and false alarm into standard deviations away from the mean of an equivalent Gaussian distribution, and reports the difference between them. A larger value indicates a detection scheme with better discrimination between the two classes.<br />
<br />
For the voicing detection, we pool the frames from all excerpts in a dataset to get an overall frame-level voicing detection performance. Because some excerpts had no unvoiced frames, averaging over the excerpts can give some misleading results.<br />
<br />
Now we move on to the actual pitch detection.<br />
*'''Raw Pitch Accuracy''' is the probability of a correct pitch value (to within ± ¼ tone) given that the frame is indeed pitched. This includes the pitch guesses for frames that were judged unvoiced i.e. (TPC + FNC)/GV.<br />
*'''Raw Chroma Accuracy''' is the probability that the chroma (i.e. the note name) is correct over the voiced frames. This ignores errors where the pitch is wrong by an exact multiple of an octave (octave errors). It is (TPCch + FNCch)/GV.<br />
*'''Overall Accuracy''' combines both the voicing detection and the pitch estimation to give the proportion of frames that were correctly labeled with both pitch and voicing, i.e. (TPC + TN)/TO.<br />
<br />
When averaging the pitch statistics, we calculate the performance for each of the excerpts individually, then report the average of these measures. This helps increase the effective weight of some of the minority genres, which had shorter excerpts.<br />
<br />
== Relevant Development Collections == <br />
* [http://unvoicedsoundseparation.googlepages.com/mir-1k MIR-1K]: [http://mirlab.org/dataset/public/MIR-1K_for_MIREX.rar MIR-1K for MIREX](Note that this is not the one used for evaluation. The MIREX 2009 dataset used for evaluation last year was created in the same way but has different content and singers).<br />
<br />
* Graham's collection: you find the test set here and further explanations on the pages http://www.ee.columbia.edu/~graham/mirex_melody/ and http://labrosa.ee.columbia.edu/projects/melody/<br />
<br />
* For the ISMIR 2004 Audio Description Contest, the Music Technology Group of the Pompeu Fabra University assembled a diverse of audio segments and corresponding melody transcriptions including audio excerpts from such genres as Rock, R&B, Pop, Jazz, Opera, and MIDI. http://ismir2004.ismir.net/melody_contest/results.html (full test set with the reference transcriptions (28.6 MB))<br />
<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 12 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Query_by_Singing/Humming&diff=87382012:Query by Singing/Humming2012-06-07T21:33:27Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
<br />
The text of this section is copied from the 2010 page. Please add your comments and discussions for 2012. <br />
<br />
<br />
The goal of the Query-by-Singing/Humming (QBSH) task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. More information can be found in:<br />
<br />
* [[2009:Query_by_Singing/Humming]]<br />
* [[2008:Query_by_Singing/Humming]]<br />
* [[2007:Query_by_Singing/Humming]]<br />
* [[2006:QBSH:_Query-by-Singing/Humming]]<br />
<br />
<br />
<br />
=== Subtask 1: Classic QBSH evaluation ===<br />
This is the classic QBSH problem where we need to find the ground-truth midi from a user's singing or humming.<br />
* '''Queries''': human singing/humming snippets (.wav). Queries are from Roger Jang's corpus and ThinkIT corpus.<br />
* '''Database''': ground-truth and noise MIDI files(which are monophonic). Comprised of 48+106 Roger Jang's and ThinkIT's ground-truth along with a cleaned version of Essen Database(2000+ MIDIs which are used last year) <br />
* '''Output''': top-10 candidate list. <br />
* '''Evaluation''': Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise).<br />
<br />
=== Subtask 2: Variants QBSH evaluation ===<br />
This is based on Prof. Downie's idea that queries are variants of "ground-truth" midi. In fact, this becomes more important since user-contributed singing/humming is an important part of the song database to be searched, as evidenced by the QBSH search service at [http://www.midomi.com/ www.midomi.com].<br />
* '''Queries''': human singing/humming snippets (.wav). Queries are from Roger Jang's corpus and ThinkIT corpus.<br />
* '''Database''': human singing/humming snippets (.wav) from all available corpora (excluding the query input being searched).<br />
* '''Output''': top-10 candidate list. <br />
* '''Evaluation''': Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise).<br />
<br />
To make algorithms able to share intermediate steps, participants are encouraged to submit separate tracker and matcher modules instead of integrated ones, which is according to Rainer Typke's suggestion. So trackers and matchers from different submissions could work together with the same pre-defined interface and thus for us it's possible to find the best combination.<br />
<br />
== Data ==<br />
Currently we have 2 publicly available corpora for QBSH:<br />
<br />
* Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] which is comprised of 4431 queries along with 48 ground-truth MIDI files. All queries are from the beginning of references. Manually labeled pitch for each recording is available. <br />
<br />
* [http://mirlab.org/dataSet/public/IOACAS_QBH.rar IOACAS corpus] comprised of 759 queries and 298 monophonic ground-truth MIDI files (with MIDI 0 or 1 format). There are no "singing from beginning" guarantee.<br />
<br />
The noise MIDI will be the 5000+ Essen collection(can be accessed from http://www.esac-data.org/).<br />
<br />
To build a large test set which can reflect real-world queries, it is suggested that every participant makes a contribution to the evaluation corpus. Sometimes this is hard in practice. So we shall adopt "no hidden dataset" policy if there are not enough user-contributed copora.<br />
<br />
== Evaluation Corpus Contribution ==<br />
Every participant will be asked to contribute 100~200 wave queries (8k 16bits) as well as the ground truth MIDI as test data. Please make your contributed data conformed to the format used in the ThinkIT corpus ([http://159.226.60.224/en/Thinkit.QBH.corpus.rar TITcorpus]). These test data will be released after the competition as a public-domain QBSH dataset.<br />
<br />
[http://mirlab.org/users/davidson833/code/downloads/QBSH_RecordingProgram.rar Here] is a simple tool for recording query data. You may need to have .NET 2.0 or above installed in your system in order to run this program. The generated files conform to the format used in the ThinkIT corpus. Of course you are also welcomed to use your own program to record the query data.<br />
<br />
If there are not enough user-contributed corpora, then we shall adopt "no hidden dataset" policy for QBSH task as usual.<br />
<br />
== Submission Format ==<br />
<br />
=== Breakdown Version ===<br />
The following was based on the suggestion by Xiao Wu last year with some modifications.<br />
<br />
1. Database indexing/building. Command format should look like this: <br />
<br />
indexing %dbMidi.list% %dir_workspace_root%<br />
<br />
where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example: <br />
<br />
./QBSH/midiDatabase/00001.mid<br />
./QBSH/midiDatabase/00002.mid<br />
./QBSH/midiDatabase/00003.mid<br />
./QBSH/midiDatabase/00004.mid<br />
...<br />
<br />
Output indexed files are placed into %dir_workspace_root%. (For task 2, %dbMidi.list% is in fact a list of wav files in the database.)<br />
<br />
2. Pitch tracker. Command format: <br />
<br />
pitch_tracker %queryWave.list% %dir_query_pitch%<br />
<br />
where %queryWave.list% looks like <br />
<br />
queryWave/query_00001.wav<br />
queryWave/query_00002.wav<br />
queryWave/query_00003.wav<br />
...<br />
<br />
Each input file dir_query/query_xxxxx.wav in %queryWave.list% outputs a corresponding transcription %dir_query_pitch%/query_xxxxx.pitch which gives the pitch sequence in midi note scale with the resolution of 10ms: <br />
<br />
0<br />
0<br />
62.23<br />
62.25<br />
62.21<br />
...<br />
<br />
Thus a query with x seconds should output a pitch file with 100*x lines. Places of silence/rest are set to be 0. <br />
<br />
3. Pitch matcher. Command format: <br />
<br />
pitch_matcher %dbMidi.list% %queryPitch.list% %resultFile%<br />
<br />
where %queryPitch.list% looks like <br />
<br />
queryPitch/query_00001.pitch<br />
queryPitch/query_00002.pitch<br />
queryPitch/query_00003.pitch<br />
...<br />
<br />
and the result file gives top-10 candidates(if has) for each query: <br />
<br />
queryPitch/query_00001.pitch: 00025 01003 02200 ... <br />
queryPitch/query_00002.pitch: 01547 02313 07653 ... <br />
queryPitch/query_00003.pitch: 03142 00320 00973 ... <br />
...<br />
<br />
=== Integrated Version ===<br />
If you want to pack everything together, the command format should be much simpler:<br />
<br />
qbshProgram %dbMidi.list% %queryWave.list% %resultFile% %dir_workspace_root%<br />
<br />
You can use %dir_workspace_root% to store any temporary indexing/database structures. The result file should have the same format as mentioned previously. (For task 2, %dbMidi.list% is in fact a list of wav files in the database to be retrieved.)<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of <br />
dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the <br />
information:<br />
<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks,<br />
hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Real-time_Audio_to_Score_Alignment_(a.k.a_Score_Following)&diff=87372012:Real-time Audio to Score Alignment (a.k.a Score Following)2012-06-07T21:33:01Z<p>AndreasEhmann: </p>
<hr />
<div>''Real-time Audio to Score Alignment'', also known as ''Score Following''<br />
<br />
== Description ==<br />
Score Following is the real-time alignment of an incoming music signal to the music score. The music signal can be symbolic (MIDI) or audio, but we will concentrate here on audio following, unless there are some candidates who'd want their symbolic followers to be evaluated and can propose reference data. <br />
<br />
This page describes a proposal for evaluation of score following systems.<br />
<br />
<br />
Submissions will be required to estimate alignment precision according to the indexed times. In order for your system to participate, please specify the type of alignment (monophonic, polyphonic), type of training and realtime performance, also separated into two domains (upon enough submissions) for symbolic and audio systems. Note that we also do accept systems that don't run in real-time in practice, as soon as their algorithm is on-line, i.e. without making use of global knowledge of the input.<br />
<br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data == <br />
46 recordings and their corresponding MIDI representations of the score will be used in the evaluation. These 46 excerpts were extracted from 4 distinct musical pieces.<br />
Recordings are in 44.1Khz 16bit wav format. The reference scores are in MIDI format.<br />
<br />
Zhiyao Duan and Prof. Bryan Pardo contributed another polyphonic dataset. This dataset consists of 10 pieces of four-part J.S. Bach chorales. The audio file was performed by a quartet of instruments: violin, clarinet, saxophone and bassoon. The ground-truth alignment between audio and midi were generated by human annotation.<br />
<br />
Andreas Arzt contributed a heavily polyphonic dataset consisting of 3 piano performances of the Prelude in G minor op. 23-5 by Sergei Rachmaninoff. The 3 performances (by Ashkenazy, Gavrilov and Shelley) differ heavily in their style of interpretation. The ground truth data was compiled by extensive manual correction of off-line alignments. ''Due to an oversight this data was not used for the evaluation runs.''<br />
<br />
== Evaluation procedures ==<br />
<br />
Evaluation procedure consists of running score followers on a database of aligned audio to score where the database contains score, and performance audio (for system call) and a reference alignment (for evaluations) -- <br />
See http://ismir2007.ismir.net/proceedings/ISMIR2007_p315_cont.pdf for details.<br />
<br />
See the details of 2006 proposal on the [[2006:Score_Following_Proposal|MIREX 2006 Wiki]]<br />
<br />
<br />
=== I/O Format ===<br />
Each system should conform to the following format:<br />
<br />
''doScofo.sh "/path/to/audiofile.wav" "/path/to/midi_score_file.mid" "/path/to/result/filename.txt" <br />
<br />
The stdout and stderr will be logged.<br />
<br />
"/path/to/result/filenam.txt" should be have one line per detected note with the following 4 columns<br />
<br />
1. estimated note onset time in performance audio file (ms)<br />
2. detection time relative to performance audio file (ms)<br />
3. note start time in score (ms)<br />
4. MIDI note number in score (int) <br />
<br />
Example :<br />
''1800 1800 0 75''<br />
''2021 2022 187.5 73''<br />
''... ... ... ...''<br />
<br />
Remarks: The third column with the detected note's start time in score serves as the unique identifier of a note (or chord for polyphonic scores) that links it to the ground truth onset of that note within the reference alignment files. The fourth column of MIDI note number is there only for your convenience, to know your way around in the result files, if you know the melody in MIDI.<br />
<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of <br />
dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the <br />
information:<br />
<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks,<br />
hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 12 hours will be imposed on rthe total runtime of algorithms. Submissions that exceed this runtime may not receive a result.<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Key_Detection&diff=87362012:Audio Key Detection2012-06-07T21:32:33Z<p>AndreasEhmann: </p>
<hr />
<div>==Description==<br />
<br />
Determination of the key is a prerequisite for any analysis of tonal music. As a result, extensive work has been done in the area of automatic key detection. The goal of this task is the identification of the key from music in audio format.<br />
<br />
== Data == <br />
=== Collections ===<br />
The collection used for this year's evaluation is the same as the one used in 2005. It consists of 1252 classical music audio pieces rendered from MIDI using the timidity MIDI synthesizer. The ground-truth key is drawn from the title of the piece. The entire piece is not used, but rather the first 30 seconds. This is done because usually the beginnings of pieces are in the labeled key before they possibly deviate due to key modulation.<br />
<br />
=== Audio Formats ===<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
<br />
<br />
== Evaluation Procedures ==<br />
The error analysis will center on comparing the key identified by the algorithm to the actual key of the piece. The key of the piece is the one defined by the composer in the title of the piece. We will then determine how "close" each identified key is to the corresponding correct key. Keys will be considered as "close" if they have one of the following relationships: distance of perfect fifth, relative major and minor, and parallel major and minor. A correct key assignment will be given a full point, and incorrect assignments will be allocated fractions of a point according to the following table:<br />
<br />
{|border="1"<br />
|'''Relation to Correct Key''' ||'''Points'''<br />
|-<br />
|Same||1.0<br />
|-<br />
|Perfect fifth||0.5<br />
|-<br />
|Relative major/minor||0.3<br />
|-<br />
|Parallel major/minor||0.2<br />
|-<br />
|Other||0.0<br />
|}<br />
<br />
The points are counted over all files and averaged. The number of correctly identified keys as well as the distribution of the errors is also reported.<br />
<br />
<br />
== Submission Format ==<br />
<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
=== Input Data ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44.1 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
=== Output Data ===<br />
<br />
The audio key detection algorithms will return the estimated key in an individual ASCII text file for each input .wav audio file. The specification of this output file is immediately below.<br />
<br />
=== Output File Format (Audio Key Detection) ===<br />
<br />
The Audio Key Detection output file format is a single-line tab-delimited ASCII text format. The tonic is reported, followed by a TAB and the mode. For sharps, the "#" symbol is used (e.g. A# for A sharp), for flats, a lowercase "b" is used, e.g. (Bb for B flat). Therefore, the output file should be of the form:<br />
<br />
<tonic {A, A#, Bb, ...}>\t<mode {major, minor}>\n<br />
<br />
where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:<br />
<br />
C major<br />
<br />
or<br />
<br />
G# minor<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the melody extraction on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
<br />
=== Packaging submissions ===<br />
<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2012 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
<br />
==== README File ====<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:<br />
<br />
foobar -param1 .1 -i %input -o %output<br />
<br />
For a submission using MATLAB, the README file could look like:<br />
<br />
matlab -r "foobar(.1,'%input','%output');quit;"<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 6 hours will be imposed on analysis times.<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Onset_Detection&diff=87352012:Audio Onset Detection2012-06-07T21:32:10Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
<br />
Audio Onset Detection concerns itself with finding the time-locations of all sonic events in a piece of audio. This task was originally proposed in 2005 by Paul Brossier and Pierre Leveau . It has subsequently been run in 2005, 2006, 2007, 2009, 2010, 2011.<br />
<br />
== Data ==<br />
=== Collections ===<br />
The dataset will be the same as in 2005/2006/2007/2009 unless new or updated datasets are made available. The current dataset is subdivided into classes, because onset detection is sometimes performed in applications dedicated to a single type of signal (ex: segmentation of a single track in a mix, drum transcription, complex mixes databases segmentation...). The performance of each algorithm will be assessed on the whole dataset but also on each class separately.<br />
<br />
The dataset contains 85 files from 5 classes annotated as follows:<br />
<br />
* 30 solo drum excerpts cross-annotated by 3 people<br />
* 30 solo monophonic pitched instruments excerpts cross-annotated by 3 people<br />
* 10 solo polyphonic pitched instruments excerpts cross-annotated by 3 people<br />
* 15 complex mixes cross-annotated by 5 people <br />
<br />
Moreover the monophonic pitched instruments class is divided into 6 sub-classes: brass (2 excerpts), winds (4), sustained strings (6), plucked strings (9), bars and bells (4), singing voice (5).<br />
<br />
<br />
=== Audio Formats ===<br />
<br />
The data are monophonic sound files, with the associated onset times and data about the annotation robustness.<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
* file length between 2 and 36 seconds (total time: 14 minutes) <br />
<br />
<br />
== Evaluation Procedures ==<br />
<br />
The detected onset times will be compared with the ground-truth ones. For a given ground-truth onset time, if there is a detection in a tolerance time-window around it, it is considered as a correct detection (CD). If not, there is a false negative (FN). The detections outside all the tolerance windows are counted as false positives (FP). Doubled onsets (two detections for one ground-truth onset) and merged onsets (one detection for two ground-truth onsets) will be taken into account in the evaluation. Doubled onsets are a subset of the FP onsets, and merged onsets a subset of FN onsets.<br />
<br />
We define:<br />
<br />
*'''Precision''' P = Ocd / (Ocd +Ofp) <br />
*'''Recall''' R = Ocd / (Ocd + Ofn) <br />
*'''F-measure''' F = 2*P*R/(P+R) <br />
<br />
with these notations:<br />
<br />
*'''Ocd''' number of correctly detected onsets (CD) <br />
*'''Ofn''' number of missed onsets (FN) <br />
*'''Om''' number of merged onsets <br />
*'''Ofp''' number of false positive onsets (FP) <br />
*'''Od''' number of double onsets <br />
<br />
Other indicative measurements:<br />
<br />
*'''FP rate''' FP = 100. * (Ofp) / (Ocd+Ofp) <br />
*'''Doubled Onset rate in FP''' D = 100 * Od / Ofp <br />
*'''Merged Onset rate in FN''' M = 100 * Om / Ofn <br />
<br />
Because files are cross-annotated, the mean Precision and Recall rates are defined by averaging Precision and Recall rates computed for each annotation.<br />
<br />
To establish a ranking, we will use the F-measure, widely used in string comparisons. This criterion is arbitrary, but gives an indication of performance. It must be remembered that onset detection is a preprocessing step, so the real cost of an error of each type (false positive or false negative) depends on the application following this task.<br />
<br />
=== Evaluation measures ===<br />
<br />
* percentage of correct detections / false positives (can also be expressed as precision/recall)<br />
* time precision (tolerance from +/- 50 ms to less). For certain file, we can't be much more accurate than 50 ms because of the weak annotation precision. This must be taken into account.<br />
* separate scoring for different instrument types (percussive, strings, winds, etc) <br />
* percentage of doubled detections<br />
* speed measurements of the algorithms<br />
<br />
<br />
== Submission Format ==<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
=== Input Data ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44.1 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
=== Output Data ===<br />
<br />
The onset detection algorithms will return onset times in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.<br />
<br />
==== Output File Format (Audio Onset Detection) ====<br />
<br />
The Audio Onset Detection output file format is an ASCII text format. Each onset time is specified, in seconds, on its own line. Specifically, <br />
<br />
<onset time(in seconds)>\n<br />
<br />
where \n denotes the end of line. The < and > characters are not included. An example output file would look something like:<br />
<br />
0.243<br />
1.476<br />
1.987<br />
2.449<br />
3.224<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the onset detection on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, such as a detection threshold, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
=== Parameter Sweeps ===<br />
In past iterations of MIREX, submitters have been allowed to specify a parameter sweep so as to generate a precision-recall operator characteristic to better evaluate and understand the algorithm. If you wish to do so, please specify TEN different settings for your sweepable parameter. There are no guarantees that all ten will be tested and evaluated, however, as the time-constraints for MIREX are getting ever smaller as the number of submissions are getting ever larger. Therefore, please also specify the ONE single parameterization you feel is best in the README. If the whole parameter sweep cannot be evaluated, this single parameterization will be used.<br />
<br />
<br />
=== Packaging submissions ===<br />
<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2012 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
<br />
==== README File ====<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
For instance, to test the program foobar with different values for parameters param1, the README file would look like:<br />
<br />
foobar -param1 .1 -i %input -o %output<br />
foobar -param1 .15 -i %input -o %output<br />
foobar -param1 .2 -i %input -o %output<br />
foobar -param1 .25 -i %input -o %output<br />
foobar -param1 .3 -i %input -o %output<br />
...<br />
<br />
For a submission using MATLAB, the README file could look like:<br />
<br />
matlab -r "foobar(.1,'%input','%output');quit;"<br />
matlab -r "foobar(.15,'%input','%output');quit;"<br />
matlab -r "foobar(.2,'%input','%output');quit;" <br />
matlab -r "foobar(.25,'%input','%output');quit;"<br />
matlab -r "foobar(.3,'%input','%output');quit;"<br />
...<br />
<br />
The different command lines to evaluate the performance of each parameter set over the whole database will be generated automatically from each line in the README file containing both '%input' and '%output' strings.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 6 hours will be imposed on analysis times.</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Symbolic_Melodic_Similarity&diff=87342012:Symbolic Melodic Similarity2012-06-07T21:31:49Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
The goal of SMS is to retrieve the most similar items from a collection of symbolic pieces, given a symbolic query, and rank them by melodic similarity. There will be only 1 task this year which comprises a set of six "base" monophonic MIDI queries to be matched against a monophonic MIDI collection. <br />
<br />
Each system will be given a query and is asked to return the 10 most melodically similar songs from those taken from the Essen Collection (5274 pieces in the MIDI format; see [http://www.esac-data.org/ ESAC Data Homepage] for more information). For each of the six "base" queries, we have created four classes of error-mutations, thus the query set comprises the following query classes:<br />
<br />
# No errors (i.e., "base")<br />
# One note deleted<br />
# One note inserted<br />
# One interval enlarged<br />
# One interval compressed<br />
<br />
Each system will be asked to return the top ten items for each of the 30 total queries. That is to say, 6(base queries) X 5(versions) = 30 query/candidate lists to be returned.<br />
<br />
<br />
== Data ==<br />
* 5,274 tunes belonging to the Essen folksong collection. The tunes are in standard MIDI file format. [http://www.ldc.usb.ve/~cgomez/essen.tar.gz Download] (< 1 MB)<br />
<br />
<br />
==Evaluation ==<br />
<br />
The 2012 SMS task replicates the 2007 task. After the algorithms have been submitted, their results will be pooled for every query, and human evaluators, using the Evalutron 6000 system, will asked to judge the relevance of the matches to the queries.[[2012:Evalutron6000_Walkthrough |2012 Evalutron6000 Walkthrough Page]].<br />
<br />
For each query (and its four mutations), the returned results (candidates) from all systems will be anonymously grouped together (query set) for evaluation by the human graders. The graders will be provided with only the "base" perfect version against which to evaluate the candidates and thus did not know whether the candidates came from a perfect or mutated query. We expect that each query/candidate set will be evaluated by one individual grader. Using the Evalutron 6000 system, the graders will give each query/candidate pair two types of scores. Graders will be asked to provide one "BROAD" categorical score with three categories: NS,SS,VS as explained below, and one "FINE" score (in the range from 0 to 10).<br />
<br />
For more information, do take a look at the [[2007:Symbolic_Melodic_Similarity_Results |2007 SMS Results Page]].<br />
<br />
== Submission Format == <br />
<br />
=== Input ===<br />
A path to the folder containing midi files. <br />
<br />
=== Output ===<br />
A list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity output to the stdout. Please use stderr for the log and error messages.<br />
<br />
E.g.<br />
song242.mid song213.mid song1242.mid ...<br />
song5454.mid song423.mid song454.mid ...<br />
...<br />
<br />
=== Commandline Calling Format ===<br />
<br />
There are 2 different calling formats depending on if the program does indexing or not. We encourage the indexing step since programs are called for each query. <br />
<br />
==== 2 Step Calling Format ====<br />
<br />
First Indexing step:<br />
E.g.<br />
doIndex.sh -numThreads 4(optional) /path/to/scratch/folder(optional) /path/to/folder/withMIDIfiles/ /path/indexFileOrDirectory/<br />
<br />
Where <br />
<br />
/path/to/folder/withMIDIfiles/ - the path to the directory containing about 5,000 MIDI files containing monophonic folk songs<br />
<br />
/path/indexFileOrDirectory/ - The path to the index file or folder that will be used in the query step.<br />
<br />
Query in the second step:<br />
E.g.<br />
doQuery.sh -numThreads 4(optional) /path/to/scratch/folder(optional) /path/to/indexFileOrFolder/ /path/to/query.mid<br />
<br />
/path/to/indexFileOrFolder/ - Path to the index file or folder created in the first step.<br />
<br />
/path/to/query.mid - path to the query midi file. <br />
<br />
The program will be called once for each query.<br />
<br />
==== 1 Step Calling Format ====<br />
If your program does not perform indexing. <br />
<br />
E.g.<br />
doSMS.sh -numThreads 4(optional) /path/to/scratch/folder /path/to/folder/withMIDIfiles/ /path/to/query.mid<br />
<br />
The program will be called once for each query. <br />
<br />
<br />
=== Packaging submissions ===<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2012 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 48 hours total runtime will be imposed on each submission. Submissions exceeding this runtime may not receive a result.<br />
<br />
== Potential Participant List == <br />
<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Music_Similarity_and_Retrieval&diff=87332012:Audio Music Similarity and Retrieval2012-06-07T21:31:16Z<p>AndreasEhmann: </p>
<hr />
<div>== Description ==<br />
<br />
As the size of digitial music collections grow, music similarity has an increasingly important role as an aid to music discovery. A music similarity system can help a music consumer find new music by finding the music that is most musically similar to specific query songs (or is nearest to songs that the consumer already likes). <br />
<br />
This page presents the Audio Music Similarity Evaluation, including the submission rules and formats. Additionally background information can be found here that should help explain some of the reasoning behind the approach taken in the evaluation. The intention of the Music Audio Search track is to evaluate music similarity searches (A music search engine that takes a single song as a query aka Query-by-example), not playlist generation or music recommendation.<br />
<br />
The Audio Music Similarity and Retrieval task has been run in MIREX 2010, 2009, 2007, and 2006. <br />
<br />
[[2010:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2010]] || [[2010:Audio_Music_Similarity_and_Retrieval_Results|Results]]<br />
<br />
[[2009:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2009]] || [[2009:Audio_Music_Similarity_and_Retrieval_Results|Results]]<br />
<br />
[[2007:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2007]] || [[2007:Audio_Music_Similarity_and_Retrieval_Results|Results]]<br />
<br />
[[2006:Audio_Music_Similarity_and_Retrieval|Audio Music Similarity and Retrieval task in MIREX 2006]] || [[2006:Audio_Music_Similarity_and_Retrieval_Results|Results]]<br />
<br />
<br />
=== Task specific mailing list ===<br />
In continuing with the changes that took place last year, task specific lists have been dropped. Instead we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
Collection statistics: 7000 30-second audio clips drawn from 10 genres (700 clips from each genre).<br />
<br />
The Genres that data was drawn from are:<br />
*Blues<br />
*Jazz<br />
*Country/Western<br />
*Baroque<br />
*Classical<br />
*Romantic<br />
*Electronica<br />
*Hip-Hop<br />
*Rock<br />
*HardRock/Metal<br />
<br />
<br />
=== Audio formats ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 22 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV<br />
* clip length: 30 secs from the middle of each file<br />
<br />
<br />
== Evaluation ==<br />
Two distinct evaluations will be performed<br />
* Human Evaluation<br />
* Objective statistics derived from the results lists<br />
<br />
Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks, however, in subsequent years we have also supported sparse distance matrix format (detailed below) where only the distances of the top 100 results for each query in the collection are returned.<br />
<br />
<br />
=== Human Evaluation ===<br />
The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators. <br />
<br />
* Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query. <br />
* ~120 randomly selected queries, 5 results per query, 1 set of ears, ~10 participating labs<br />
* Higher number of queries preferred as IR research indicates variance is in queries<br />
* The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)<br />
* It will be possible for researchers to use this data for other types of system comparisons after MIREX 2012 results have been finalized.<br />
* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006 (see: [[2006:Evalutron6000_Issues|Evalutron Issues in MIREX 2006]]).<br />
* Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)<br />
<br />
=== Objective Statistics derived from the distance matrix ===<br />
Statistics of each distance matrix will be calculated including:<br />
<br />
* Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50<br />
* Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results<br />
* Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)<br />
* Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results<br />
* % File never similar (never in a top 5, 10, 20 & 50 result list)<br />
* % of 'test-able' song triplets where triangular inequality holds<br />
** Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.<br />
* Plot of the "number of times similar curve" - plot of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.<br />
<br />
<br />
=== Runtimes ===<br />
In addition computation times for feature extraction/Index-building and querying <br />
will be measured.<br />
<br />
<br />
== Submission format ==<br />
Submission to this task will have to conform to a specified format detailed below. <br />
<br />
<br />
=== Implementation details ===<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.<br />
<br />
The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below. <br />
<br />
Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.<br />
<br />
Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.<br />
<br />
<br />
=== I/O formats ===<br />
In this section the input and output files used in this task are described as<br />
are the command line calling format requirements for submissions.<br />
<br />
<br />
==== Audio collection list file (input)====<br />
The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).<br />
<br />
e.g.<br />
<br />
/aDirectory/collectionFolder/b002342.wav<br />
/aDirectory/collectionFolder/a005921.wav<br />
...<br />
<br />
<br />
==== Distance matrix output files ====<br />
Participants should return one of two available output file formats, a full distance matrix or a sparse distance matrix. The sparse distance matrix format is preferred (as the dense distance matrices can be very large).<br />
<br />
<br />
<br />
===== Sparse Distance Matrix =====<br />
If computation or exhaustive search is a concern or not a normal output of the indexing algorithm employed, the sparse distance matric format detailed below may be used:<br />
<br />
A simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection. <br />
<br />
This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.<br />
<br />
<pre><br />
MyAlgorithm (my.email@address.com)<br />
<example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance><br />
<example 2 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance><br />
...<br />
</pre><br />
<br />
which might look like:<br />
<br />
<pre><br />
MyAlgorithm (my.email@address.com)<br />
a009342.wav b229311.wav,0.16 a023821.wav,0.19 a001329,0.24 ... etc.<br />
a009343.wav a661931.wav,0.12 a043322.wav,0.17 c002346,0.21 ... etc.<br />
a009347.wav a671239.wav,0.13 c112393.wav,0.20 b083293,0.25 ... etc.<br />
...<br />
</pre><br />
<br />
The path to which this list file should be written must be accepted as a parameter on the command line.<br />
<br />
<br />
===== Full Distance Matrix =====<br />
Full distance matrix files should be generated in the the following format: <br />
<br />
* A simple ASCII file listing a name for the algorithm on the first line,<br />
* Numbered paths for each file appearing in the matrix, these can be in any order (i.e. the files don't have to be i the same order as they appeared in the list file) but should index into the columns/rows of of the distance matrix.<br />
* A line beginning with 'Q/R' followed by a tab and tab separated list of the numbers 1 to N, where N is the files covered by the matrix.<br />
* One line per file in the matrix give the distances of that files to each other file in the matrix. All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a single tab character. Obviously the diagonal of the matrix (distance or a track to itself) should be zero.<br />
<br />
<pre><br />
Distance matrix header text with system name<br />
1\t</path/to/audio/file/1.wav><br />
2\t</path/to/audio/file/2.wav><br />
3\t</path/to/audio/file/3.wav><br />
...<br />
N\t</path/to/audio/file/N.wav><br />
Q/R\t1\t2\t3\t...\tN<br />
1\t0.0\t<dist 1 to 2>\t<dist 1 to 3>\t...\t<dist 1 to N><br />
2\t<dist 2 to 1>\t0.0\t<dist 2 to 3>\t...\t<dist 2 to N><br />
3\t<dist 3 to 2>\t<dist 3 to 2>\t0.0\t...\t<dist 3 to N><br />
...\t...\t...\t...\t...\t...<br />
N\t<dist N to 1>\t<dist N to 2>\t<dist N to 3>\t...\t0.0<br />
</pre><br />
<br />
which might look like:<br />
<br />
<pre><br />
Example distance matrix 0.1<br />
1 /path/to/audio/file/1.wav<br />
2 /path/to/audio/file/2.wav<br />
3 /path/to/audio/file/3.wav<br />
4 /path/to/audio/file/4.wav<br />
Q/R 1 2 3 4<br />
1 0.00000 1.24100 0.2e-4 0.42559<br />
2 1.24100 0.00000 0.62640 0.23564<br />
3 50.2e-4 0.62640 0.00000 0.38000<br />
4 0.42559 0.23567 0.38000 0.00000<br />
</pre><br />
<br />
==== Example submission calling formats ====<br />
extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt<br />
Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt<br />
<br />
or<br />
<br />
doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of <br />
dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the <br />
information:<br />
<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks,<br />
hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.<br />
<br />
<br />
== AMS evaluation software ==<br />
The legacy software for performing various AMS related function is available [https://www.music-ir.org/mirex/results/2010/AMS_tools.zip here], [https://www.music-ir.org/mirex/results/2010/AMS_TOOLS_README%20.txt README file]. It maybe used to benchmark systems prior to submission and to check distance matrix file formats.<br />
<br />
This tool set supports the following functions:<br />
* the import of collection metadata from a delimited text file (e.g. TAB or CSV)<br />
* the selection of a stratified random list of queries from the collection (i.e. an equal number of queries are chosen for each class of a particular metadata field, such as genre).<br />
* the generation of results from distance matrices based on a list of pre-chosen queries.<br />
* (pseudo-)objective statistical evaluation of distance matrices by comparing query metadata to the metadata of the top N results retrieved. Supports artist, album, genre and artist-filtered genre (where results form the same artist as query are skipped). Additionally, the number tracks never returned as results for all possible queries (orphans) and the largest hub (track similar to the most other tracks) are measured. Finally, the number of cases where the triangular inequality holds.<br />
* preparation and post processing of results for the IMIRSEL Evalutron 6k human evaluation interface.<br />
<br />
<br />
== Potential Participants ==<br />
<br />
name / email<br />
<br />
Byeong-jun Han / hbj1147 _at_ korea.ac.kr</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Tag_Classification&diff=87322012:Audio Tag Classification2012-06-07T21:30:52Z<p>AndreasEhmann: </p>
<hr />
<div>__TOC__<br />
<br />
== Description ==<br />
This task will compare various algorithms' abilities to associate descriptive tags with 10-second audio clips of songs. Two datasets are used to implement a pair of sub tasks, based on the MajorMiner and Mood tag datasets. This task is very much related to the other audio classification tasks, however, multiple tags may be applied to each example rather than single-label classification. <br />
<br />
Algorithms will be evaluated both on their ability to apply binary classifications of tags to examples, but also on their ability to rank tags for a track by asking them to return an affinity score for each tag/track pair.<br />
<br />
Audio tag classification was first run at MIREX 2008 [[2008:Audio_Tag_Classification]] and as a special MIREX task at 2009<br />
[[2009:SpecialTagatuneEvaluation]] and at 2010 [[2010:Audio_Tag_Classification]]<br />
<br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
Two datasets will be used to evaluate tagging algorithms: The MajorMiner and Mood tag datasets.<br />
<br />
<br />
=== MajorMiner Tag Dataset ===<br />
The tags come from the [http://majorminer.org MajorMiner game]. <br />
All of the data is browseable via the [http://majorminer.org/search MajorMiner search] page.<br />
<br />
The music consists of 2300 clips selected at random from 3900 tracks. Each clip is 10 seconds long. The 2300 clips represent a total of 1400 different tracks on 800 different albums by 500 different artists. To give a sense for the music collection, the following genre tags have been applied to these artists, albums, and tracks on Last.fm: electronica, rock, indie, alternative, pop, britpop, idm, new wave, hip-hop, singer-songwriter, trip-hop, post-punk, ambient, jazz.<br />
<br />
<br />
The MajorMiner game has collected a total of about 73000 taggings, 12000 of which have been verified by at least two users. In these verified taggings, there are 43 tags that have been verified at least 35 times, for a total of about 9000 verified uses. These are the tags we will be using in this task.<br />
<br />
Note that these data do not include strict negative labels. While many clips are tagged ''rock'', none are tagged ''not rock''. Frequently, however, a clip will be tagged many times without being tagged ''rock''. We take this as an indication that ''rock'' does not apply to that clip. More specifically, a negative example of a particular tag is a clip on which another tag has been verified, but the tag in question has not.<br />
<br />
Here is a list of the top 50 tags along with an approximate number of times each has been verified, how many times it's been used in total, and how many different users have ever used it:<br />
<br />
{| class="wikitable" style="margin: 1em auto 1em auto"<br />
! Tag || Verified || Total || Users<br />
|-<br />
| drums || 962 || 3223 || 127 <br />
|-<br />
| guitar || 845 || 3204 || 181 <br />
|-<br />
| male || 724 || 2452 || 95 <br />
|-<br />
| rock || 658 || 2619 || 198 <br />
|-<br />
| synth || 498 || 1889 || 105 <br />
|-<br />
| electronic || 490 || 1878 || 131 <br />
|-<br />
| pop || 479 || 1761 || 151 <br />
|-<br />
| bass || 417 || 1632 || 99 <br />
|-<br />
| vocal || 355 || 1378 || 99 <br />
|-<br />
| female || 342 || 1387 || 100 <br />
|-<br />
| dance || 322 || 1244 || 115 <br />
|-<br />
| techno || 246 || 943 || 104 <br />
|-<br />
| piano || 179 || 826 || 120 <br />
|-<br />
| electronica || 168 || 686 || 67 <br />
|-<br />
| hip hop || 166 || 701 || 126 <br />
|-<br />
| voice || 160 || 790 || 55 <br />
|-<br />
| slow || 157 || 727 || 90 <br />
|-<br />
| beat || 154 || 708 || 90 <br />
|-<br />
| rap || 151 || 723 || 129 <br />
|-<br />
| jazz || 136 || 735 || 154 <br />
|-<br />
| 80s || 130 || 601 || 94 <br />
|-<br />
| fast || 109 || 494 || 70 <br />
|-<br />
| instrumental || 103 || 539 || 62 <br />
|-<br />
| drum machine || 89 || 427 || 35 <br />
|-<br />
| british || 81 || 383 || 60 <br />
|-<br />
| country || 74 || 360 || 105 <br />
|-<br />
| distortion || 73 || 366 || 55 <br />
|-<br />
| saxophone || 70 || 316 || 86 <br />
|-<br />
| house || 65 || 298 || 66 <br />
|-<br />
| ambient || 61 || 335 || 78 <br />
|-<br />
| soft || 61 || 351 || 58 <br />
|-<br />
| silence || 57 || 200 || 35 <br />
|-<br />
| r&b || 57 || 242 || 59 <br />
|-<br />
| strings || 55 || 252 || 62 <br />
|-<br />
| quiet || 54 || 261 || 57 <br />
|-<br />
| solo || 53 || 268 || 56 <br />
|-<br />
| keyboard || 53 || 424 || 41 <br />
|-<br />
| punk || 51 || 242 || 76 <br />
|-<br />
| horns || 48 || 204 || 38 <br />
|-<br />
| drum and bass || 48 || 191 || 50 <br />
|-<br />
| noise || 46 || 249 || 61 <br />
|-<br />
| funk || 46 || 266 || 90 <br />
|-<br />
| acoustic || 40 || 193 || 58 <br />
|-<br />
| trumpet || 39 || 174 || 68 <br />
|-<br />
| end || 38 || 178 || 36 <br />
|-<br />
| loud || 37 || 218 || 62 <br />
|-<br />
| organ || 35 || 169 || 46 <br />
|-<br />
| metal || 35 || 178 || 64 <br />
|-<br />
| folk || 33 || 195 || 58 <br />
|-<br />
| trance || 33 || 226 || 49 <br />
|}<br />
<br />
<br />
=== Mood Tag Dataset ===<br />
The Mood tag dataset is derived from mood related tags on last.fm. All tags in this set are identified by a general affect lexicon (WordNet-Affect) and by human experts. Similar tags are grouped together to define a mood tag group and each song may belong to multiple mood tag groups.<br />
<br />
There are 18 mood tag groups containing 135 unique tags. The dataset contains 3,469 unique songs. The following table lists the tag groups, their member tags and number of songs in each group: <br />
<br />
{| class="wikitable" style="margin: 1em auto 1em auto"<br />
! Group id || Tags || num. of tags || num. of songs<br />
|-<br />
| G12 || calm, comfort, quiet, serene, mellow, chill out, calm down, calming, chillout, comforting, content, cool down, mellow music, mellow rock, peace of mind, quietness, relaxation, serenity, solace, soothe, soothing, still, tranquil, tranquility, tranquility || 25 || 1,680<br />
|-<br />
| G15 || sad, sadness, unhappy, melancholic, melancholy, feeling sad, mood: sad - slightly, sad song || 8 || 1,178<br />
|-<br />
| G5 || happy, happiness, happy songs, happy music, glad, mood: happy || 6 || 749<br />
|-<br />
| G32 || romantic, romantic music || 2 || 619<br />
|-<br />
| G2 || upbeat, gleeful, high spirits, zest, enthusiastic, buoyancy, elation, mood: upbeat|| 8 || 543<br />
|-<br />
| G16 || depressed, blue, dark, depressive, dreary, gloom, darkness, depress, depression, depressing, gloomy || 11 || 471<br />
|-<br />
| G28 || anger, angry, choleric, fury, outraged, rage, angry music || 7 || 254<br />
|-<br />
| G17 || grief, heartbreak, mournful, sorrow, sorry, doleful, heartache, heartbreaking, heartsick, lachrymose, mourning, plaintive, regret, sorrowful || 14 || 183<br />
|-<br />
| G14 || dreamy || 1 || 146<br />
|-<br />
| G6 || cheerful, cheer up, festive, jolly, jovial, merry, cheer, cheering, cheery, get happy, rejoice, songs that are cheerful, sunny || 13 || 142<br />
|-<br />
| G8 || brooding, contemplative, meditative, reflective, broody, pensive, pondering, wistful || 8 || 116<br />
|-<br />
| G29 || aggression, aggressive || 2 || 115<br />
|-<br />
| G25 || angst, anxiety, anxious, jumpy, nervous, angsty || 6 || 80<br />
|-<br />
| G9 || confident, encouraging, encouragement, optimism, optimistic || 5 || 61<br />
|-<br />
| G7 || desire, hope, hopeful, mood: hopeful || 4 || 45<br />
|-<br />
| G11 || earnest, heartfelt || 2 || 40<br />
|-<br />
| G31 || pessimism, cynical, pessimistic, weltschmerz, cynical/sarcastic || 5 || 38<br />
|-<br />
| G1 || excitement, exciting, exhilarating, thrill, ardor, stimulating, thrilling, titillating || 8 || 30<br />
|-<br />
| TOTAL || || 135 || 6,490 <br />
|}<br />
<br />
The songs are mostly from the USPOP collection, a detailed breakdown of the songs are listed in the following table: <br />
<br />
{| class="wikitable" style="margin: 1em auto 1em auto"<br />
! Collection || num. of songs in the dataset || percentage of songs in the dataset<br />
|-<br />
| USPOP || 2764 || 80%<br />
|-<br />
| Assorted pop || 366 || 10%<br />
|-<br />
| American music || 145 || 4%<br />
|-<br />
| Beatles || 128 || 4%<br />
|-<br />
| USCRAP || 40 || 1%<br />
|-<br />
| Metal music || 25 || 1%<br />
|-<br />
| Magnatune || 1 || 0%<br />
|-<br />
| TOTAL || 3469 || 100%<br />
|}<br />
<br />
Details on how the mood tag groups were derived are described in [https://www.music-ir.org/archive/papers/ISMIR2009_MoodClassification.pdf X. Hu, J. S. Downie, A.Ehmann, Lyric Text Mining in Music Mood Classification, In Proceedings of the 10th International Symposium on Music Information Retrieval (ISMIR), Oct. 2009, Kobe , Japan] <br />
<br />
Details on how the songs were selected are available in the [https://www.music-ir.org/archive/papers/Mood_Multi_Tag_Data_Description.pdf description].<br />
<br />
== Evaluation ==<br />
Participating algorithms will be evaluated with 3-fold artist-filtered cross-validation. An introduction to the evaluation statistics computed is given in the following subsections.<br />
<br />
<br />
=== Binary (Classification) Evaluation ===<br />
Algorithms are evaluated on their performance at tag classification using F-measure. Results are also reported for simple accuracy, however, as this statistic is dominated by the negative example accuracy it is not a reliable indicator of performance (as a system that returns no tags for any example will achieve a high score on this statistic). However, the accuracies are also reported for positive and negative examples separately as these can help elucidate the behaviour of an algorithm (for example demonstrating if the system is under or over predicting).<br />
<br />
<br />
=== Affinity (Ranking) Evaluation ===<br />
Algorithms are evaluated on their performance at tag ranking using the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The affinity scores for each tag to be applied to a track are sorted prior to the computation of the AUC-ROC statistic, which gives higher scores to ranked tag sets where the correct tags appear towards the top of the set.<br />
<br />
<br />
=== Ranking and significance testing ===<br />
Additionally, more standard tests could be performed on the average classification accuracy, although the cross-tag variance tends to increase each algorithm's variance, interfering with significance tests without further handling. One test that can help resolve these issues is Friedman's ANOVA with Tukey-Kramer HSD.<br />
<br />
We wish to compare a number of treatments/systems (the submissions) over a number of blocks/rows. We can either compute average classification accuracy and/or precision metrics over all the tags and use the cross validation folds as the blocks/rows - which will handle variance between different folds. However, we are more interested in considering each tag (averaged over all folds) or (perhaps better) each tag on each fold as a separate block.<br />
<br />
The Friedman test should handle the variance between tags (caused by different difficulties of modeling each tag and different numbers of positive and negative examples per tag) by replacing the actual scores achieved by each system on each block (tag) with the rank achieved by that system on that tag amongst all the systems. Hence, we make the assumption that each tag (or combination of tag and fold) is of equal importance in the evaluation. This is an often used approach at TREC (Text Retrieval Conference) when considering retrieval results (where each query is of equal importance, but unequal variance/difficulty).<br />
<br />
Tukey-Kramer Honestly Significant Difference multiple comparisons are made over the results of Friedman's ANOVA as this (and other tests, such as multiply applied Student's T-tests) can only safely tell you if one system is statistically significantly different from the rest. If you try to do the full NxN comparisons with such tests then the experiment wide alpha value is cumulative over all the tests. E.g. if we compared 12 systems at an alpha level of 0.05, a total of 66 pairwise comparisons are made and the chance of incorrectly rejecting the hypothesis of no difference in error rates is: 1 - (0.95^66) = 0.97 = 97%. This explanation is lifted from a paper by Tague-Sutcliffe and Blustein:<br />
<br />
@article{taguesutcliffe1995sat,<br />
title={A Statistical Analysis of the TREC-3 Data},<br />
author={Tague-Sutcliffe, J. and Blustein, J.},<br />
journal={Overview of the Third Text Retrieval Conference (Trec-3)},<br />
year={1995},<br />
publisher={DIANE Publishing}<br />
}<br />
<br />
For further details on the use of Friedman's ANOVA with Tukey-Kramer HSD in MIR, please see:<br />
<br />
@InProceedings{jones2007hsj,<br />
title={"Human Similarity Judgments: Implications for the Design of Formal Evaluations"},<br />
author="M.C. Jones and J.S. Downie and A.F. Ehmann",<br />
BOOKTITLE ="Proceedings of ISMIR 2007 International Society of Music Information Retrieval", <br />
year="2007"<br />
}<br />
<br />
=== Runtime performance ===<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
== Submission format ==<br />
Submission to this task will have to conform to a specified format detailed below, which is very similar to the audio genre classification task, among others.<br />
<br />
<br />
=== Audio formats ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 2 (stereo)<br />
* Encoding: WAV (decoded from MP3 files by IMIRSEL)<br />
* Duration: 10 second clips<br />
<br />
<br />
=== Implementation details ===<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.<br />
<br />
The audio files to be used in the task will be specified in a simple ASCII list file. For feature extraction and classification this file will contain one path per line with no header line. For model training this file will contain one path per line, followed by a tab character and the tag label, again with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.<br />
<br />
Algorithms should divide their feature extraction and training/classification into separate executables/scripts. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.<br />
<br />
Multi-processor compute nodes (8 cores) will be used to run this task. Hence, participants should attempt to use parallelism where-ever possible. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.<br />
<br />
<br />
=== I/O formats ===<br />
In this section the input and output files used in this task are described as are the command line calling format requirements for submissions.<br />
<br />
<br />
==== Feature extraction list file ====<br />
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.<br />
<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
==== Training list file ====<br />
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and a tag label, again with no header line.<br />
<br />
I.e. <br />
<br />
<example path and filename>\t<tag classification>\n<br />
<br />
<br />
E.g.<br />
/path/to/track1.wav drum<br />
/path/to/track1.wav silence<br />
...<br />
<br />
<br />
In this way, the input file will represent the sparse ground truth matrix. While no line will be duplicated, multiple lines may contain the same path, one for each tag associated with that clip. Any tag that is not specified as applying to a clip does not apply to that clip. The ordering of the lines is arbitrary and should not be depended upon.<br />
<br />
==== Test (classification) list file ====<br />
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.<br />
<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
==== Classification output files ====<br />
Participating algorithms should produce '''two''' simple ASCII list files similar in format to the Training list file. The path to which each list file should be written must be accepted as a parameter on the command line.<br />
<br />
<br />
===== Tag Affinity file =====<br />
The first file will contain one path per line, followed by a tab character and the tag label, followed by another tab character and the affinity of that tag for that file, again with no header line.<br />
<br />
I.e.:<br />
<br />
<example path and filename>\t<tag classification>\t<affinity>\n<br />
<br />
E.g.:<br />
<br />
/data/file1.wav rock 0.9<br />
/data/file1.wav guitar 0.7<br />
/data/file1.wav vocal 0.3<br />
/data/file2.wav rock 0.5<br />
...<br />
<br />
In this way, the output file will represent the sparse classification matrix. A path should be repeated on a separate line for each tag that the submission deems applies to it. If a (path, tag) pair is not specified, it will be assumed to have an affinity of 0. The ordering of the lines is not important and can be arbitrary.<br />
<br />
The affinity will be used for retrieval evaluation metrics, and its only specification is that for a given tag, larger (closer to +infinity) numbers indicate that the tag is more appropriate to a clip than smaller (closer to -infinity) numbers. As submissions are asked to also return a binary relevance listing, submissions that do not compute an affinity should provide only the binary relevance listing file.<br />
<br />
<br />
===== Binary relevance file =====<br />
The second file to be produced is a binary version of the tag classifications, where a tag must be marked as relevant or not relevant to a track. This file will contain one path per line, followed by a tab character and the tag label, followed by another tab character and either a 1 or a 0 indicating the relevance of that tag for that file, again with no header line.<br />
<br />
I.e.:<br />
<br />
<example path and filename>\t<tag classification>\t<relevant? [0 | 1]>\n<br />
<br />
E.g.:<br />
<br />
/data/file1.wav rock 1<br />
/data/file1.wav guitar 1<br />
/data/file1.wav vocal 0<br />
/data/file2.wav rock 1<br />
...<br />
<br />
If a (path, tag) pair is not specified, it will be assumed to be non-relevant (0). Any line with path but no numerical value will be assumed to be relevant (1).<br />
<br />
Hence, the following is equivalent to the example above:<br />
<br />
/data/file1.wav rock<br />
/data/file1.wav guitar<br />
/data/file2.wav rock<br />
<br />
The ordering of the lines is not important and can be arbitrary.<br />
<br />
<br />
=== Example submission calling formats ===<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputAffinityFile.txt /path/to/outputBinaryRelevanceFile.txt<br />
<br />
extractFeatures.sh -numThreads 8 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh -numThreads 8 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputAffinityFile.txt /path/to/outputBinaryRelevanceFile.txt<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt <br />
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputAffinityFile.txt /path/to/outputBinaryRelevanceFile.txt<br />
<br />
myAlgo.sh -extract -numThreads 8 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -TrainAndClassify -numThreads 8 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputAffinityFile.txt /path/to/outputBinaryRelevanceFile.txt<br />
<br />
myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt <br />
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputAffinityFile.txt /path/to/outputBinaryRelevanceFile.txt<br />
<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed).<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments libraries and architectures (including version information) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm <br />
<br />
Note that the information that you place in the README file is extremely important in ensuring that your submission is evaluated properly.<br />
<br />
=== Time and hardware limits ===<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be specified.<br />
<br />
A hard limit of 72 hours will be imposed on the full execution of a submission on each dataset (to include feature extraction time and the 3 training/testing cycles required for the 3-fold cross-validated experiment. <br />
<br />
These limits will likely be strictly imposed at MIREX 2012 (due to the very high level of participation that is expected).</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Cover_Song_Identification&diff=87312012:Audio Cover Song Identification2012-06-07T21:30:13Z<p>AndreasEhmann: /* Submission closing date */</p>
<hr />
<div>__TOC__<br />
<br />
==Description==<br />
This task requires that algorithms identify, for a query audio track, other recordings of the same composition, or "cover songs".<br />
<br />
Within the a collection of pieces in the cover song datasets, there are embedded a number of different "original songs" or compositions each represented by a number of different "versions". The "cover songs" or "versions" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations. <br />
<br />
Using each of these version files in turn as as the "seed/query" file, we examine the returned ranked lists of items from each algorithm for the presence of the other versions of the "seed/query" file.<br />
<br />
Two datasets are used in this task, the MIREX 2006 US Pop Music Cover Song dataset Audio Cover Song dataset the [http://www.mazurka.org.uk/ Mazurka dataset]. <br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
Two datasets will be used to evaluate cover song identification:<br />
<br />
===US Pop Music Collection Cover Song (aka Mixed Collection)===<br />
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files. <br />
<br />
Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.<br />
<br />
Collection statistics:<br />
* 16bit, monophonic, 22.05khz, wav<br />
* The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations. <br />
* Size: 1000 tracks<br />
* Queries: 330 tracks<br />
<br />
=== Sapp's Mazurka Collection Information ===<br />
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems should return a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.<br />
<br />
Collection statistics:<br />
* 16bit, monophonic, 22.05khz, wav<br />
* Size: 539 tracks<br />
* Queries: 539 tracks<br />
<br />
<br />
== Evaluation ==<br />
The following evaluation metrics will be computed for each submission:<br />
* Total number of covers identified in top 10<br />
* Mean number of covers identified in top 10 (average performance)<br />
* Mean (arithmetic) of Avg. Precisions<br />
* Mean rank of first correctly identified cover<br />
<br />
<br />
=== Ranking and significance testing ===<br />
Friedman's ANOVA with Tukey-Kramer HSD will be run against the Average Precision summary data over the individual song groups to assess the significance of differences in performance and to rank the performances.<br />
<br />
For further details on the use of Friedman's ANOVA with Tukey-Kramer HSD in MIR, please see:<br />
@InProceedings{jones2007hsj,<br />
title={"Human Similarity Judgements: Implications for the Design of Formal Evaluations"},<br />
author="M.C. Jones and J.S. Downie and A.F. Ehmann",<br />
BOOKTITLE ="Proceedings of ISMIR 2007 International Society of Music Information Retrieval", <br />
year="2007"<br />
}<br />
<br />
<br />
=== Runtime performance ===<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
<br />
== Submission Format ==<br />
Submission to this task will have to conform to a specified format detailed below.<br />
<br />
<br />
=== Implementation details ===<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.<br />
<br />
The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.<br />
<br />
Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.<br />
<br />
<br />
=== I/O formats ===<br />
=== Input Files ===<br />
<br />
The feature extraction list file format will be of the form: <br />
<br />
/path/to/audio/file/000.wav\n<br />
/path/to/audio/file/001.wav\n<br />
/path/to/audio/file/002.wav\n<br />
... <br />
<br />
The query list file format will be very similar, taking the form, and listing a subset of files from the feature extraction list file: <br />
<br />
/path/to/audio/file/182.wav\n<br />
/path/to/audio/file/245.wav\n<br />
/path/to/audio/file/432.wav\n<br />
...<br />
<br />
For a total of ''<number of queries>'' rows -- query ids are assigned from the pool of ''<number of candidates>'' collection ids and should match the ids within the candidate collection.<br />
<br />
Lines will be terminated by a '\n' character.<br />
<br />
=== Output File ===<br />
The only output will be a '''distance''' matrix file that is ''<number of queries>'' rows by ''<number of candidates>'' columns in the following format: <br />
<br />
<br />
<pre><br />
Distance matrix header text with system name<br />
1\t</path/to/audio/file/track1.wav><br />
2\t</path/to/audio/file/track2.wav><br />
3\t</path/to/audio/file/track3.wav><br />
4\t</path/to/audio/file/track4.wav><br />
...<br />
N\t</path/to/audio/file/trackN.wav><br />
Q/R\t1\t2\t3\t4\t...\tN<br />
1\t<dist 1 to 1>\t<dist 1 to 2>\t<dist 1 to 3>\t<dist 1 to 4>\t...\t<dist 1 to N><br />
3\t<dist 3 to 2>\t<dist 3 to 2>\t<dist 3 to 3>\t<dist 3 to 4>\t...\t<dist 3 to N><br />
</pre><br />
<br />
where N is <number of candidates> and the queries are drawn from this set (and bear the same track indexes if possible).<br />
<br />
which might look like:<br />
<br />
<pre><br />
Example distance matrix 0.1<br />
1 /path/to/audio/file/track1.wav<br />
2 /path/to/audio/file/track2.wav<br />
3 /path/to/audio/file/track3.wav<br />
4 /path/to/audio/file/track4.wav<br />
5 /path/to/audio/file/track5.wav<br />
Q/R 1 2 3 4 5<br />
1 0.00000 1.24100 0.2e-4 0.42559 0.21313<br />
3 50.2e-4 0.62640 0.00000 0.38000 0.15152<br />
</pre><br />
<br />
Note that indexes of the queries refer back to the track list at the top of the distance matrix file to identify the query track. However, as long as you ensure that the query songs are listed in exactly the same order as they appear in the query list file you are passed we will be able to interpret the data.<br />
<br />
All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a TAB.<br />
<br />
To summarize, the distance matrix should be preceded by a system name, ''<number of candidates>'' rows of file paths and should be composed of ''<number of candidates>'' columns of distance (separated by tab characters) and ''<number of queries>'' rows (one for each original track query). Each row corresponds to a particular query song (the track to find covers of).<br />
<br />
<br />
=== Command Line Calling Format ===<br />
<br />
/path/to/submission <collection_list_file> <query_list_file> <working_directory> <output_file><br />
'''<collection_list_file>''': Text file containing ''<number of candidates>'' full path file names for the<br />
''<number of candidates>'' audio files in the collection (including the ''<number of queries>'' <br />
query documents).<br />
'''Example: /path/to/coversong/collection.txt'''<br />
'''<query_list_file>''' : Text file containing the ''<number of queries>'' full path file names for the <br />
''<number of queries>'' query documents.<br />
'''Example: /path/to/coversong/queries.txt'''<br />
'''<working_directory>''' : Full path to a temporary directory where submission will <br />
have write access for caching features or calculations.<br />
'''Example: /tmp/submission_id/'''<br />
'''<output_file>''' : Full path to file where submission should output the similarity <br />
matrix (''<number of candidates>'' header rows + ''<number of queries>'' x ''<number of candidates>'' data matrix).<br />
'''Example: /path/to/coversong/results/submission_id.txt'''<br />
<br />
E.g.<br />
/path/to/m/submission.sh /path/to/feat_extract_file.txt /path/to/query_file.txt /path/to/scratch/dir /path/to/output_file.txt<br />
<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.<br />
<br />
<br />
<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Cover_Song_Identification&diff=87302012:Audio Cover Song Identification2012-06-07T21:30:03Z<p>AndreasEhmann: /* Submission opening date */</p>
<hr />
<div>__TOC__<br />
<br />
==Description==<br />
This task requires that algorithms identify, for a query audio track, other recordings of the same composition, or "cover songs".<br />
<br />
Within the a collection of pieces in the cover song datasets, there are embedded a number of different "original songs" or compositions each represented by a number of different "versions". The "cover songs" or "versions" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations. <br />
<br />
Using each of these version files in turn as as the "seed/query" file, we examine the returned ranked lists of items from each algorithm for the presence of the other versions of the "seed/query" file.<br />
<br />
Two datasets are used in this task, the MIREX 2006 US Pop Music Cover Song dataset Audio Cover Song dataset the [http://www.mazurka.org.uk/ Mazurka dataset]. <br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
Two datasets will be used to evaluate cover song identification:<br />
<br />
===US Pop Music Collection Cover Song (aka Mixed Collection)===<br />
This is the "original" ACS collection. Within the 1000 pieces in the Audio Cover Song database, there are embedded 30 different "cover songs" each represented by 11 different "versions" for a total of 330 audio files. <br />
<br />
Using each of these cover song files in turn as as the "seed/query" file, we will examine the returned lists of items for the presence of the other 10 versions of the "seed/query" file.<br />
<br />
Collection statistics:<br />
* 16bit, monophonic, 22.05khz, wav<br />
* The "cover songs" represent a variety of genres (e.g., classical, jazz, gospel, rock, folk-rock, etc.) and the variations span a variety of styles and orchestrations. <br />
* Size: 1000 tracks<br />
* Queries: 330 tracks<br />
<br />
=== Sapp's Mazurka Collection Information ===<br />
In addition to our original ACS dataset, we used the [http://www.mazurka.org.uk/ Mazurka.org dataset] put together by Craig Sapp. We randomly chose 11 versions from 49 mazurkas and ran it as a separate ACS subtask. Systems should return a distance matrix of 539x539 from which we located the ranks of each of the associated cover versions.<br />
<br />
Collection statistics:<br />
* 16bit, monophonic, 22.05khz, wav<br />
* Size: 539 tracks<br />
* Queries: 539 tracks<br />
<br />
<br />
== Evaluation ==<br />
The following evaluation metrics will be computed for each submission:<br />
* Total number of covers identified in top 10<br />
* Mean number of covers identified in top 10 (average performance)<br />
* Mean (arithmetic) of Avg. Precisions<br />
* Mean rank of first correctly identified cover<br />
<br />
<br />
=== Ranking and significance testing ===<br />
Friedman's ANOVA with Tukey-Kramer HSD will be run against the Average Precision summary data over the individual song groups to assess the significance of differences in performance and to rank the performances.<br />
<br />
For further details on the use of Friedman's ANOVA with Tukey-Kramer HSD in MIR, please see:<br />
@InProceedings{jones2007hsj,<br />
title={"Human Similarity Judgements: Implications for the Design of Formal Evaluations"},<br />
author="M.C. Jones and J.S. Downie and A.F. Ehmann",<br />
BOOKTITLE ="Proceedings of ISMIR 2007 International Society of Music Information Retrieval", <br />
year="2007"<br />
}<br />
<br />
<br />
=== Runtime performance ===<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
<br />
== Submission Format ==<br />
Submission to this task will have to conform to a specified format detailed below.<br />
<br />
<br />
=== Implementation details ===<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.<br />
<br />
The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below.<br />
<br />
Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.<br />
<br />
<br />
=== I/O formats ===<br />
=== Input Files ===<br />
<br />
The feature extraction list file format will be of the form: <br />
<br />
/path/to/audio/file/000.wav\n<br />
/path/to/audio/file/001.wav\n<br />
/path/to/audio/file/002.wav\n<br />
... <br />
<br />
The query list file format will be very similar, taking the form, and listing a subset of files from the feature extraction list file: <br />
<br />
/path/to/audio/file/182.wav\n<br />
/path/to/audio/file/245.wav\n<br />
/path/to/audio/file/432.wav\n<br />
...<br />
<br />
For a total of ''<number of queries>'' rows -- query ids are assigned from the pool of ''<number of candidates>'' collection ids and should match the ids within the candidate collection.<br />
<br />
Lines will be terminated by a '\n' character.<br />
<br />
=== Output File ===<br />
The only output will be a '''distance''' matrix file that is ''<number of queries>'' rows by ''<number of candidates>'' columns in the following format: <br />
<br />
<br />
<pre><br />
Distance matrix header text with system name<br />
1\t</path/to/audio/file/track1.wav><br />
2\t</path/to/audio/file/track2.wav><br />
3\t</path/to/audio/file/track3.wav><br />
4\t</path/to/audio/file/track4.wav><br />
...<br />
N\t</path/to/audio/file/trackN.wav><br />
Q/R\t1\t2\t3\t4\t...\tN<br />
1\t<dist 1 to 1>\t<dist 1 to 2>\t<dist 1 to 3>\t<dist 1 to 4>\t...\t<dist 1 to N><br />
3\t<dist 3 to 2>\t<dist 3 to 2>\t<dist 3 to 3>\t<dist 3 to 4>\t...\t<dist 3 to N><br />
</pre><br />
<br />
where N is <number of candidates> and the queries are drawn from this set (and bear the same track indexes if possible).<br />
<br />
which might look like:<br />
<br />
<pre><br />
Example distance matrix 0.1<br />
1 /path/to/audio/file/track1.wav<br />
2 /path/to/audio/file/track2.wav<br />
3 /path/to/audio/file/track3.wav<br />
4 /path/to/audio/file/track4.wav<br />
5 /path/to/audio/file/track5.wav<br />
Q/R 1 2 3 4 5<br />
1 0.00000 1.24100 0.2e-4 0.42559 0.21313<br />
3 50.2e-4 0.62640 0.00000 0.38000 0.15152<br />
</pre><br />
<br />
Note that indexes of the queries refer back to the track list at the top of the distance matrix file to identify the query track. However, as long as you ensure that the query songs are listed in exactly the same order as they appear in the query list file you are passed we will be able to interpret the data.<br />
<br />
All distances should be zero or positive (0.0+) and should not be infinite or NaN. Values should be separated by a TAB.<br />
<br />
To summarize, the distance matrix should be preceded by a system name, ''<number of candidates>'' rows of file paths and should be composed of ''<number of candidates>'' columns of distance (separated by tab characters) and ''<number of queries>'' rows (one for each original track query). Each row corresponds to a particular query song (the track to find covers of).<br />
<br />
<br />
=== Command Line Calling Format ===<br />
<br />
/path/to/submission <collection_list_file> <query_list_file> <working_directory> <output_file><br />
'''<collection_list_file>''': Text file containing ''<number of candidates>'' full path file names for the<br />
''<number of candidates>'' audio files in the collection (including the ''<number of queries>'' <br />
query documents).<br />
'''Example: /path/to/coversong/collection.txt'''<br />
'''<query_list_file>''' : Text file containing the ''<number of queries>'' full path file names for the <br />
''<number of queries>'' query documents.<br />
'''Example: /path/to/coversong/queries.txt'''<br />
'''<working_directory>''' : Full path to a temporary directory where submission will <br />
have write access for caching features or calculations.<br />
'''Example: /tmp/submission_id/'''<br />
'''<output_file>''' : Full path to file where submission should output the similarity <br />
matrix (''<number of candidates>'' header rows + ''<number of queries>'' x ''<number of candidates>'' data matrix).<br />
'''Example: /path/to/coversong/results/submission_id.txt'''<br />
<br />
E.g.<br />
/path/to/m/submission.sh /path/to/feat_extract_file.txt /path/to/query_file.txt /path/to/scratch/dir /path/to/output_file.txt<br />
<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.<br />
<br />
<br />
<br />
<br />
== Submission closing date ==<br />
Friday September 2nd 2012<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Classification_(Train/Test)_Tasks&diff=87292012:Audio Classification (Train/Test) Tasks2012-06-07T21:29:12Z<p>AndreasEhmann: /* Submission closing date */</p>
<hr />
<div>= Audio Classification (Test/Train) tasks =<br />
<br />
== Description ==<br />
<br />
Many tasks in music classification can be characterized into a two-stage process: training classification models using labeled data and testing the models using new/unseen data. Therefore, we propose this "meta" task which includes various audio classification tasks that follow this Train/Test process. For MIREX 2012, five classification sub-tasks are included: <br />
<br />
* Audio Classical Composer Identification<br />
* Audio US Pop Music Genre Classification <br />
* Audio Latin Music Genre Classification <br />
* Audio Mood Classification<br />
<br />
All five classification tasks were conducted in previous MIREX runs (please see ). This page presents the evaluation of these tasks, including the datasets as well as the submission rules and formats.<br />
<br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
=== Audio Classical Composer Identification ===<br />
This dataset requires algorithms to classify music audio according to the composer of the track (drawn from a collection of performances of a variety of classical music genres). The collection used at MIREX 2009 will be re-used.<br />
<br />
Collection statistics: <br />
* 2772 30-second 22.05 kHz mono wav clips<br />
* 11 "classical" composers (252 clips per composer), including:<br />
** Bach<br />
** Beethoven<br />
** Brahms<br />
** Chopin<br />
** Dvorak<br />
** Handel<br />
** Haydn<br />
** Mendelssohn<br />
** Mozart<br />
** Schubert<br />
** Vivaldi<br />
<br />
<br />
=== Audio US Pop Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of US Pop music tracks). The MIREX 2007 Genre dataset will be re-used, which was drawn from the USPOP 2002 and USCRAP collections.<br />
<br />
Collection statistics: <br />
* 7000 30-second audio clips in 22.05kHz mono WAV format <br />
* 10 genres (700 clips from each genre), including:<br />
** Blues<br />
** Jazz<br />
** Country/Western<br />
** Baroque<br />
** Classical<br />
** Romantic<br />
** Electronica<br />
** Hip-Hop<br />
** Rock<br />
** HardRock/Metal<br />
<br />
<br />
=== Audio Latin Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of Latin popular and dance music, sourced from Brazil and hand labeled by music experts). Carlos Silla's (cns2 (at) kent (dot) ac (dot) uk) Latin popular and dance music dataset [http://ismir2008.ismir.net/papers/ISMIR2008_106.pdf] will be re-used. This collection is likely to contain a greater number of styles of music that will be differentiated by rhythmic characteristics than the MIREX 2007 dataset.<br />
<br />
Collection statistics: <br />
* 3,227 audio files in 22.05kHz mono WAV format <br />
* 10 Latin music genres, including:<br />
** Axe<br />
** Bachata<br />
** Bolero<br />
** Forro <br />
** Gaucha<br />
** Merengue<br />
** Pagode<br />
** Sertaneja<br />
** Tango<br />
<br />
<br />
=== Audio Mood Classification ===<br />
This dataset requires algorithms to classify music audio according to the mood of the track (drawn from a collection of production msuic sourced from the APM collection [http://www.apmmusic.com]). The MIREX 2007 Mood Classification dataset [http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf] will be re-used. <br />
<br />
Collection statistics: <br />
* 600 30 second audio clips in 22.05kHz mono WAV format selected from the APM collection [http://www.apmmusic.com], and labeled by human judges using the Evalutron6000 system. <br />
* 5 mood categories [http://ismir2007.ismir.net/proceedings/ISMIR2007_p067_hu.pdf] each of which contains 120 clips:<br />
**Cluster_1: passionate, rousing, confident,boisterous, rowdy <br />
**Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured <br />
**Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding <br />
**Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry <br />
**Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral<br />
<br />
== Audio Formats ==<br />
For all datasets, participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 22 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV<br />
<br />
<br />
== Evaluation ==<br />
This section first describes evaluation methods common to all the datasets, then specifies settings unique to each of the tasks. <br />
<br />
Participating algorithms will be evaluated with 3-fold cross validation. For '''Artist Identification''' and '''Classical Composer Classification''', album filtering will be used the test and training splits, i.e. training and test sets will contain tracks from different albums; for '''US Pop Genre Classification''' and '''Latin Genre Classification''', artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists. <br />
<br />
The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.<br />
<br />
Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance. <br />
<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
== Submission Format ==<br />
=== File I/O Format ===<br />
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below: <br />
<br />
<br />
==== Feature extraction list file ====<br />
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Training list file ====<br />
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.<br />
<br />
I.e. <br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav rock<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
==== Test (classification) list file ====<br />
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.<br />
<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Classification output file ====<br />
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.<br />
<br />
I.e.<br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav classical<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
=== Submission calling formats ===<br />
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.<br />
<br />
Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.<br />
<br />
Executables will have to accept the paths to the aforementioned list files as command line parameters.<br />
<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.<br />
<br />
<br />
==== Example submission calling formats ====<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt <br />
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt <br />
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations. <br />
<br />
extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
=== Packaging submissions ===<br />
<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2012 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
=== Time and hardware limits ===<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 24 hours will be imposed on feature extraction times.<br />
<br />
A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.<br />
<br />
=== Potential Participants ===<br />
name / email<br />
<br />
= Participation in previous years and Links to Results =<br />
<br />
<br />
{| border="1"<br />
|-<br />
| <br />
'''Year'''<br />
| <br />
'''Participating Algorithms ''' <br />
| <br />
'''URL'''<br />
|-<br />
|<br />
|-<br />
|<br />
2010<br />
|<br />
27<br />
|<br />
https://nema.lis.illinois.edu/nema_out/4ffcb482-b83c-4ba6-bc42-9b538b31143c/results/evaluation/<br />
|-<br />
|<br />
|<br />
24<br />
|<br />
https://nema.lis.illinois.edu/nema_out/6731c97a-240c-4d3d-8be9-90d715ea04e1/results/evaluation/<br />
|-<br />
|<br />
|<br />
24<br />
|<br />
https://nema.lis.illinois.edu/nema_out/2b5839b3-3012-4f76-8807-31823588ae25/results/evaluation/<br />
|-<br />
|<br />
|<br />
36<br />
|<br />
https://nema.lis.illinois.edu/nema_out/9b11a5c8-9fcf-4029-95eb-51ed561cfb5f/results/evaluation/<br />
|-<br />
|<br />
2009<br />
|<br />
30<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Latin_Set%29_Results<br />
|-<br />
|<br />
|<br />
31<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Mixed_Set%29_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2008<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2007<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
9<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2005<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-artist/index.html<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-genre/index.html<br />
|}</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:Audio_Classification_(Train/Test)_Tasks&diff=87282012:Audio Classification (Train/Test) Tasks2012-06-07T21:28:59Z<p>AndreasEhmann: /* Submission opening date */</p>
<hr />
<div>= Audio Classification (Test/Train) tasks =<br />
<br />
== Description ==<br />
<br />
Many tasks in music classification can be characterized into a two-stage process: training classification models using labeled data and testing the models using new/unseen data. Therefore, we propose this "meta" task which includes various audio classification tasks that follow this Train/Test process. For MIREX 2012, five classification sub-tasks are included: <br />
<br />
* Audio Classical Composer Identification<br />
* Audio US Pop Music Genre Classification <br />
* Audio Latin Music Genre Classification <br />
* Audio Mood Classification<br />
<br />
All five classification tasks were conducted in previous MIREX runs (please see ). This page presents the evaluation of these tasks, including the datasets as well as the submission rules and formats.<br />
<br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
=== Audio Classical Composer Identification ===<br />
This dataset requires algorithms to classify music audio according to the composer of the track (drawn from a collection of performances of a variety of classical music genres). The collection used at MIREX 2009 will be re-used.<br />
<br />
Collection statistics: <br />
* 2772 30-second 22.05 kHz mono wav clips<br />
* 11 "classical" composers (252 clips per composer), including:<br />
** Bach<br />
** Beethoven<br />
** Brahms<br />
** Chopin<br />
** Dvorak<br />
** Handel<br />
** Haydn<br />
** Mendelssohn<br />
** Mozart<br />
** Schubert<br />
** Vivaldi<br />
<br />
<br />
=== Audio US Pop Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of US Pop music tracks). The MIREX 2007 Genre dataset will be re-used, which was drawn from the USPOP 2002 and USCRAP collections.<br />
<br />
Collection statistics: <br />
* 7000 30-second audio clips in 22.05kHz mono WAV format <br />
* 10 genres (700 clips from each genre), including:<br />
** Blues<br />
** Jazz<br />
** Country/Western<br />
** Baroque<br />
** Classical<br />
** Romantic<br />
** Electronica<br />
** Hip-Hop<br />
** Rock<br />
** HardRock/Metal<br />
<br />
<br />
=== Audio Latin Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of Latin popular and dance music, sourced from Brazil and hand labeled by music experts). Carlos Silla's (cns2 (at) kent (dot) ac (dot) uk) Latin popular and dance music dataset [http://ismir2008.ismir.net/papers/ISMIR2008_106.pdf] will be re-used. This collection is likely to contain a greater number of styles of music that will be differentiated by rhythmic characteristics than the MIREX 2007 dataset.<br />
<br />
Collection statistics: <br />
* 3,227 audio files in 22.05kHz mono WAV format <br />
* 10 Latin music genres, including:<br />
** Axe<br />
** Bachata<br />
** Bolero<br />
** Forro <br />
** Gaucha<br />
** Merengue<br />
** Pagode<br />
** Sertaneja<br />
** Tango<br />
<br />
<br />
=== Audio Mood Classification ===<br />
This dataset requires algorithms to classify music audio according to the mood of the track (drawn from a collection of production msuic sourced from the APM collection [http://www.apmmusic.com]). The MIREX 2007 Mood Classification dataset [http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf] will be re-used. <br />
<br />
Collection statistics: <br />
* 600 30 second audio clips in 22.05kHz mono WAV format selected from the APM collection [http://www.apmmusic.com], and labeled by human judges using the Evalutron6000 system. <br />
* 5 mood categories [http://ismir2007.ismir.net/proceedings/ISMIR2007_p067_hu.pdf] each of which contains 120 clips:<br />
**Cluster_1: passionate, rousing, confident,boisterous, rowdy <br />
**Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured <br />
**Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding <br />
**Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry <br />
**Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral<br />
<br />
== Audio Formats ==<br />
For all datasets, participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 22 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV<br />
<br />
<br />
== Evaluation ==<br />
This section first describes evaluation methods common to all the datasets, then specifies settings unique to each of the tasks. <br />
<br />
Participating algorithms will be evaluated with 3-fold cross validation. For '''Artist Identification''' and '''Classical Composer Classification''', album filtering will be used the test and training splits, i.e. training and test sets will contain tracks from different albums; for '''US Pop Genre Classification''' and '''Latin Genre Classification''', artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists. <br />
<br />
The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.<br />
<br />
Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance. <br />
<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
== Submission Format ==<br />
=== File I/O Format ===<br />
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below: <br />
<br />
<br />
==== Feature extraction list file ====<br />
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Training list file ====<br />
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.<br />
<br />
I.e. <br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav rock<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
==== Test (classification) list file ====<br />
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.<br />
<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Classification output file ====<br />
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.<br />
<br />
I.e.<br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav classical<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
=== Submission calling formats ===<br />
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.<br />
<br />
Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.<br />
<br />
Executables will have to accept the paths to the aforementioned list files as command line parameters.<br />
<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.<br />
<br />
<br />
==== Example submission calling formats ====<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt <br />
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt <br />
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations. <br />
<br />
extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
=== Packaging submissions ===<br />
<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2012 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
=== Time and hardware limits ===<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 24 hours will be imposed on feature extraction times.<br />
<br />
A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.<br />
<br />
=== Submission closing date ===<br />
<br />
Friday August 26th 2012<br />
<br />
=== Potential Participants ===<br />
name / email<br />
<br />
= Participation in previous years and Links to Results =<br />
<br />
<br />
{| border="1"<br />
|-<br />
| <br />
'''Year'''<br />
| <br />
'''Participating Algorithms ''' <br />
| <br />
'''URL'''<br />
|-<br />
|<br />
|-<br />
|<br />
2010<br />
|<br />
27<br />
|<br />
https://nema.lis.illinois.edu/nema_out/4ffcb482-b83c-4ba6-bc42-9b538b31143c/results/evaluation/<br />
|-<br />
|<br />
|<br />
24<br />
|<br />
https://nema.lis.illinois.edu/nema_out/6731c97a-240c-4d3d-8be9-90d715ea04e1/results/evaluation/<br />
|-<br />
|<br />
|<br />
24<br />
|<br />
https://nema.lis.illinois.edu/nema_out/2b5839b3-3012-4f76-8807-31823588ae25/results/evaluation/<br />
|-<br />
|<br />
|<br />
36<br />
|<br />
https://nema.lis.illinois.edu/nema_out/9b11a5c8-9fcf-4029-95eb-51ed561cfb5f/results/evaluation/<br />
|-<br />
|<br />
2009<br />
|<br />
30<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Latin_Set%29_Results<br />
|-<br />
|<br />
|<br />
31<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Mixed_Set%29_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2008<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2007<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
9<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2005<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-artist/index.html<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-genre/index.html<br />
|}</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX_Home&diff=87272012:MIREX Home2012-06-07T19:53:59Z<p>AndreasEhmann: /* MIREX 2012 Task Participation Poll */</p>
<hr />
<div>==Welcome to MIREX 2012==<br />
This page is under construction!!<br />
<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2012). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2012. <br />
<br />
The MIREX 2012 community will hold its annual meeting as part of [http://ismir2012.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2012, which will be held in Porto, Portugal, the 8-12 October, 2012. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 11 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2012 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open July 30, 2012.<br />
<br />
===Tasks with a '''20 August 2012''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
<br />
===Tasks with a '''27 August 2012''' deadline:===<br />
<br />
# All remaining MIREX 2012 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
=== AMS & SMS Deadlines ===<br />
<br />
We hope to open the Evalutron 6000 system on Friday, 30 September and close it on Wednesday October 12. <br />
Please sign up in the following wiki pages<br />
<br />
[https://www.music-ir.org/mirex/wiki/2012:Audio_Similarity_2012_Graders AMS sign up page]<br />
<br />
[https://www.music-ir.org/mirex/wiki/2012:Symbolic_Similarity_2012_Graders SMS sign up page]<br />
<br />
Note that you have register for MIREX wiki first.<br />
<br />
==MIREX 2012 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2012:MIREX 2012 Submission Instructions | MIREX 2012 Submission Instructions ]] including both the tutorial video and the text<br />
* The submission system will open Monday, July 30, 2012<br />
<br />
==MIREX 2012 Task Participation Poll==<br />
<br />
Please answer [https://spreadsheets.google.com/spreadsheet/viewform?authkey=CI-w_csM&hl=en_US&formkey=dG4ybGt1OUYtNHpnRF9RVm1GWjBxckE6MA#gid=0 MIREX 2012 Task Participation Poll]<br />
on your likelihood of participation in each task. <br />
Poll is going to close on Sunday July 29th 2012.<br />
<br />
==MIREX 2012 Possible Evaluation Tasks==<br />
<br />
* [[2012:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2012:Audio Cover Song Identification]]<br />
* [[2012:Audio Tag Classification]] <br />
* [[2012:Audio Music Similarity and Retrieval]]<br />
* [[2012:Symbolic Melodic Similarity]]<br />
* [[2012:Audio Onset Detection]]<br />
* [[2012:Audio Key Detection]]<br />
* [[2012:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2012:Query by Singing/Humming]]<br />
* [[2012:Audio Melody Extraction]]<br />
* [[2012:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2012:Audio Chord Estimation]]<br />
* [[2012:Query by Tapping]]<br />
* [[2012:Audio Beat Tracking]]<br />
* [[2012:Structural Segmentation]]<br />
* [[2012:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2012 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2012 poster session at ISMIR 2012 (Wednesday, 11 August 2012)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2012, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2012==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2012 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2012 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2012, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2012 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2012 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2011 Wikis==<br />
Content from MIREX 2005 - 2011 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2011]]''' <br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX_Home&diff=87262012:MIREX Home2012-06-07T19:53:37Z<p>AndreasEhmann: /* MIREX 2012 Evaluation Tasks */</p>
<hr />
<div>==Welcome to MIREX 2012==<br />
This page is under construction!!<br />
<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2012). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2012. <br />
<br />
The MIREX 2012 community will hold its annual meeting as part of [http://ismir2012.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2012, which will be held in Porto, Portugal, the 8-12 October, 2012. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 11 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2012 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open July 30, 2012.<br />
<br />
===Tasks with a '''20 August 2012''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
<br />
===Tasks with a '''27 August 2012''' deadline:===<br />
<br />
# All remaining MIREX 2012 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
=== AMS & SMS Deadlines ===<br />
<br />
We hope to open the Evalutron 6000 system on Friday, 30 September and close it on Wednesday October 12. <br />
Please sign up in the following wiki pages<br />
<br />
[https://www.music-ir.org/mirex/wiki/2012:Audio_Similarity_2012_Graders AMS sign up page]<br />
<br />
[https://www.music-ir.org/mirex/wiki/2012:Symbolic_Similarity_2012_Graders SMS sign up page]<br />
<br />
Note that you have register for MIREX wiki first.<br />
<br />
==MIREX 2012 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2012:MIREX 2012 Submission Instructions | MIREX 2012 Submission Instructions ]] including both the tutorial video and the text<br />
* The submission system will open Monday, July 30, 2012<br />
<br />
==MIREX 2012 Task Participation Poll==<br />
<br />
Please answer [https://spreadsheets.google.com/spreadsheet/viewform?authkey=CI-w_csM&hl=en_US&formkey=dG4ybGt1OUYtNHpnRF9RVm1GWjBxckE6MA#gid=0 MIREX 2012 Task Participation Poll]<br />
on your likelihood of participation in each task. <br />
Poll is going to close on Friday July 29th 2012.<br />
<br />
==MIREX 2012 Possible Evaluation Tasks==<br />
<br />
* [[2012:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2012:Audio Cover Song Identification]]<br />
* [[2012:Audio Tag Classification]] <br />
* [[2012:Audio Music Similarity and Retrieval]]<br />
* [[2012:Symbolic Melodic Similarity]]<br />
* [[2012:Audio Onset Detection]]<br />
* [[2012:Audio Key Detection]]<br />
* [[2012:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2012:Query by Singing/Humming]]<br />
* [[2012:Audio Melody Extraction]]<br />
* [[2012:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2012:Audio Chord Estimation]]<br />
* [[2012:Query by Tapping]]<br />
* [[2012:Audio Beat Tracking]]<br />
* [[2012:Structural Segmentation]]<br />
* [[2012:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2012 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2012 poster session at ISMIR 2012 (Wednesday, 11 August 2012)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2012, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2012==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2012 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2012 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2012, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2012 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2012 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2011 Wikis==<br />
Content from MIREX 2005 - 2011 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2011]]''' <br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2012:MIREX_Home&diff=87242012:MIREX Home2012-06-07T19:51:34Z<p>AndreasEhmann: /* MIREX 2012 Deadline Dates */</p>
<hr />
<div>==Welcome to MIREX 2012==<br />
This page is under construction!!<br />
<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2012). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2012. <br />
<br />
The MIREX 2012 community will hold its annual meeting as part of [http://ismir2012.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2012, which will be held in Porto, Portugal, the 8-12 October, 2012. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 11 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2012 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open July 30, 2012.<br />
<br />
===Tasks with a '''20 August 2012''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
<br />
===Tasks with a '''27 August 2012''' deadline:===<br />
<br />
# All remaining MIREX 2012 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
=== AMS & SMS Deadlines ===<br />
<br />
We hope to open the Evalutron 6000 system on Friday, 30 September and close it on Wednesday October 12. <br />
Please sign up in the following wiki pages<br />
<br />
[https://www.music-ir.org/mirex/wiki/2012:Audio_Similarity_2012_Graders AMS sign up page]<br />
<br />
[https://www.music-ir.org/mirex/wiki/2012:Symbolic_Similarity_2012_Graders SMS sign up page]<br />
<br />
Note that you have register for MIREX wiki first.<br />
<br />
==MIREX 2012 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2012:MIREX 2012 Submission Instructions | MIREX 2012 Submission Instructions ]] including both the tutorial video and the text<br />
* The submission system will open Friday, August 5, 2012<br />
<br />
==MIREX 2012 Task Participation Poll==<br />
<br />
Please answer [https://spreadsheets.google.com/spreadsheet/viewform?authkey=CI-w_csM&hl=en_US&formkey=dG4ybGt1OUYtNHpnRF9RVm1GWjBxckE6MA#gid=0 MIREX 2012 Task Participation Poll]<br />
on your likelihood of participation in each task. <br />
Poll is going to close on Friday July 29th 2012.<br />
<br />
==MIREX 2012 Evaluation Tasks==<br />
<br />
* [[2012:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2012:Audio Cover Song Identification]]<br />
* [[2012:Audio Tag Classification]] <br />
* [[2012:Audio Music Similarity and Retrieval]]<br />
* [[2012:Symbolic Melodic Similarity]]<br />
* [[2012:Audio Onset Detection]]<br />
* [[2012:Audio Key Detection]]<br />
* [[2012:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2012:Query by Singing/Humming]]<br />
* [[2012:Audio Melody Extraction]]<br />
* [[2012:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2012:Audio Chord Estimation]]<br />
* [[2012:Query by Tapping]]<br />
* [[2012:Audio Beat Tracking]]<br />
* [[2012:Structural Segmentation]]<br />
* [[2012:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2012 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2012 poster session at ISMIR 2012 (Wednesday, 11 August 2012)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2012, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2012==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2012 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2012 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2012, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2012 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2012 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2011 Wikis==<br />
Content from MIREX 2005 - 2011 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2011]]''' <br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:Structural_Segmentation&diff=80812011:Structural Segmentation2011-07-08T21:22:25Z<p>AndreasEhmann: /* Data */</p>
<hr />
<div>== Description ==<br />
<br />
The aim of the MIREX structural segmentation evaluation is to identify the key structural sections in musical audio. The segment structure (or form) is one of the most important musical parameters. It is furthermore special because musical structure -- especially in popular music genres (e.g. verse, chorus, etc.) -- is accessible to everybody: it needs no particular musical knowledge. This task was first run in 2009.<br />
<br />
== Data == <br />
<br />
=== Collections ===<br />
* The MIREX 2009 Collection: 297 pieces, most of it derived from the work of the Beatles.<br />
<br />
* MIREX 2010 RWC collection. 100 pieces of popular music. There are two ground truths. The first is the one originally included with the RWC dataset. The explanation of the second set of annotations can be found at http://hal.inria.fr/docs/00/47/34/79/PDF/PI-1948.pdf. The second set of annotations contains no labels for segments, but rather provides an annotation of segment boundaries.<br />
<br />
* MIREX 2011 dataset. The new data set contains over 1,000 annotated pieces covering a range of musical styles. The majority of the pieces have been annotated by two independent annotators. <br />
<br />
=== Audio Formats ===<br />
<br />
* CD-quality (PCM, 16-bit, 44100 Hz)<br />
* single channel (mono)<br />
<br />
== Submission Format ==<br />
<br />
Submissions to this task will have to conform to a specified format detailed below. Submissions should be packaged and contain at least two files: The algorithm itself and a README containing contact information and detailing, in full, the use of the algorithm.<br />
<br />
=== Input Data ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 44.1 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
=== Output Data ===<br />
<br />
The structural segmentation algorithms will return the segmentation in an ASCII text file for each input .wav audio file. The specification of this output file is immediately below.<br />
<br />
=== Output File Format (Structural Segmentation) ===<br />
<br />
The Structural Segmentation output file format is a tab-delimited ASCII text format. This is the same as Chris Harte's chord labelling files (.lab), and so is the same format as the ground truth as well. Onset and offset times are given in seconds, and the labels are simply letters: 'A', 'B', ... with segments referring to the same structural element having the same label.<br />
<br />
Three column text file of the format<br />
<br />
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n<br />
<onset_time(sec)>\t<offset_time(sec)>\t<label>\n<br />
...<br />
<br />
where \t denotes a tab, \n denotes the end of line. The < and > characters are not included. An example output file would look something like:<br />
<br />
0.000 5.223 A<br />
5.223 15.101 B<br />
15.101 20.334 A<br />
<br />
=== Algorithm Calling Format ===<br />
<br />
The submitted algorithm must take as arguments a SINGLE .wav file to perform the structural segmentation on as well as the full output path and filename of the output file. The ability to specify the output path and file name is essential. Denoting the input .wav file path and name as %input and the output file path and name as %output, a program called foobar could be called from the command-line as follows:<br />
<br />
foobar %input %output<br />
foobar -i %input -o %output<br />
<br />
Moreover, if your submission takes additional parameters, foobar could be called like:<br />
<br />
foobar .1 %input %output<br />
foobar -param1 .1 -i %input -o %output <br />
<br />
If your submission is in MATLAB, it should be submitted as a function. Once again, the function must contain String inputs for the full path and names of the input and output files. Parameters could also be specified as input arguments of the function. For example: <br />
<br />
foobar('%input','%output')<br />
foobar(.1,'%input','%output')<br />
<br />
=== README File ===<br />
<br />
A README file accompanying each submission should contain explicit instructions on how to to run the program (as well as contact information, etc.). In particular, each command line to run should be specified, using %input for the input sound file and %output for the resulting text file.<br />
<br />
For instance, to test the program foobar with a specific value for parameter param1, the README file would look like:<br />
<br />
foobar -param1 .1 -i %input -o %output<br />
<br />
For a submission using MATLAB, the README file could look like:<br />
<br />
matlab -r "foobar(.1,'%input','%output');quit;"<br />
<br />
== Evaluation Procedures ==<br />
At the last ISMIR conference [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich] proposed a measure for segmentation evaluation. Because of the complexity of the structural segmentation task definition, several different evaluation measures will be employed to address different aspects. It should be noted that none of the evaluation measures cares about the true labels of the sections: they only denote the clustering. This means that it does not matter if the systems produce true labels such as "chorus" and "verse", or arbitrary labels such as "A" and "B".<br />
<br />
=== Boundary retrieval ===<br />
'''Hit rate''' Found segment boundaries are accepted to be correct if they are within 0.5s ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007]) or 3s ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008]) from a border in the ground truth. Based on the matched hits, ''boundary retrieval recall rate'', ''boundary retrieval precision rate'', and ''boundary retrieval F-measure'' are be calculated.<br />
<br />
'''Median deviation''' Two median deviation measure between boundaries in the result and ground truth are calculated: ''median true-to-guess'' is the median time from boundaries in ground truth to the closest boundaries in the result, and ''median guess-to-true'' is similarly the median time from boundaries in the result to boundaries in ground truth. ([http://ismir2007.ismir.net/proceedings/ISMIR2007_p051_turnbull.pdf Turnbull et al. ISMIR2007])<br />
<br />
=== Frame clustering ===<br />
Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets <math>P_E</math> (for the system result) and <math>P_A</math> (for the ground truth). The ''pairwise precision rate'' can be calculated by <math>P = \frac{|P_E \cap P_A|}{|P_E|}</math>, ''pairwise recall rate'' by <math>R = \frac{|P_E \cap P_A|}{|P_A|}</math>, and ''pairwise F-measure'' by <math>F=\frac{2 P R}{P + R}</math>. ([http://dx.doi.org/10.1109/TASL.2007.910781 Levy & Sandler TASLP2008])<br />
<br />
=== Normalised conditional entropies ===<br />
Over- and under segmentation based evaluation measures proposed in [http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf Lukashevich ISMIR2008].<br />
Structure descriptions are represented as frame sequences with the associated cluster information (similar to the Frame clustering measure). Confusion matrix between the labels in ground truth and the result is calculated. The matrix C is of size |L_A| * |L_E|, i.e., number of unique labels in the ground truth times number of unique labels in the result. From the confusion matrix, the joint distribution is calculated by normalising the values with the total number of frames F:<br />
<br />
<math>p_{i,j} = C_{i,j} / F</math><br />
<br />
Similarly, the two marginals are calculated:<br />
<br />
<math>p_i^a = \sum_{j=1}^{|L_E|} C{i,j}/F</math>, and<br />
<br />
<math>p_j^e = \sum_{i=1}^{|L_A|} C{i,j}/F</math><br />
<br />
Conditional distributions:<br />
<br />
<math>p_{i,j}^{a|e} = C_{i,j} / \sum_{i=1}^{|L_A|} C{i,j}</math>, and<br />
<br />
<math>p_{i,j}^{e|a} = C_{i,j} / \sum_{j=1}^{|L_E|} C{i,j}</math><br />
<br />
The conditional entropies will then be<br />
<br />
<math>H(E|A) = - \sum_{i=1}^{|L_A|} p_i^a \sum_{j=1}^{|L_E|} p_{i,j}^{e|a} \log_2(p_{i,j}^{e|a})</math>, and<br />
<br />
<math>H(A|E) = - \sum_{j=1}^{|L_E|} p_j^e \sum_{i=1}^{|L_A|} p_{i,j}^{a|e} \log_2(p_{i,j}^{a|e})</math><br />
<br />
The final evaluation measures will then be the oversegmentation score<br />
<br />
<math>S_O = 1 - \frac{H(E|A)}{\log_2(|L_E|)}</math> , and the undersegmentation score<br />
<br />
<math>S_U = 1 - \frac{H(A|E)}{\log_2(|L_A|)}</math><br />
<br />
== Relevant Development Collections == <br />
*Jouni Paulus's [http://www.cs.tut.fi/sgn/arg/paulus/structure.html structure analysis page] links to a corpus of 177 Beatles songs ([http://www.cs.tut.fi/sgn/arg/paulus/beatles_sections_TUT.zip zip file]). The Beatles annotations are not a part of the TUTstructure07 dataset. That dataset contains 557 songs, a list of which is available [http://www.cs.tut.fi/sgn/arg/paulus/TUTstructure07_files.html here].<br />
<br />
*Ewald Peiszer's [http://www.ifs.tuwien.ac.at/mir/audiosegmentation.html thesis page] links to a portion of the corpus he used: 43 non-Beatles pop songs (including 10 J-pop songs) ([http://www.ifs.tuwien.ac.at/mir/audiosegmentation/dl/ep_groundtruth_excl_Paulus.zip zip file]).<br />
<br />
These public corpora give a combined 220 songs.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 24 hours will be imposed on analysis times. Submissions exceeding this limit may not receive a result.<br />
<br />
<br />
<br />
== Submission opening date ==<br />
<br />
Friday August 5th 2011<br />
<br />
== Submission closing date ==<br />
Friday September 2nd 2011<br />
<br />
== Potential Participants ==<br />
name / email</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:Audio_Classification_(Train/Test)_Tasks&diff=80802011:Audio Classification (Train/Test) Tasks2011-07-08T20:48:19Z<p>AndreasEhmann: /* Data */</p>
<hr />
<div>= Audio Classification (Test/Train) tasks =<br />
<br />
== Description ==<br />
<br />
Many tasks in music classification can be characterized into a two-stage process: training classification models using labeled data and testing the models using new/unseen data. Therefore, we propose this "meta" task which includes various audio classification tasks that follow this Train/Test process. For MIREX 2011, five classification sub-tasks are included: <br />
<br />
* Audio Classical Composer Identification<br />
* Audio US Pop Music Genre Classification <br />
* Audio Latin Music Genre Classification <br />
* Audio Mood Classification<br />
<br />
All five classification tasks were conducted in previous MIREX runs (please see ). This page presents the evaluation of these tasks, including the datasets as well as the submission rules and formats.<br />
<br />
<br />
=== Task specific mailing list ===<br />
In the past we have use a specific mailing list for the discussion of this task and related tasks. This year, however, we are asking that all discussions take place on the MIREX [https://mail.lis.illinois.edu/mailman/listinfo/evalfest "EvalFest" list]. If you have an question or comment, simply include the task name in the subject heading.<br />
<br />
== Data ==<br />
=== Audio Classical Composer Identification ===<br />
This dataset requires algorithms to classify music audio according to the composer of the track (drawn from a collection of performances of a variety of classical music genres). The collection used at MIREX 2009 will be re-used.<br />
<br />
Collection statistics: <br />
* 2772 30-second 22.05 kHz mono wav clips<br />
* 11 "classical" composers (252 clips per composer), including:<br />
** Bach<br />
** Beethoven<br />
** Brahms<br />
** Chopin<br />
** Dvorak<br />
** Handel<br />
** Haydn<br />
** Mendelssohn<br />
** Mozart<br />
** Schubert<br />
** Vivaldi<br />
<br />
<br />
=== Audio US Pop Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of US Pop music tracks). The MIREX 2007 Genre dataset will be re-used, which was drawn from the USPOP 2002 and USCRAP collections.<br />
<br />
Collection statistics: <br />
* 7000 30-second audio clips in 22.05kHz mono WAV format <br />
* 10 genres (700 clips from each genre), including:<br />
** Blues<br />
** Jazz<br />
** Country/Western<br />
** Baroque<br />
** Classical<br />
** Romantic<br />
** Electronica<br />
** Hip-Hop<br />
** Rock<br />
** HardRock/Metal<br />
<br />
<br />
=== Audio Latin Music Genre Classification === <br />
This dataset requires algorithms to classify music audio according to the genre of the track (drawn from a collection of Latin popular and dance music, sourced from Brazil and hand labeled by music experts). Carlos Silla's (cns2 (at) kent (dot) ac (dot) uk) Latin popular and dance music dataset [http://ismir2008.ismir.net/papers/ISMIR2008_106.pdf] will be re-used. This collection is likely to contain a greater number of styles of music that will be differentiated by rhythmic characteristics than the MIREX 2007 dataset.<br />
<br />
Collection statistics: <br />
* 3,227 audio files in 22.05kHz mono WAV format <br />
* 10 Latin music genres, including:<br />
** Axe<br />
** Bachata<br />
** Bolero<br />
** Forro <br />
** Gaucha<br />
** Merengue<br />
** Pagode<br />
** Sertaneja<br />
** Tango<br />
<br />
<br />
=== Audio Mood Classification ===<br />
This dataset requires algorithms to classify music audio according to the mood of the track (drawn from a collection of production msuic sourced from the APM collection [http://www.apmmusic.com]). The MIREX 2007 Mood Classification dataset [http://ismir2008.ismir.net/papers/ISMIR2008_263.pdf] will be re-used. <br />
<br />
Collection statistics: <br />
* 600 30 second audio clips in 22.05kHz mono WAV format selected from the APM collection [http://www.apmmusic.com], and labeled by human judges using the Evalutron6000 system. <br />
* 5 mood categories [http://ismir2007.ismir.net/proceedings/ISMIR2007_p067_hu.pdf] each of which contains 120 clips:<br />
**Cluster_1: passionate, rousing, confident,boisterous, rowdy <br />
**Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured <br />
**Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding <br />
**Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry <br />
**Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral<br />
<br />
== Audio Formats ==<br />
For all datasets, participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 22 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV<br />
<br />
<br />
== Evaluation ==<br />
This section first describes evaluation methods common to all the datasets, then specifies settings unique to each of the tasks. <br />
<br />
Participating algorithms will be evaluated with 3-fold cross validation. For '''Artist Identification''' and '''Classical Composer Classification''', album filtering will be used the test and training splits, i.e. training and test sets will contain tracks from different albums; for '''US Pop Genre Classification''' and '''Latin Genre Classification''', artist filtering will be used the test and training splits, i.e. training and test sets will contain different artists. <br />
<br />
The raw classification (identification) accuracy, standard deviation and a confusion matrix for each algorithm will be computed.<br />
<br />
Classification accuracies will be tested for statistically significant differences using Friedman's Anova with Tukey-Kramer honestly significant difference (HSD) tests for multiple comparisons. This test will be used to rank the algorithms and to group them into sets of equivalent performance. <br />
<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
<br />
== Submission Format ==<br />
=== File I/O Format ===<br />
The audio files to be used in these tasks will be specified in a simple ASCII list file. The formats for the list files are specified below: <br />
<br />
<br />
==== Feature extraction list file ====<br />
The list file passed for feature extraction will be a simple ASCII list file. This file will contain one path per line with no header line.<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Training list file ====<br />
The list file passed for model training will be a simple ASCII list file. This file will contain one path per line, followed by a tab character and the class (artist, genre or mood) label, again with no header line.<br />
<br />
I.e. <br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav rock<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
==== Test (classification) list file ====<br />
The list file passed for testing classification will be a simple ASCII list file identical in format to the Feature extraction list file. This file will contain one path per line with no header line.<br />
<br />
I.e.<br />
<example path and filename><br />
<br />
E.g. <br />
/path/to/track1.wav<br />
/path/to/track2.wav<br />
...<br />
<br />
<br />
==== Classification output file ====<br />
Participating algorithms should produce a simple ASCII list file identical in format to the Training list file. This file will contain one path per line, followed by a tab character and the artist label, again with no header line.<br />
<br />
I.e.<br />
<example path and filename>\t<class label><br />
<br />
E.g.<br />
/path/to/track1.wav classical<br />
/path/to/track2.wav blues<br />
...<br />
<br />
<br />
=== Submission calling formats ===<br />
Algorithms should divide their feature extraction and training/classification into separate runs. This will facilitate a single feature extraction step for the task, while training and classification can be run for each cross-validation fold.<br />
<br />
Hence, participants should provide two executables or command line parameters for a single executable to run the two separate processes.<br />
<br />
Executables will have to accept the paths to the aforementioned list files as command line parameters.<br />
<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique file names will be assigned to each audio track.<br />
<br />
<br />
==== Example submission calling formats ====<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
Train.sh /path/to/scratch/folder /path/to/trainListFile.txt <br />
Classify.sh /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -train /path/to/scratch/folder /path/to/trainListFile.txt <br />
myAlgo.sh -classify /path/to/scratch/folder /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
Multi-processor compute nodes will be used to run this task, however, we ask that submissions use no more than 4 cores (as we will be running a lot of submissions and will need to run some in parallel). Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 1, 2 or 4 thread/core configurations. <br />
<br />
extractFeatures.sh -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
TrainAndClassify.sh -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
myAlgo.sh -extract -numThreads 4 /path/to/scratch/folder /path/to/featureExtractionListFile.txt<br />
myAlgo.sh -TrainAndClassify -numThreads 4 /path/to/scratch/folder /path/to/trainListFile.txt /path/to/testListFile.txt /path/to/outputListFile.txt<br />
<br />
=== Packaging submissions ===<br />
<br />
* All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guaranteed). [mailto:mirproject@lists.lis.uiuc.edu IMIRSEL] should be notified of any dependencies that you cannot include with your submission at the earliest opportunity (in order to give them time to satisfy the dependency).<br />
* Be sure to follow the [[2006:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2011 Submission Instructions]]<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Command line calling format for all executables including examples<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
* Any required environments/architectures (and versions) such as Matlab, Java, Python, Bash, Ruby etc.<br />
* Any special notice regarding to running your algorithm<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
=== Time and hardware limits ===<br />
Due to the potentially high number of participants in this and other audio tasks, hard limits on the runtime of submissions will be imposed.<br />
<br />
A hard limit of 24 hours will be imposed on feature extraction times.<br />
<br />
A hard limit of 48 hours will be imposed on the 3 training/classification cycles, leading to a total runtime limit of 72 hours for each submission.<br />
<br />
=== Submission opening date ===<br />
<br />
Friday August 5th 2011<br />
<br />
=== Submission closing date ===<br />
<br />
Friday August 26th 2011<br />
<br />
=== Potential Participants ===<br />
name / email<br />
<br />
= Participation in previous years and Links to Results =<br />
<br />
<br />
{| border="1"<br />
|-<br />
| <br />
'''Year'''<br />
| <br />
'''Participating Algorithms ''' <br />
| <br />
'''URL'''<br />
|-<br />
|<br />
|-<br />
|<br />
2010<br />
|<br />
27<br />
|<br />
https://nema.lis.illinois.edu/nema_out/4ffcb482-b83c-4ba6-bc42-9b538b31143c/results/evaluation/<br />
|-<br />
|<br />
|<br />
24<br />
|<br />
https://nema.lis.illinois.edu/nema_out/6731c97a-240c-4d3d-8be9-90d715ea04e1/results/evaluation/<br />
|-<br />
|<br />
|<br />
24<br />
|<br />
https://nema.lis.illinois.edu/nema_out/2b5839b3-3012-4f76-8807-31823588ae25/results/evaluation/<br />
|-<br />
|<br />
|<br />
36<br />
|<br />
https://nema.lis.illinois.edu/nema_out/9b11a5c8-9fcf-4029-95eb-51ed561cfb5f/results/evaluation/<br />
|-<br />
|<br />
2009<br />
|<br />
30<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Latin_Set%29_Results<br />
|-<br />
|<br />
|<br />
31<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Mixed_Set%29_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2008<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2007<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
9<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2005<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-artist/index.html<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-genre/index.html<br />
|}</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:Audio_Classification_(Train/Test)_Tasks&diff=80422011:Audio Classification (Train/Test) Tasks2011-07-08T19:40:00Z<p>AndreasEhmann: /* Description */</p>
<hr />
<div>= Audio Classification (Test/Train) tasks =<br />
<br />
== Description ==<br />
<br />
Many tasks in music classification can be characterized into a two-stage process: training classification models using labeled data and testing the models using new/unseen data. Therefore, we propose this "meta" task which includes various audio classification tasks that follow this Train/Test process. For MIREX 2011, five classification sub-tasks are included: <br />
<br />
* Audio Classical Composer Identification<br />
* Audio US Pop Music Genre Classification <br />
* Audio Latin Music Genre Classification <br />
* Audio Mood Classification<br />
<br />
All five classification tasks were conducted in previous MIREX runs (please see ). This page presents the evaluation of these tasks, including the datasets as well as the submission rules and formats.<br />
<br />
== Participation in previous years ==<br />
<br />
<br />
{| border="1"<br />
|-<br />
| <br />
'''Year'''<br />
| <br />
'''Participating Algorithms ''' <br />
| <br />
'''URL'''<br />
|-<br />
|<br />
2009<br />
|<br />
30<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Latin_Set%29_Results<br />
|-<br />
|<br />
|<br />
31<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Genre_Classification_%28Mixed_Set%29_Results<br />
|-<br />
|<br />
|<br />
33<br />
|<br />
https://www.music-ir.org/mirex/wiki/2009:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2008<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
11<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/mirex/wiki/2008:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2007<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Artist_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Classical_Composer_Identification_Results<br />
|-<br />
|<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Genre_Classification_Results<br />
|-<br />
|<br />
|<br />
9<br />
|<br />
https://www.music-ir.org/mirex/wiki/2007:Audio_Music_Mood_Classification_Results<br />
|-<br />
|<br />
2005<br />
|<br />
7<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-artist/index.html<br />
|-<br />
|<br />
|<br />
13<br />
|<br />
https://www.music-ir.org/evaluation/mirex-results/audio-genre/index.html<br />
|}</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80412011:MIREX Home2011-07-08T19:39:43Z<p>AndreasEhmann: /* MIREX 2011 Evaluation Tasks */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 26 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open August 5, 2011.<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
* The submission system will open Friday, August 5, 2011<br />
<br />
==MIREX 2011 Task Participation Poll==<br />
<br />
Please answer [https://spreadsheets.google.com/spreadsheet/viewform?authkey=CI-w_csM&hl=en_US&formkey=dG4ybGt1OUYtNHpnRF9RVm1GWjBxckE6MA#gid=0 MIREX 2011 Task Participation Poll]<br />
on your likelihood of participation in each task. <br />
Poll is going to close on Friday July 29th 2011.<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80262011:MIREX Home2011-07-08T19:06:02Z<p>AndreasEhmann: /* MIREX 2011 Deadline Dates */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 26 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations. The submission system will open August 5, 2011.<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
* The submission system will open Friday, August 5, 2011<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80242011:MIREX Home2011-07-08T19:05:29Z<p>AndreasEhmann: /* MIREX 2011 Submission Instructions */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 26 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations.<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
* The submission system will open Friday, August 5, 2011<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80112011:MIREX Home2011-07-08T18:43:31Z<p>AndreasEhmann: /* MIREX 2005 - 2010 Wikis */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 26 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations.<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2010:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80102011:MIREX Home2011-07-08T18:42:12Z<p>AndreasEhmann: /* Welcome to MIREX 2011 */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held Thursday, October 26 during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations.<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2009:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80092011:MIREX Home2011-07-08T18:40:15Z<p>AndreasEhmann: /* MIREX 2011 Deadline Dates */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held at a time to be determined during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations.<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2009:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80082011:MIREX Home2011-07-08T18:40:00Z<p>AndreasEhmann: /* MIREX 2011 Deadline Dates */</p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held at a time to be determined during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations.<br />
<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2009:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmannhttps://www.music-ir.org/mirex/w/index.php?title=2011:MIREX_Home&diff=80072011:MIREX Home2011-07-08T18:39:27Z<p>AndreasEhmann: </p>
<hr />
<div>==Welcome to MIREX 2011==<br />
This is the main page for the sixth running of the Music Information Retrieval Evaluation eXchange (MIREX 2011). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.illinois.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.illinois.edu UIUC]) is the principal organizer of MIREX 2011. <br />
<br />
The MIREX 2011 community will hold its annual meeting as part of [http://ismir2011.ismir.net/ The 12th International Conference on Music Information Retrieval], ISMIR 2011, which will be held in Miami, Florida, the week of October the 23rd, 2011. The MIREX plenary (working lunch) and poster sessions will be held at a time to be determined during the conference.<br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==MIREX 2011 Deadline Dates==<br />
We have <b>very</b> tight scheduling constraints this year because of the early convening of ISMIR 2010 in Utrecht.<br />
<br />
We have two sets of deadlines for submissions. We have to stagger the deadlines because of runtime and human evaluation considerations.<br />
<br />
<br />
===Tasks with a '''26 August 2011''' deadline:===<br />
# Audio Classification (Train/Test) Tasks<br />
# Audio Music Similarity and Retrieval<br />
# Symbolic Melodic Similarity<br />
===Tasks with a '''2 September 2011''' deadline:===<br />
<br />
# All remaining MIREX 2011 tasks.<br />
<br />
<i><b>Nota Bene:</b> </i>In the past we have been rather flexible about deadlines. This year, however, we simply do not have the time flexibility, sorry.<br />
<br />
Please, please, please, let's start getting those submissions made. The sooner we have the code, the sooner we can start running the evaluations.<br />
<br />
PS: If you have a slower running algorithm, help us help you by getting your code in ASAP. Please do pay attention to runtime limits.<br />
<br />
==MIREX 2011 Submission Instructions==<br />
* Be sure to read through the rest of this page<br />
* Be sure to read though the task pages for which you are submitting<br />
* Be sure to follow the [[2009:Best Coding Practices for MIREX | Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[2011:MIREX 2011 Submission Instructions | MIREX 2011 Submission Instructions ]] including both the tutorial video and the text<br />
<br />
<br />
==MIREX 2011 Evaluation Tasks==<br />
<br />
* [[2011:Audio Classification (Train/Test) Tasks]], incorporating:<br />
** Audio Artist Identification<br />
** Audio US Pop Genre Classification<br />
** Audio Latin Genre Classification<br />
** Audio Music Mood Classification<br />
** Audio Classical Composer Identification<br />
* [[2011:Audio Cover Song Identification]]<br />
* [[2011:Audio Tag Classification]] <br />
* [[2011:Audio Music Similarity and Retrieval]]<br />
* [[2011:Symbolic Melodic Similarity]]<br />
* [[2011:Audio Onset Detection]]<br />
* [[2011:Audio Key Detection]]<br />
* [[2011:Real-time Audio to Score Alignment (a.k.a Score Following)]]<br />
* [[2011:Query by Singing/Humming]]<br />
* [[2011:Audio Melody Extraction]]<br />
* [[2011:Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[2011:Audio Chord Estimation]]<br />
* [[2011:Query by Tapping]]<br />
* [[2011:Audio Beat Tracking]]<br />
* [[2011:Structural Segmentation]]<br />
* [[2011:Audio Tempo Estimation]]<br />
<br />
===Note to New Participants===<br />
Please take the time to read the following review article that explains the history and structure of MIREX.<br />
<br />
Downie, J. Stephen (2008). The Music Information Retrieval Evaluation Exchange (2005-2007):<br><br />
A window into music information retrieval research.''Acoustical Science and Technology 29'' (4): 247-255. <br><br />
Available at: [http://dx.doi.org/10.1250/ast.29.247 http://dx.doi.org/10.1250/ast.29.247]<br />
<br />
===Note to All Participants===<br />
Because MIREX is premised upon the sharing of ideas and results, '''ALL''' MIREX participants are expected to:<br />
<br />
# submit a DRAFT 2-3 page extended abstract PDF in the ISMIR format about the submitted programme(s) to help us and the community better understand how the algorithm works when submitting their programme(s).<br />
# submit a FINALIZED 2-3 page extended abstract PDF in the ISMIR format prior to ISMIR 2011 for posting on the respective results pages (sometimes the same abstract can be used for multiple submissions; in many cases the DRAFT and FINALIZED abstracts are the same)<br />
# present a poster at the MIREX 2011 poster session at ISMIR 2011 (Wednesday, 11 August 2011)<br />
<br />
<br />
===Software Dependency Requests===<br />
If you have not submitted to MIREX before or are unsure whether IMIRSEL/NEMA currently supports some of the software/architecture dependencies for your submission a [https://spreadsheets.google.com/embeddedform?formkey=dDltRjc4NDBDdkZiaF9qZXV0bU5ScUE6MA dependency request form is available]. Please submit details of your dependencies on this form and the IMIRSEL team will attempt to satisfy them for you. <br />
<br />
Due to the high volume of submissions expected at MIREX 2011, submissions with difficult to satisfy dependencies that the team has not been given sufficient notice of may result in the submission being rejected.<br />
<br />
<br />
Finally, you will also be expected to detail your software/architecture dependencies in a README file to be provided to the submission system.<br />
<br />
==Getting Involved in MIREX 2011==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2011 the best yet.<br />
<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2011 tasks. Subscription information at: <br />
[https://mail.lis.illinois.edu/mailman/listinfo/evalfest EvalFest Central]. <br />
<br />
If you are participating in MIREX 2011, it is VERY IMPORTANT that you are subscribed to EvalFest. Deadlines, task updates and other important information will be announced via this mailing list. Please use the EvalFest for discussion of MIREX task proposals and other MIREX related issues. This wiki (MIREX 2011 wiki) will be used to embody and disseminate task proposals, however, task related discussions should be conducted on the MIREX organization mailing list (EvalFest) rather than on this wiki, but should be summarized here. <br />
<br />
Where possible, definitions or example code for new evaluation metrics or tasks should be provided to the IMIRSEL team who will embody them in software as part of the NEMA analytics framework, which will be released to the community at or before ISMIR 2011 - providing a standardised set of interfaces and output to disciplined evaluation procedures for a great many MIR tasks.<br />
<br />
<br />
===Wiki Participation===<br />
Please create an account via: [[Special:Userlogin]].<br />
<br />
Please note that because of "spam-bots", MIREX wiki registration requests may be moderated by IMIRSEL members. It might take up to 24 hours for approval (Thank you for your patience!).<br />
<br />
==MIREX 2005 - 2010 Wikis==<br />
Content from MIREX 2005 - 2010 are available at:<br />
<br />
'''[[2009:Main_Page|MIREX 2010]]''' <br />
'''[[2009:Main_Page|MIREX 2009]]''' <br />
'''[[2008:Main_Page|MIREX 2008]]''' <br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
'''[[2005:Main_Page|MIREX 2005]]'''</div>AndreasEhmann