https://www.music-ir.org/mirex/w/api.php?action=feedcontributions&user=WikiSysop&feedformat=atomMIREX Wiki - User contributions [en]2024-03-29T01:21:17ZUser contributionsMediaWiki 1.31.1https://www.music-ir.org/mirex/w/index.php?title=User:CameronJones&diff=5863User:CameronJones2009-10-10T20:27:44Z<p>WikiSysop: Creating user page with biography of new user.</p>
<hr />
<div>UIUC</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=User_talk:CameronJones&diff=5864User talk:CameronJones2009-10-10T20:27:44Z<p>WikiSysop: Welcome!</p>
<hr />
<div>'''Welcome to ''MIREX 2010''!'''<br />
We hope you will contribute much and well.<br />
You will probably want to read the [[Help:Contents|help pages]].<br />
Again, welcome and have fun! [[User:WikiSysop|WikiSysop]] 20:27, 10 October 2009 (UTC)</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2008:Main_Page&diff=35132008:Main Page2007-12-11T18:27:07Z<p>WikiSysop: </p>
<hr />
<div>==Welcome to MIREX 2008==<br />
This is the main page for the Third Music Information Retrieval Evaluation eXchange (MIREX 2008). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.uiuc.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.uiuc.edu UIUC]) is the principal organizer of MIREX 2008. <br />
<br />
The MIREX 2007 community will hold its annual plenary meeting as part of [http://ismir2008.ismir.net/ The 9th International Conference on Music Information Retrieval], ISMIR 2008, which will be held in Philadelphia, Pennsylvania, USA, from September 14th to September 18th, 2008. <br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==Getting Involved in MIREX 2008==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2008 the best yet.<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2008 tasks. Subscription information at: <br />
https://mail.isrl.uiuc.edu/mailman/listinfo/evalfest <br />
===Wiki Participation===<br />
'''''Please note that you must create a NEW login for this wiki even if you have a login that you previously used for editing the MIREX 2005 or 2006 or 2007 wikis.'''''<br />
<br />
Please create an account via: [https://www.music-ir.org/mirex/2007/index.php?title=Special:Userlogin&type=signup&returnto=Special:Userlogin Create Account].<br />
<br />
==MIREX 2005 - 2007 Wikis==<br />
This is the new wiki for MIREX 2008. The wikis for MIREX 2005 - 2007 are available at:<br />
<br />
'''[[2007:Main_Page|MIREX 2007]]''' <br />
https://www.music-ir.org/mirex/2007/<br />
<br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
https://www.music-ir.org/mirex/2006/<br />
<br />
'''[[2005:Main_Page|MIREX 2005]]''' <br />
https://www.music-ir.org/mirex/2005/<br />
<br />
You can interlink between this wiki and the previous wikis using '''2005:''' prefix on links to connect to pages in MIREX 2005 and '''2006:''' for MIREX 2006 and '''2007:''' for MIREX 2007.</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Query_by_Singing/Humming&diff=28132007:Query by Singing/Humming2007-12-11T18:20:54Z<p>WikiSysop: </p>
<hr />
<div>== Status ==<br />
The goal of the Query-by-Singing/Humming (QBSH) task is the evaluation of MIR systems that take as query input queries sung or hummed by real-world users. More information can be found in:<br />
<br />
* [[2006:QBSH:_Query-by-Singing/Humming MIREX2006 QBSH Task Proposal]]<br />
* [[2006:QBSH_Discussion_Page MIREX2006 QBSH Task Discussion]]<br />
<br />
Please feel free to edit this page.<br />
<br />
== Query Data ==<br />
1. Roger Jang's corpus ([http://neural.cs.nthu.edu.tw/jang2/dataSet/childSong4public/QBSH-corpus/ MIREX2006 QBSH corpus]) which is comprised of 2797 queries along with 48 ground-truth MIDI files. All queries are from the beginning of references. <br />
<br />
2. ThinkIT corpus comprised of 355 queries and 106 monophonic ground-truth midi files (with MIDI 0 or 1 format). There are no "singing from beginning" gurantee. This corpus will be published after the task running.<br />
<br />
3. Noise MIDI will be the 5000+ Essen collection(can be accessed from http://www.esac-data.org/).<br />
<br />
To build a large test set which can reflect real-world queries, it is suggested that every participant makes a contribution to the evaluation corpus.<br />
<br />
== Task description == <br />
Classic QBSH evaluation:<br />
* '''Input''': human singing/humming snippets (.wav). Queries are from Roger Jang's corpus and ThinkIT corpus.<br />
* '''Database''': ground-truth and noise midi files(which are monophonic). Comprised of 48+106 Roger Jang's and ThinkIT's ground-truth along with a cleaned version of Essen Database(2000+ MIDIs which are used last year) <br />
* '''Output''': top-20 candidate list. <br />
* '''Evaluation''': Mean Reciprocal Rank (MRR) and Top-X hit rate.<br />
<br />
To make algorithms able to share intermediate steps, participants are encouraged to submit separate transcriber and matcher modules instead of integrated ones, which is according to Rainer Typke's suggestion. So transcribers and matchers from different submissions could work together with the same pre-defined interface and thus for us it's possible to find the best combination. Besides, note based approaches (symbolic approaches) and pitch contour based approaches (non-symbolic approaches?) are compared.<br />
<br />
[[Image:framework.jpg]]<br />
<br />
== Participants == <br />
If you think there is a slight chance that you might want to participate, please add your name and e-mail address to this list<br />
* Xiao Wu & Ming Li ({xwu,mli} at hccl dot ioa dot ac dot cn)<br />
* Mohamed Sordo & Maarten Grachten (e-mail at my [http://waits.cp.jku.at/~maarten/ homepage])<br />
* Niko Mikkila (mikkila at cs dot helsinki dot fi)<br />
* Rainer Typke (rainer dot typke at ofai dot at) (note matcher; I need a MySQL database to participate)<br />
* Carlos G├│mez (cgomez at ldc dot usb dot ve) (note matcher)<br />
* Ean Nugent (nugente at andrews dot edu) (I would like more background information)<br />
* Jiang Danning (jiangdn at cn dot ibm dot com)<br />
* J.-S. Roger Jang (jang at cs dot nthu dot edu dot tw)<br />
* Alexandra Uitdenbogerd (sandrau at rmit dot edu dot au)<br />
* Ming Xuan, Zou (felix at wayne dot cs dot nthu dot edu dot tw)<br />
* Shen huang & Lei wang ((shenhuang,leiwang) at hitic dot ia dot ac dot cn )<br />
<br />
== Interface suggestion commented by Xiao Wu ==<br />
1. Database indexing/building. Calling format should look like<br />
''indexing %db_list% %dir_workspace_root%''<br />
where ''db_list'' is the input list of database midi files named as ''uniq_key.mid''. For example:<br />
''./QBSH/Database/00001.mid''<br />
''./QBSH/Database/00002.mid''<br />
''./QBSH/Database/00003.mid''<br />
''./QBSH/Database/00004.mid''<br />
...<br />
Output indexed files are placed into ''dir_workspace_root''.<br />
<br />
2. Note transcriber. Calling format:<br />
''note_transcriber %query.list% %dir_query_note%''<br />
Each input file ''dir_query/query_xxxxx.wav'' in ''query.list'' outputs a transcription ''dir_query_note/query_xxxxx.note'', and each text line of the generated note file represents a query note formated as ''note %onset_time% %duration% %midi_note%''. Example:<br />
''note 2000 250 62.25''<br />
''note 2250 250 62.03''<br />
''note 2500 500 64.42''<br />
''note 3200 220 62.30''<br />
...<br />
Here ''onset_time'' and ''duration'' are counted in millisecond.<br />
<br />
3. Note matcher. Calling format:<br />
''note_matcher %note.list% %result%''<br />
where ''note.list'' looks like <br />
''dir_query_note/query_00001.note''<br />
''dir_query_note/query_00002.note''<br />
''dir_query_note/query_00003.note''<br />
...<br />
and the result file gives top-20 candidates(if has) for each query:<br />
''query_00001: 00025 01003 02200 ... ''<br />
''query_00002: 01547 02313 07653 ... ''<br />
''query_00003: 03142 00320 00973 ... ''<br />
...<br />
<br />
4. Pitch tracker. Calling format:<br />
''pitch_tracker %query.list% %dir_query_pitch%''<br />
Each input file ''dir_query/query_xxxxx.wav'' in ''query.list'' outputs a corresponding transcription<br />
''dir_query_pitch/query_xxxxx.pitch'' which gives the pitch sequence in midi note scale with the resolution of 10ms:<br />
''0''<br />
''0''<br />
''62.23''<br />
''62.25''<br />
''62.21''<br />
...<br />
Thus a query with ''x'' seconds should output a pitch file with ''100*x'' lines. Places of silence/rest are set to be 0. <br />
<br />
5. Pitch matcher. Similar with note matcher:<br />
''pitch_matcher %pitch.list% %result%''<br />
<br />
6. Hybrid matcher. Both note and pitch are utilized. Calling format:<br />
''note_pitch_matcher %note.list% %pitch.list% %result%''<br />
<br />
== Comments from Xiao Wu ==<br />
ThinkIT QBH corpus now is available at [http://159.226.60.224/en/Thinkit.QBH.corpus.rar TITcorpus]. In all there are 355 audio files along with 106 MIDI files.<br />
<br />
== Comments from Roger Jang ==<br />
Is there any time constraint on running each query against 5000+ MIDIs? [It seems a little bit quiet on this page.]<br />
== Comments from Xiao Wu ==<br />
To Roger: Stephen suggested to use the "cleaned version" of essen folks(2000+ MIDIs which are also adopted in MIREX QBSH 2006). So the problem size is not that large.<br />
<br />
== Comments from Xiao Wu ==<br />
It should be noticed the list parameters in command line such as "query.list" and "note.list" are list FILEs instead of multiline arguments. Thanks Carlos for pointing out this ambiguity.</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Audio_Music_Similarity_and_Retrieval&diff=27342007:Audio Music Similarity and Retrieval2007-12-11T18:20:03Z<p>WikiSysop: fixed hyperlinks to mirex2006 wiki</p>
<hr />
<div>== Status ==<br />
A provisional specification of the Audio Music Similarity task is detailed below. This proposal may be refined based on feedback from the particpants.<br />
<br />
Note that audio music similarity and retrieval algorithms have been evaluated at MIREX 2006 ( [[2006:Audio_Music_Similarity_and_Retrieval]] ).<br />
<br />
Related MIREX 2007 task proposals: <br />
* [[Audio Music Mood Classification]]<br />
* [[Audio Artist Identification]]<br />
* [[Audio Genre Classification]]<br />
* [[Audio Cover song identification]]<br />
<br />
Please feel free to edit this page.<br />
<br />
== Data ==<br />
Collection statistics: 7000 30-second audio clips in 22.05kHz mono wav format drawn from 10 genres (700 clips from each genre).<br />
Genres:<br />
*Blues<br />
*Jazz<br />
*Country/Western<br />
*Baroque<br />
*Classical<br />
*Romantic<br />
*Electronica<br />
*Hip-Hop<br />
*Rock<br />
*HardRock/Metal<br />
<br />
== Evaluation ==<br />
<br />
Two distinct evaluations will be performed<br />
* Human Evaluation<br />
* Objective statistics derived from the results lists<br />
<br />
Note that at MIREX 2006 particpating algorithms were required to return full distance matrices showing the distance between all tracks. This year we are also going to allow a sparse distance matrix format (detailed below) where only the<br />
ranks of the top 100 results for each query in the collection are returned.<br />
<br />
=== Human Evaluation ===<br />
<br />
The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system. This year algorithms will be presented with the same 30 second preview clip that will be reviewed by the human evaluators. <br />
<br />
* Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query. <br />
* ~120 randomly selected queries, 5 results per query, 1 set of eyes, ~10 participating labs<br />
* Higher number of queries preferred as IR research indicates variance is in queries<br />
* The songs by the same artist as the query will be filtered out of each result list (artist-filtering) to avoid colouring an evaluators judgement (a cover song or song by the same artist in a result list is likely to reduce the relative ranking of other similar but independent songs - use of songs by the same artist may allow over-fitting to affect the results)<br />
* It will be possible for researchers to use this data for other types of system comparisons after MIREX 2007 results have been finalized.<br />
* Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006<br />
* Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)<br />
<br />
=== Objective Statistics derived from the distance matrix ===<br />
<br />
Statistics of each distance matrix will be calculated including:<br />
<br />
* Average % of Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Precision at 5, 10, 20 & 50<br />
* Average % of Genre matches in the top 5, 10, 20 & 50 results after artist filtering of results<br />
* Average % of available Genre, Artist and Album matches in the top 5, 10, 20 & 50 results - Recall at 5, 10, 20 & 50 (just normalising scores when less than 20 matches for an artist, album or genre are available in the database)<br />
* Always similar - Maximum # times a file was in the top 5, 10, 20 & 50 results<br />
* % File never similar (never in a top 5, 10, 20 & 50 result list)<br />
* % of 'test-able' song triplets where triangular inequality holds<br />
** Note that as we are not requiring full distance matrices this year we will only be testing triangles that are found in the sparse distance matrix.<br />
* Plot of the "number of times similar curve" - plot of song number vs. number of times it appeared in a top 20 list with songs sorted according to number times it appeared in a top 20 list (to produce the curve). Systems with a sharp rise at the end of this plot have "hubs", while a long 'zero' tail shows many never similar results.<br />
<br />
=== Additional Data Reported ===<br />
<br />
In addition computation times for feature extraction/Index-building and querying <br />
will be measured.<br />
<br />
== Submission format ==<br />
Submission to this task will have to conform to a specified format detailed <br />
below. <br />
<br />
=== Audio formats ===<br />
Participating algorithms will have to read audio in the following format:<br />
<br />
* Sample rate: 22 KHz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV<br />
* clip length: 30 secs from the middle of each file<br />
<br />
=== Implementation details ===<br />
Scratch folders will be provided for all submissions for the storage of feature files and any model or index files to be produced. Executables will have to accept the path to their scratch folder as a command line parameter. Executables will also have to track which feature files correspond to which audio files internally. To facilitate this process, unique filenames will be assigned to each audio track.<br />
<br />
The audio files to be used in the task will be specified in a simple ASCII list file. This file will contain one path per line with no header line. Executables will have to accept the path to these list files as a command line parameter. The formats for the list files are specified below. <br />
<br />
Multi-processor compute nodes (2, 4 or 8 cores) will be used to run this task. Hence, participants could attempt to use parrallelism. Ideally, the number of threads to use should be specified as a command line parameter. Alternatively, implementations may be provided in hard-coded 2, 4 or 8 thread configurations. Single threaded submissions will, of course, be accepted but may be disadvantaged by time constraints.<br />
<br />
Submissions will have to output either a full distance matrix or a search results file with the top 100 search results for each track in the collection. This list of results will be used to extract the artist-filtered results to present to the human evaluators and will facilitate the computation of the objective statistics.<br />
<br />
=== I/O formats ===<br />
In this section the input and output files used in this task are described as<br />
are the command line calling format requirements for submissions.<br />
<br />
==== Audio collection list file (input)====<br />
The list file passed for feature extraction and indexing will be a simple ASCII list file. This file will contain one path per line with no header line, all paths will be absolute (full paths).<br />
<br />
e.g.<br />
<br />
/aDirectory/collectionFolder/b002342.wav<br />
/aDirectory/collectionFolder/a005921.wav<br />
...<br />
<br />
==== Search results output file ====<br />
<br />
It is encouraged for participants to generate full distance matrices in the MIREX 2006 format. Please see [2006:Audio_Music_Similarity_and_Retrieval#Output_File]] for details of this format.<br />
<br />
Alternatively, if computation is a concern, we will accept the sparse format as detailed below.<br />
<br />
SPARSE FORMAT:<br />
Participating algorithms should perform feature extraction, indexing and output a simple ASCII file listing a name for the algorithm and the top 100 search results for every track in the collection. <br />
<br />
This file should start with a header line with a name for the algorithm and should be followed by the results for one query per line, prefixed by the filename portion of the query path. This should be followed by a tab character and a tab separated, ordered list of the top 100 search results. Each result should include the result filename (e.g. a034728.wav) and the distance (e.g. 17.1 or 0.23) separated by a a comma.<br />
<br />
E.g.<br />
<br />
MyAlgorithm (my.email@address.com)<br />
<example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance><br />
<example 1 filename>\t<result 1 name>,<result 1 distance>,\t<result 2 name>,<result 2 distance>, ... \t<result 100 name>,<result 100 distance><br />
...<br />
<br />
<br />
which might look like:<br />
<br />
MyAlgorithm (my.email@address.com)<br />
a009342.wav b229311.wav,0.16 a023821.wav,0.19 a001329,0.24 ... etc.<br />
a009343.wav a661931.wav,0.12 a043322.wav,0.17 c002346,0.21 ... etc.<br />
a009347.wav a671239.wav,0.13 c112393.wav,0.20 b083293,0.25 ... etc.<br />
...<br />
<br />
The path to which this list file should be written must be accepted as a parameter on the command line.<br />
<br />
==== Example submission calling formats ====<br />
<br />
extractFeatures.sh /path/to/scratch/folder /path/to/collectionListFile.txt<br />
Query.sh /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt<br />
<br />
doAudioSim.sh -numThreads 8 /path/to/scratch/folder /path/to/collectionListFile.txt /path/to/outputResultsFile.txt<br />
<br />
<br />
=== Packaging submissions ===<br />
All submissions should be statically linked to all libraries (the presence of <br />
dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the <br />
information:<br />
<br />
* Command line calling format for all executables<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks,<br />
hard limits on the runtime of submissions will be specified. <br />
<br />
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times).<br />
<br />
== Submission opening date ==<br />
7th August 2007 - provisional<br />
<br />
== Submission closing date ==<br />
21st August 2007 - provisional<br />
<br />
== Past polls ==<br />
<br />
Polling is more or less closed. The audio format, as above is 22.05kHz mono wavs. The output format is encouraged to be the full distance matrix as per 2006, but the new sparse format will be accepted if need be.<br />
<br />
=== Audio format poll ===<br />
<br />
<poll><br />
Use the same 30 sec clips for analysis by participating algorithms as presented to human evaluators (necessary due to copyright restrictions)?<br />
Yes<br />
No, my algorithm needs longer clips<br />
No, I don't think this necessary<br />
No, for some other reason <br />
</poll><br />
<br />
<poll><br />
What is your preferred audio format? Remember that the less audio data we have to process the larger the dataset can be...<br />
22 khz mono WAV<br />
22 khz stereo WAV<br />
44 khz mono WAV<br />
44 khz stereo WAV<br />
22 khz mono MP3 128kb<br />
22 khz stereo MP3 128kb<br />
44 khz mono MP3 128kb<br />
44 khz stereo MP3 128kb<br />
</poll><br />
<br />
=== Return format poll ===<br />
<poll><br />
Do you prefer to return a full distance matrix (NxN) or a sparse matrix (Nx100)?<br />
Full distance matrix<br />
Sparse Matrix<br />
</poll><br />
<br />
Note: If there is no significant demand for a sparse format it will be dropped.<br />
<br />
* I just wanted to say I am mostly ambivalent about this, but lean ever so slightly to the sparse matrix, as it will result in a few less disc writes and I am going to be pushing that 72 hr cap, so any trimming of run time is useful. That said I do see the value of preserving a more exhaustive dataset so either way...<br />
[[User:Bfields|Bfields]] 08:24, 3 August 2007 (CDT)<br />
<br />
== Participants ==<br />
If you think there is a slight chance that you might want to participate please add your name and email address here. <br />
<br />
* Klaas Bosteels (''firstname.lastname''@gmail.com)<br />
* Thomas Lidy (''lastname''@ifs.tuwien.ac.at)<br />
* Elias Pampalk (''firstname.lastname''@gmail.com)<br />
* Tim Pohle (''firstname.lastname''@jku.at)<br />
* Kris West (kw at cmp dot uea dot ac dot uk)<br />
* Julien Ricard (''firstname.lastname''@gmail.com)<br />
* Abhinav Singh (abhinavs at iitg.ernet.in) and S.R.M.Prasanna (prasanna at iitg.ernet.in)<br />
* Ben Fields (map01bf at gold dot ac dot uk)<br />
* Christoph Bastuck (bsk at idmt.fhg.de)<br />
* Aliaksandr Paradzinets (aliaksandr.paradzinets {at} ec-lyon.fr)<br />
* Vitor Soares (''firstname.lastname''{at} clustermedialabs.com)<br />
* Kai Chen('lastnamefirstname dot dr @gmail.com)<br />
* Kurt Jacobson (''firstname.lastname''@elec.qmul.ac.uk)<br />
* Yi-Hsuan Yang (affige at gmail dot com)<br />
* Luke Barrington (lukeinusa at gmail)<br />
* Douglas Turnbull (dturnbul AT cs.ucsd.edu)<br />
* George Tzanetakis (gtzan at cs dot uvic dot ca)</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Audio_Music_Mood_Classification&diff=25082007:Audio Music Mood Classification2007-12-11T18:19:13Z<p>WikiSysop: fixed hyperlinks to mirex2006 wiki</p>
<hr />
<div>=FINAL 2007 AMC EVALUATION SCENARIO OVERVIEW=<br />
This section is put here to clarify what will happen for this year's "beta" run of the Audio Mood Classification (AMC) task.<br />
<br />
# We will operate the AMC task as a classic train-test classification task.<br />
# We will n-fold the runs with n to be determined by the size of the final data set, number of participants, etc.<br />
# We will hand-craft the n-fold test-train split lists.<br />
# We will NOT be doing post-run human mood judgments this year using the Evalutron 6000. <br />
# Audio files: 30 sec., 22kHz, mono, 16 bit<br />
<br />
Do take a look at the [[Audio Genre Classification]] task wiki as we are basing the underlying structure of this task on Audio Genre. In fact, an Audio Genre submission should work out of the box with Audio Mood Classification. Note: we really want folks to do a FEATURE EXTRACTION phase first against all the files and then have these features cached some place for re-use during the TRAIN-TEST phase. This way we can really speed up the n-fold processing. Thus, like GENRE, we need to pass three input files to your algos:<br />
<br />
==== 1. Feature extraction list file ====<br />
The list file passed for feature extraction will a simple ASCII list <br />
file. This file will contain one path per line with no header line.<br />
<br />
==== 2. Training list file ====<br />
The list file passed for model training will be a simple ASCII list <br />
file. This file will contain one path per line, followed by a tab character and <br />
the genre label, again with no header line. <br />
<br />
E.g. <example path and filename>\t<mood classification><br />
<br />
==== 3. Test (classification) list file ====<br />
The list file passed for testing classification will be a simple ASCII list <br />
file identical in format to the Feature extraction list file. This file will <br />
contain one path per line with no header line.<br />
<br />
==== Classification output files ====<br />
Participating algorithms should produce a simple ASCII list file identical in <br />
format to the Training list file. This file will contain one path per line, <br />
followed by a tab character and the MOOD label, again with no header line. <br />
E.g.:<br />
<example path and filename>\t<mood classification><br />
<br />
The path to which this list file should be written must be accepted as a <br />
parameter on the command line.<br />
<br />
********************************************<br />
<br />
== Audio collection poll ==<br />
<br />
<poll><br />
Would you like to use 30 secs clips from tracks for analysis to avoid mood change within tracks and reduce processing load ?<br />
Yes<br />
No, I like 60 secs clips<br />
No, I like the whole track <br />
</poll><br />
<br />
<poll><br />
How important do you think cross-validation is?<br />
Very important<br />
Important<br />
Not important<br />
</poll><br />
<br />
<poll><br />
Would you like your algorithm(s) to be evaluated on a closed groundtruth set (as in traditional classification problems, both training and testing data are labeled well before the contest) or on an unlabeled audio pool (in the way described in this wiki page, please see section 7,8,9) ?<br />
On a closed groundtruth set (the size of the set is smaller, but evaluation metrics are more rigorous and support cross-validation)<br />
On an unlabeled audio pool (the size of the pool can be very big, but only a small portion will be judged by human.)<br />
Both <br />
</poll><br />
<br />
<poll><br />
If you like a closed groundtruth set, what is the MINIMUM size of the set you can accept (including training and testing)?<br />
400 clips in total (~80 clips in each category)<br />
600 clips in total (~120 clips in each category)<br />
800 clips in total (~160 clips in each category)<br />
1000 clips in total (~200 clips in each category)<br />
more than 1000 clips<br />
</poll><br />
<br />
<poll><br />
If you like an unlabeled audio pool, what is the MINIMUM size of training audio you can accept?<br />
30 clips in each category<br />
50 clips in each category<br />
80 clips in each category<br />
100 clips in each category<br />
more than 100 clips in each category<br />
</poll><br />
<br />
<br />
<poll><br />
What is your preferred audio format? (the less audio data to process the larger the dataset can be) <br />
22 khz mono WAV<br />
22 khz stereo WAV<br />
44 khz mono WAV<br />
44 khz stereo WAV<br />
22 khz mono MP3 128kb<br />
22 khz stereo MP3 128kb<br />
44 khz mono MP3 128kb<br />
44 khz stereo MP3 128kb<br />
</poll><br />
<br />
<poll><br />
How many algorithms will you likely to submit? (for estimating the number of human assessors needed)<br />
0<br />
1<br />
2<br />
3<br />
</poll><br />
<br />
== Introduction ==<br />
In music psychology and music education, emotion component of music has been recognized as the most strongly associated with music expressivity.(e.g. Juslin et al 2006[[#Related Papers]]). Music information behavior studies (e.g.Cunningham, Jones and Jones 2004, Cunningham, Vignoli 2004, Bainbridge and Falconer 2006 [[#Related Papers]]) have also identified music mood/ emotion as an important criterion used by people in music seeking and organization. Several experiments have been conducted in the MIR community to classify music by mood (e.g. Lu, Liu and Zhang 2006, Pohle, Pampalk, and Widmer 2005, Mandel, Poliner and Ellis 2006, Feng, Zhuang and Pan 2003[[#Related Papers]]). Please note: the MIR community tends to use the word "mood" while musicpsychologists like to use "emotion". We follow the MIR tradition to use "mood" thereafter. <br />
<br />
However, evaluation of music mood classification is difficult as music mood is a very subjective notion. Each aforementioned experiement used different mood categories and different datasets, making comparison on previous work a virtually impossible mission. A contest on music mood classification in MIREX will help build the first ever community available test set and precious ground truth.<br />
<br />
This is the first time in MIREX to attempt a music mood classification evaluation. There are many issues involved in this evaluation task, and let us start discuss them on this wiki. If needed, we will set up a mailing list devoting to the discussion.<br />
<br />
== Mood Categories ==<br />
<br />
The IMIRSEL has derived a set of 5 mood clusters from the AMG mood repository (Hu & Downie 2007[[#Related Papers]]). The mood clusters effectively reduce the diverse mood space into a tangible set of categories, and yet root in the social-cultural context of pop music. Therefore, we propose to use the 5 mood clusters as the categories in this yearΓÇÖs audio mood classification contest. Each of the clusters is a collection of the AMG mood labels which collectively define the cluster: <br />
<br />
*Cluster_1: passionate, rousing, confident,boisterous, rowdy <br />
*Cluster_2: rollicking, cheerful, fun, sweet, amiable/good natured <br />
*Cluster_3: literate, poignant, wistful, bittersweet, autumnal, brooding <br />
*Cluster_4: humorous, silly, campy, quirky, whimsical, witty, wry <br />
*Cluster_5: aggressive, fiery,tense/anxious, intense, volatile,visceral <br />
<br />
At this moment, the IMIRSEL and Cyril Laurier at the Music Technology Group of Barcelona have manually validated the mood clusters and exemplar songs in each cluster. Please see [[#Exemplar Songs in Each Category]] for details. <br />
<br />
We are still seeking additional songs across different genres to enrich this set, and during the process, the cluster with least cross-listener consistency may be dropped, or two clusters often confusing each other may be combined. <br />
<br />
<br />
[[Previous Discussion on Mood Taxonomy]]<br />
<br />
[[Discussion on Mood Categories]]<br />
<br />
== Exemplar Songs in Each Category == <br />
Exemplar songs for each mood cluster are manually selected by multiple human assessors. The purpose is to further clarify the perceptual identities of the mood clusters.<br />
<br />
There are 190 candidate songs in the intersection of AMG mood repository and the USPOP collection in IMIRSEL, and each of these songs has only one unanimous mood cluster label assigned by AMG editors. The mood labels by AMG editors are important benchmark which can help us reach cross-listener consistency on such a subjective task. So far, 6 human assessors have listened to the 190 songs and assigned cluster labels to them. 50 songs are unanimously labeled by the 6 human assessors, 42 songs are unanimously labeled by 5 of the 6 human assessors, and another 40 songs by 4 of the 6 human assessors. The song titles are listed in [[exemplar songs]]. <br />
<br />
The advantages of the exemplar songs are two folds: 1. they will help people better understand what kind of mood each cluster refers to; 2. they can possibly be taken as training data for the algorithms (see the section of [[#Training Set]]). <br />
<br />
Note: Lyrics issue: when labeling the songs, the human assessors were asked to ignore lyrics. As this is a contest focuses on music audio, lyrics should not be taken into consideration. <br />
<br />
[[Previous Discussion on Ground Truth]]<br />
<br />
== Two Evaluation Scenarios ==<br />
<br />
1. Evaluation on a closed groundtruth set.<br />
As in traditional classification problems, both training and testing data are labeled well before the contest. <br />
Pros: evaluation metrics are more rigorous; support cross-validation <br />
cons: training/testing set is limited<br />
<br />
2. Training on a labeled set, but testing on an unlabeled audio pool <br />
As in audio similarity and retrieval contest, each algorithm returns a list of candidates in each mood category, then human assessors make judgments on the returned candidates. <br />
Pros: testing pool can be arbitrarily big; training set is bigger as well (which can be the whole groundtruth set in scenario 1 .) <br />
Cons: innovative but limited evaluation metrics (see below)<br />
<br />
For both scenarios, this is a single-label classification contest, and thus each song can only be classified into one mood cluster. <br />
<br />
'''We will go for scenario 1'''<br />
<br />
== Groundtruth Set ==<br />
<br />
The IMIRSEL is preparing a ground-truth set of audio clips selected from the USPOP collection decribed above and the APM collection (www.apmmusic.com). The bibliographic information of the exemplar songs has been released as above, which is to help participants reach agreements on the meanings of the mood categories.<br />
<br />
The APM audio set has been pre-labeled with the 5 mood clusters according to their metadata provided by APM, and covers a variety of genres: each category covers about 7 major genres (with 20-30 tracks each) and a few minor genres. To make the problem more interesting, the distribution among major genres within each category is made as even as possible. <br />
<br />
To make sure the mood labels are correct, this APM audio collection will subject to human validation before the contest. We prepared a set of 1250 audio clips (250 per category). The audio clips whose mood category assignments reach agreements among 2 out of 3 human assessors will serve as a ground truth set. We are aiming at least 120 audio clips in each mood category. <br />
<br />
After the human validation on this audio set, participating algorithms/ models will be trained and tested within IMIRSEL.<br />
<br />
'''Audio format: 30 second clips, 22.05kHz, mono, 16bit, WAV files''' <br />
<br />
=== Human Validation ===<br />
Subjective judgments by human assessors will be collected for the above mentioned APM audio set using a web-based system, Evalutron6000, developed by the IMIRSEL. (An introduction of this piece of Evalutron 6000 is shown here [[Evalutron6000_Walkthrough_For_Audio_Mood_Classification]]<br />
<br />
Each audio clip is 30 seconds long, and will have 3 human judges listen to it and choose which mood category it belongs to. If 2 of the 3 judges agree on its category, an audio clip will be selected into the groundtruth set.<br />
<br />
== Evaluation Metrics == <br />
<br />
Metrics frequently used in classification problems include: accuracy, precision, recall and F measures (combining precision and recall). The single most important metrics would be accuracy, which allows direct system comparison: <br />
<br />
''Accuracy = # of correctly classified songs / #. of all songs.'' <br />
<br />
Accuracy can be calculated for all clusters as a whole (macro average) or for each cluster then take average of them (micro average).<br />
<br />
Test significance of differences among systems, possibly using<br />
<br />
*a) McNemarΓÇÖs test <br />
<br />
McNemarΓÇÖs test (Dietterich, 1997) is a statistical process that can validate the significance of differences between two classifiers. It was used in Audio Genre Classification and Audio Artist Identification contests in MIREX 2005. <br />
<br />
*b) FriedmanΓÇÖs test<br />
<br />
FriedmanΓÇÖs test used to detect differences in treatments across multiple test attempts. (http://en.wikipedia.org/wiki/Friedman_test). It was used in Audio Similarity, Audio cover song, and Query by Singing/Humming contests in MIREX 2006. <br />
<br />
Besides, run time can be recorded and compared.<br />
<br />
== Important Dates ==<br />
<br />
* Human Validation for Groundtruth Set: August 1 - August 15<br />
* Algorithm Submission Deadline: August 25<br />
<br />
== Packaging your Submission ==<br />
* Be sure that your submission follows the [[#Submission_Format]] outlined below.<br />
* Be sure that your submission accepts the proper [[#Input_File]] format<br />
* Be sure that your submission produces the proper [[#Output_File]] format<br />
* Be sure to follow the [[https://www.music-ir.org/mirex/2006/index.php/Best_Coding_Practices_for_MIREX Best Coding Practices for MIREX]]<br />
* Be sure to follow the [[MIREX 2007 Submission Instructions]] <br />
* In the README file that is included with your submission, please answer the following additional questions:<br />
** Approximately how long will the submission take to process ~1000 wav files?<br />
** Approximately how much scratch disk space will the submission need to store any feature/cache files?<br />
** Any special notice regarding to running your algorith<br />
* Submit your system via the URL located at the bottom of [[MIREX 2007 Submission Instructions]] page<br />
<br />
Note that the information that you place in the README file is '''extremely''' important in ensuring that your submission is evaluated properly.<br />
<br />
== Submission Format ==<br />
A submission to the Audio Music Mood Classification evaluation is expected to follow the [https://www.music-ir.org/mirex/2006/index.php/Best_Coding_Practices_for_MIREX Best Coding Practices for MIREX] and must conform to the following for execution:<br />
<br />
=== One Call Format ===<br />
The one call format is appropriate for systems that perform all phases of the classification (typically features extraction, training and testing) in one step. A submission should be an executable program that takes 4 arguments: <br />
* path/to/fileContainingListOfTrainingAudioClips - the path to the list of training audio clips (see [[#File Formats]] below)<br />
* path/to/fileContainingListOfTestingAudioClips - the path to the list of testing audio clips (see [[#File Formats]] below)<br />
* path/to/cacheDir - a directory where the submission can place temporary or scratch files. Note that the contents of this directory can be retained across runs, so if, for whatever reason, the submission needs to be restarted, the submission could make use of the contents of this directory to eliminate the need for reprocessing some inputs.<br />
* path/to/output/Results - the file where the output classification results should be placed. (see [[#File Formats]] below)<br />
<br />
'''Example:'''<br />
<br />
<pre><br />
<br />
doAMC "path/to/fileContainingListOfTrainingAudioClips" "path/to/fileContainingListOfTestingAudioClips" "path/to/cacheDir" "path/to/output/Results" <br />
<br />
</pre><br />
<br />
<br />
=== Two Call Format ===<br />
The one call format is appropriate for systems that perform the training and testing separately. A submission should consists of two executable programs<br />
*trainAMC - this takes 3 arguments: <br />
** path/to/fileContainingListOfTrainingAudioClips - the path to the list of training audio clips (see [[#File Formats]] below)<br />
** path/to/trainingCacheDir - a directory where the submission can place temporary or scratch files. Note that the contents of this directory can be retained across runs, so if, for whatever reason, the submission needs to be restarted, the submission could make use of the contents of this directory to eliminate the need for reprocessing some inputs.<br />
** path/to/trainedClassificationModel - the file where the classification model should be placed<br />
*testAMC - this takes 4 arguments:<br />
** path/to/trainedClassificationModel<br />
** path/to/fileContainingListofTestingAudioClips - the path to the list of testing audio clips (see [[#File Formats]] below)<br />
** path/to/testingCacheDir - a directory where the submission can place temporary or scratch files. <br />
** path/to/output/Results - the file where the output classification results should be placed. (see [[#File Formats]] below)<br />
<br />
'''Example:'''<br />
<br />
<pre><br />
<br />
trainAMC "path/to/fileContainingListOfTrainingAudioClips" "path/to/trainingcacheDir" "path/to/trainedClassificationModel" <br />
testAMC "path/to/trainedClassificationModel" "path/to/fileContainingListofTestingAudioClips" "path/to/testingCacheDir" "path/to/output/Results"<br />
<br />
</pre><br />
<br />
=== Matlab format ===<br />
<br />
Matlab will also be supported in the form of functions in the following formats:<br />
<br />
==== Matlab One call format ====<br />
<pre><br />
doMyMatlabAMC('path/to/fileContainingListOfTrainingAudioClips','path/to/fileContainingListOfTestingAudioClips','path/to/cacheDir','path/to/output/Results')<br />
</pre><br />
<br />
<br />
==== Matlab Two call format ====<br />
<pre><br />
doMyMatlabTrainAMC('path/to/fileContainingListOfTrainingAudioClips','path/to/trainingcacheDir','path/to/trainedClassificationModel')<br />
doMyMatlabTestAMC('path/to/trainedClassificationModel','path/to/fileContainingListofTestingAudioClips','path/to/testingCacheDir','path/to/output/Results')<br />
</pre><br />
<br />
== File Formats ==<br />
<br />
=== Input Files ===<br />
<br />
The input training list file format will be of the form: <br />
<br />
<pre><br />
path/to/training/audio/file/000001.wav\tCluster_3<br />
path/to/training/audio/file/000002.wav\tCluster_5<br />
path/to/training/audio/file/000003.wav\tCluster_2<br />
...<br />
path/to/training/audio/file/00000N.wav\tCluster_1<br />
</pre><br />
<br />
"\t" stands for tab.<br />
<br />
The input testing list file format will be of the form: <br />
<br />
<pre><br />
path/to/testing/audio/file/000010.wav<br />
path/to/testing/audio/file/000020.wav<br />
path/to/testing/audio/file/000030.wav<br />
...<br />
path/to/testing/audio/file/0000N0.wav<br />
</pre><br />
<br />
"\t" stands for tab.<br />
<br />
=== Output File ===<br />
The only output will be a file containing classification results in the following format: <br />
<br />
<pre><br />
Example Classification Results 0.1 (replace this line with your system name)<br />
path/to/testing/audio/file/000010.wav\tCluster_3<br />
path/to/testing/audio/file/000020.wav\tCluster_1<br />
path/to/testing/audio/file/000030.wav\tCluster_5<br />
...<br />
path/to/testing/audio/file/0000N0.wav\tCluster_2<br />
</pre><br />
<br />
"\t" indicates tab. All audio clips should have one and only one mood cluster label.<br />
<br />
==Evaluation Scenario 2==<br />
<br />
=== Training Set ===<br />
<br />
Under evaluation scenario 2, the training set would be the whole ground truth set in scenario 1 (see [[#Groundtruth Set]]).<br />
<br />
=== Unlabeled Song Pool ===<br />
Under evaluation scenario 2, the pool of testing audio to be classified is from the same collection of the training set, i.e. USPOP and APM. We will make sure the audio covers a variety of genres in each mood cluster, which will make the contest harder and more interesting.<br />
<br />
We will randomly select a certain number (say, 1000) of songs from the collections as the audio pool. This number should make the contest interesting enough, but not too hard. And the songs need to cover all 5 mood clusters.<br />
<br />
=== Classification Results ===<br />
Each algorithm will return the top X songs in each cluster. <br />
<br />
This is a single-label classification contest, and thus each song can only be classified into one mood cluster. <br />
<br />
Note: unlike traditional classification problems where all testing samples have ground truth available, this scenario does not have a well labeled testing set. Instead, we use a ΓÇ£poolingΓÇ¥ approach like in TREC and last yearΓÇÖs audio similarity and retrieval contest. This approach collects the top X results from each algorithm and asks human assessors to make judgments on this set of collected results while assuming all other samples are irrelevant or incorrect. This approach cannot measure the absolute ΓÇ£recallΓÇ¥ metrics, but it is valid in comparing relative performances among participating algorithms. <br />
<br />
The actual value of X depends on human assessment protocol and number of available human assessors (see next section [[#Human Assessment]]).<br />
<br />
=== Human Assessment===<br />
Subjective judgments by human assessors will be collected for the pooled results using a web-based system, Evalutron6000, developed by the IMIRSEL. (An introduction of this piece of Evalutron 6000 is shown here [[Evalutron6000_Walkthrough_For_Audio_Mood_Classification]]<br />
<br />
==== How many judgments and assessors ====<br />
Each algorithm returns X songs for each of the 5 mood clusters. Suppose there are Y algorithms, in the worst case, each cluster will have 5* X*Y songs to be judged. Suppose each song needs Z sets of ears, there will be 5*X*Y*Z judgments in total. When making a judgment, a human assessor will listen to the 30 second clip of a song, and label it with one of the 5 mood clusters. <br />
<br />
Human evaluators will be drawn from the participating labs and volunteers from IMIRSEL or on the MIREX lists. Suppose we can get W evaluators, each evaluator will evaluate S = (5*X*Y*Z) / W songs.<br />
<br />
At this moment, there are 10 potential participants on the Wiki, so letΓÇÖs say Y = 6. Suppose each candidate song will be evaluated by 3 judges, Z = 3, and suppose we can get 20 assessors: W = 20: <br />
<br />
*If X = 20, number of judgments for each assessor: S = 90<br />
*If X = 10, S = 45<br />
*If X = 30, S = 135 <br />
*If X = 50, S = 225<br />
*If X = 15, S = 67.5<br />
*…<br />
<br />
In audio similarity contest last year, each assessor made 205 judgments as average. As the judgment for mood is trickier, we may need to give our assessors less burden.<br />
<br />
To eliminate possible bias, we will try to equally distribute candidates returned by each algorithm among human assessors.<br />
<br />
=== Scoring ===<br />
Each algorithm is graded by the number of votes its candidate songs win from the judges. For example, if a song, A, is judged as in Cluster_1 by 2 assessors and as in Cluster_2 by 1 assessors, then the algorithm classifying A as in Cluster_1 will score 2 on this song, while the algorithm classifiying A as Cluster_2 will score 1 on this song. An algorithmΓÇÖs final score is the sum of scores on all the songs it submits. Since each algorithm can only submit 100 songs, the one which wins the most votes of judges win the contest.<br />
<br />
=== Evaluation Metrics ===<br />
Algorithm score as mentioned in last section is a metrics that facilitates direct comparison. <br />
<br />
Besides, metrics frequently used in classification problems include: accuracy, precision, recall and F measures (combining precision and recall). As mentioned above, the pooling approach results in a relative recall measure, therefore, the single most important metrics would be accuracy: <br />
<br />
The original definition of accuracy is:<br />
''Accuracy = # of correctly classified songs / #. of all songs.'' <br />
<br />
According to the above human assessment method, ΓÇ£correctly classified songsΓÇ¥ in this scenario can be defined as songs classified as the majority vote of the judges and, in the case of ties, songs classified as any of the tie votes. For example, suppose each song has 3 judges. If a song is labeled as Cluster_1 by at least 2 judges, then this song will be counted as correct for algorithms classifying it to Cluster_1; if a song is labeled as Cluster_1, Cluster_2 and Cluster_3 once by each of the judges, then this song will be counted as correct for algorithms classifying it to Cluster_1, Cluster_2 or Cluster_3. <br />
<br />
Accuracy can be calculated for all clusters as a whole (macro average) or for each cluster then take average of them (micro average).<br />
<br />
Test significance of differences among systems, possibly using<br />
<br />
*a) McNemarΓÇÖs test <br />
*b) FriedmanΓÇÖs test<br />
<br />
Besides, run time can be recorded and compared.<br />
<br />
== Challenging Issues == <br />
# Mood changeable pieces: some pieces may start from one mood but end up with another one. <br />
<br />
We will use 30 second clips instead of whole songs. The clips will be extracted automatically from the middle of the songs which have more chances to be representative.<br />
<br />
# Multiple label classification: it is possible that one piece can have two or more correct mood labels, but as a start, we strongly suggest to hold a less confusing contest and leave the challenge to future MIREXs.So, for this year, this is a single label classification problem.<br />
<br />
== Participants ==<br />
If you think there is a slight chance that you might consider participating, please add your name and email address here. <br />
<br />
* Kris West (kw at cmp dot uea dot ac dot uk)<br />
* Cyril Laurier (claurier at iua dot upf dot edu)<br />
* Elias Pampalk (<i>firstname.lastname</i>@gmail.com)<br />
* Yuriy Molchanyuk (molchanyuk at onu.edu.ua)<br />
* Shigeki Sagayama (sagayama at hil dot t.u-tokyo.ac.jp)<br />
* Guillaume Nargeot (killy971 at gmail dot com)<br />
* Zhongzhe Xiao (zhongzhe dot xiao at ec-lyon dot fr)<br />
* Kyogu Lee (kglee at ccrma.stanford.edu)<br />
* Vitor Soares (<i>firstname.lastname</i>@clustermedialabs.com)<br />
* Wai Cheung (wlche1@infotech.monash.edu.au)<br />
* Matt Hoffman (mdhoffma <i>a t</i> cs <i>d o t</i> princeton <i>d o t</i> edu)<br />
* Yi-Hsuan Yang (affige at gmail dot com)<br />
* Jose Fornari ( fornari at campus dot jyu dot fi )<br />
<br />
== Moderators ==<br />
* J. Stephen Downie (IMIRSEL, University of Illinois, USA) - [mailto:jdownie@uiuc.edu]<br />
* Xiao Hu (IMIRSEL, University of Illinois, USA) -[mailto:xiaohu@uiuc.edu]<br />
* Cyril Laurier (Music Technology Group, Barcelona, Spain) -[mailto:claurier@iua.upf.edu]<br />
<br />
== Related Papers ==<br />
#Dietterich, T. (1997). '''Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms'''. Neural Computation, 10(7), 1895-1924.<br />
#Hu, Xiao and J. Stephen Downie (2007). '''Exploring mood metadata: Relationships with genre, artist and usage metadata'''. Accepted in the Eighth International Conference on Music Information Retrieval (ISMIR 2007),Vienna, September 23-27, 2007.<br />
# Juslin, P.N., Karlsson, J., Lindstr├╢m E., Friberg, A. and Schoonderwaldt, E(2006), '''Play It Again With Feeling: Computer Feedback in Musical Communication of Emotions'''. In Journal of Experimental Psychology: Applied 2006, Vol.12, No.2, 79-95.<br />
# [http://ismir2004.ismir.net/proceedings/p075-page-415-paper152.pdf Vignoli (ISMIR 2004)] '''Digital Music Interaction Concepts: A User Study'''<br />
# [http://ismir2004.ismir.net/proceedings/p082-page-447-paper221.pdf Cunningham, Jones and Jones (ISMIR 2004)] '''Organizing Digital Music For Use: An Examiniation of Personal Music Collections'''.<br />
# [http://ismir2006.ismir.net/PAPERS/ISMIR0685_Paper.pdf Cunningham, Bainbridge and Falconer (ISMIR 2006)] '''More of an Art than a Science': Supporting the Creation of Playlists and Mixes'''.<br />
# Lu, Liu and Zhang (2006), '''Automatic Mood Detection and Tracking of Music Audio Signals'''. IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 14, NO. 1, JANUARY 2006 <br> Part of this paper appeared in ISMIR 2003 http://ismir2003.ismir.net/papers/Liu.PDF<br />
# [http://www.cp.jku.at/research/papers/Pohle_CBMI_2005.pdf Pohle, Pampalk, and Widmer (CBMI 2005)] '''Evaluation of Frequently Used Audio Features for Classification of Music into Perceptual Categories'''. <br> It separates "mood" and "emotion" as two classifcation dimensions, which are mostly combined in other studies.<br />
# [http://www.ee.columbia.edu/~dpwe/pubs/MandPE06-svm.pdf Mandel, Poliner and Ellis (2006)] '''Support vector machine active learning for music retrieval'''. Multimedia Systems, Vol.12(1). Aug.2006.<br />
# [http://doi.acm.org/10.1145/860435.860508 Feng, Zhuang and Pan (SIGIR 2003)] '''Popular music retrieval by detecting mood'''<br />
# [http://ismir2003.ismir.net/papers/Li.PDF Li and Ogihara (ISMIR 2003)] '''Detecting emotion in music'''<br />
# [http://pubdb.medien.ifi.lmu.de/cgi-bin//info.pl?hilliges2006audio Hilliges, Holzer, Kl├╝ber and Butz (2006)] '''AudioRadar: A metaphorical visualization for the navigation of large music collections'''.In Proceedings of the International Symposium on Smart Graphics 2006, Vancouver Canada. <br> It summarized implicit problems in traditional genre/artist based music organization.<br />
# Juslin, P. N., & Laukka, P. (2004). '''Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening'''. Journal of New Music Research, 33(3), 217-238.<br />
# [http://mpac.ee.ntu.edu.tw/~yihsuan/ Yang, Liu, and Chen (ACMMM 2006)] '''Music emotion classification: A fuzzy approach '''</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Main_Page&diff=23932007:Main Page2007-12-11T18:18:22Z<p>WikiSysop: </p>
<hr />
<div>=MIREX 2007 Results Posted=<br />
<br />
Results from the 2007 MIREX evaluation runs are available now at: [[MIREX2007_Results]]!<br />
<br />
= MIREX 2007 Evaluation Tasks=<br />
* [[Audio Artist Identification]] <br />
* [[Audio Classical Composer Identification]] - [[Audio Artist Identification]] subtask<br />
* [[Audio Genre Classification]] <br />
* [[Audio Music Mood Classification]] <br />
* [[Audio Music Similarity and Retrieval]]<br />
* [[Audio Onset Detection]]<br />
* [[Audio Cover Song Identification]]<br />
* [[Real-time Audio to Score Alignment (a.k.a Score Following)]] (Postponed to possibly 2008)<br />
* [[Query by Singing/Humming]]<br />
* [[Multiple Fundamental Frequency Estimation & Tracking]]<br />
* [[Symbolic Melodic Similarity]]<br />
<br />
==2007 Submission System Open: SUBMIT ASAP!!!!!==<br />
The MIREX 2007 submission system is now open. <br />
<br />
WE ARE ON A VERY, VERY TIGHT TIMELINE THIS YEAR. PLEASE GET <br />
YOUR SUBMISSIONS IN ASAP! SEE THE [[MIREX 2007 TIMELINE]] PAGE FOR SPECIFIC <br />
TIMELINES. AUDIO SIMILARITY and SYMBOLIC SIMILARITY NEED TO HAVE ALL <br />
RUNS COMPLETED BY SEPTEMBER 3rd SO WE CAN RUN EVALUTRON. WE WANT TO HAVE <br />
ALL RESULTS BACK TO THE COMMUNITY BY SEPTEMBER 16th!!!<br />
<br />
See the [[MIREX 2007 Submissions]] page for submission instructions.<br />
<br />
==Welcome to MIREX 2007==<br />
This is the main page for the Third Music Information Retrieval Evaluation eXchange (MIREX 2007). The International Music Information Retrieval Systems Evaluation Laboratory ([https://music-ir.org/evaluation IMIRSEL]) at the Graduate School of Library and Information Science ([http://www.lis.uiuc.edu GSLIS]), University of Illinois at Urbana-Champaign ([http://www.uiuc.edu UIUC]) is the principal organizer of MIREX 2007. <br />
<br />
The MIREX 2007 community will hold its annual plenary meeting as part of [http://ismir2007.ismir.net/ The 8th International Conference on Music Information Retrieval], ISMIR 2007, which will be held at the Vienna University of Technology, from September 23rd to September 27th, 2007. <br />
<br />
J. Stephen Downie<br><br />
Director, IMIRSEL<br><br />
<br />
==Getting Involved in MIREX 2007==<br />
MIREX is a community-based endeavour. Be a part of the community and help make MIREX 2007 the best yet.<br />
<br />
===Mailing List Participation===<br />
If you are interested in formal MIR evaluation, you should also subscribe to the "MIREX" (aka "EvalFest") mail list and participate in the community discussions about defining and running MIREX 2007 tasks. Subscription information at: <br />
https://mail.isrl.uiuc.edu/mailman/listinfo/evalfest <br />
===Wiki Participation===<br />
'''''Please note that you must create a NEW login for this wiki even if you have a login that you previously used for editing the MIREX 2005 or 2006 wikis.'''''<br />
<br />
Please create an account via: [https://www.music-ir.org/mirex/2007/index.php?title=Special:Userlogin&type=signup&returnto=Special:Userlogin Create Account].<br />
<br />
==MIREX 2005 and 2006 Wikis==<br />
This is the new wiki for MIREX 2007. The wikis for MIREX 2005 and 2006 are available at:<br />
<br />
'''[[2005:Main_Page|MIREX 2005]]''' <br />
https://www.music-ir.org/mirex/2005/<br />
<br />
'''[[2006:Main_Page|MIREX 2006]]''' <br />
https://www.music-ir.org/mirex/2006/<br />
<br />
You can interlink between this wiki and the previous wikis using '''2005:''' prefix on links to connect to pages in MIREX 2005 and '''2006:''' for MIREX 2006.<br />
<br />
<br />
== MIREX 2007 Task Proposal Pages THAT WILL NOT BE RUN THIS YEAR ==<br />
<br />
<br />
Use this space to link to the individual evaluation task discussion Wiki pages. Take a look at the MIREX 2006 Evaluation Tasks section to see how it was done last year.<br />
<br />
* [[Audio Artist Similarity]] <br />
* [[Audio Chord Detection]]<br />
* [[Audio Melody Extraction]]<br />
* [[Single Fundamental Frequency Estimation & Tracking from Monophonic Audio]]<br />
* [[Melody Transcription from Monophonic Audio]]<br />
* [[Audio Drum Detection]]<br />
<br />
==Something New To Consider: MIREX DIY Demonstration==<br />
Please take a look at: http://cluster3.lis.uiuc.edu:8080/mirexdiydemo/. This is a prototype "proof-of-concept" demonstration of the Music Information Retrieval Evaluation eXchange (MIREX) "Do-It-Yourself" (DIY) web service system. <br />
<br />
A recent article in D-Lib Magazine is a good introduction to what we are trying to accomplish:<br />
<br />
Downie, J. Stephen. 2006. The Music Information Retrieval Evaluation eXchange (MIREX). In D-Lib Magazine 12 (Issue 12). Available: http://dlib.org/dlib/december06/downie/12downie.html. <br />
<br />
Would like to try to have a least one task (or some task beta-testers) use a similar set up for MIREX 2007. A [[MIREX DIY Discussion]] page has been started for community input.<br />
<br />
==Descriptions of the Past Evaluation Tasks==<br />
===[[2006:Main_Page|MIREX 2006]] Evaluation Tasks===<br />
* [[2006:Audio Beat Tracking]] <br />
* [[2006:Audio Melody Extraction]] <br />
* [[2006:Audio Music Similarity and Retrieval]] <br />
* [[2006:Audio Cover Song]]<br />
* [[2006:Audio Onset Detection]]<br />
* [[2006:Audio Tempo Extraction]]<br />
* [[2006:QBSH (Query-by-Singing/Humming)]]<br />
* [[2006:Score Following]]<br />
* [[2006:Symbolic Melodic Similarity]]<br />
<br />
===[[2005:Main_Page|MIREX 2005]] Evaluation Tasks===<br />
* [[2005:Audio Artist Identification]]<br />
* [[2005:Audio Drum Detection]]<br />
* [[2005:Audio Genre Classification]]<br />
* [[2005:Audio Melody Extraction]]<br />
* [[2005:Audio Onset Detection]]<br />
* [[2005:Audio Tempo Extraction]]<br />
* [[2005:Audio and Symbolic Key Finding]]<br />
* [[2005:Symbolic Genre Classification]]<br />
* [[2005:Symbolic Melodic Similarity]]<br />
<br />
===[http://ismir2004.ismir.net/ISMIR_Contest.html ISMIR 2004 Audio Description Contest] Tasks===<br />
* [http://ismir2004.ismir.net/genre_contest/index.htm 2004:Genre Classification/Artist Identification]<br />
* [http://ismir2004.ismir.net/melody_contest/results.html 2004:Melody Extraction]<br />
* [http://www.iua.upf.es/mtg/ismir2004/contest/tempoContest 2004:Tempo Induction]<br />
* [http://www.iua.upf.es/mtg/ismir2004/contest/rhythmContest/ 2004:Rhythm Classification]</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Sandbox&diff=23992007:Sandbox2007-04-23T09:25:59Z<p>WikiSysop: </p>
<hr />
<div>testing<br />
<br />
external link: http://www.lis.uiuc.edu/<br />
<br />
<poll><br />
What is your question?<br />
Answer 1<br />
Answer 2<br />
...<br />
Answer n<br />
</poll></div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Sandbox&diff=23982007:Sandbox2007-04-23T09:24:33Z<p>WikiSysop: </p>
<hr />
<div>testing<br />
<br />
external link: http://www.lis.uiuc.edu/<br />
<br />
<poll><br />
What is your favorite color?<br />
Red<br />
Green<br />
Blue<br />
Yellow<br />
Purple<br />
</poll></div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Sandbox&diff=23952007:Sandbox2007-04-23T07:50:51Z<p>WikiSysop: </p>
<hr />
<div>testing<br />
<br />
external link: http://www.lis.uiuc.edu/<br />
<br />
<br />
<poll> <br />
What is your favorite color?<br />
Red<br />
Green<br />
Blue<br />
Yellow<br />
</poll></div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:2007_Euro_MIREX_Planning_Meeting&diff=28372007:2007 Euro MIREX Planning Meeting2007-04-12T17:13:49Z<p>WikiSysop: /* Participant List (12 April 2007) */</p>
<hr />
<div>==Original Call for Participation (slightly modified)==<br />
'''What:''' A two-day brainstorming and planning meeting dealing with MIREX <br />
2007 tasks, evaluation data, procedures and technology.<br />
<br />
'''Where:''' Vienna University of Technology (host of ISMIR/MIREX 2007)<br />
<br />
'''When:''' 22-23 April, 2007<br />
<br />
'''Who:''' All interested in defining, running and/or participating in MIREX <br />
2007 tasks. Having at least one representative from each of the Euro <br />
research labs interested in MIR, would be ideal. Of course, all are <br />
welcomed regardless of affiliation or country of origin (i.e., if you <br />
are not part of a Euro lab but in Europe, consider joining us). Keen and eager graduate students are especially encouraged to attend.<br />
<br />
'''Attending IMIRSEL Leaders:''' Dr. J. Stephen Downie, Andreas F. Ehmann, M. <br />
Cameron Jones, and Kris West (U. of East Anglia).<br />
<br />
'''Why:''' Planning for each previous MIREX has tended to be compressed into <br />
the final summer weeks just prior to meeting at ISMIR. This year, we are <br />
trying to get things on a stronger foundation earlier in the year to <br />
create a less stressful and more robust MIREX experience for all. Holding the meeting in Vienna will also allow MIREX organizers to co-ordinate with the ISMIR 2007 Vienna local organizers.<br />
<br />
===Basic Goals:===<br />
# Review of previous MIREX tasks (successes, problems, failures)<br />
# Review of proposed MIREX 2007 tasks, data sources, evaluations<br />
# Organization issues (leaders, communications, deadlines, etc.)<br />
# M2K/MIREX Do-it-Yourself Evaluation Framework Prototype<br />
# Awards/Recognitions for MIREX 2007(?)<br />
# New Business and Ideas<br />
<br />
'''Attending IMIRSEL Leaders:''' Dr. J. Stephen Downie, Andreas F. Ehmann, M. <br />
Cameron Jones, and Kris West (U. of East Anglia).<br />
<br />
'''Local Host Leaders:''' Thomas Lidy (Technical University of Vienna (TUV), Dr. Andreas Rauber (TUV), Dr. Rainer Typke (Austrian Research Institute for Artificial Intelligence (ÖFAI)).<br />
<br />
==Open Brainstorming and Agenda Idea Page==<br />
I have started an [[Discussion_Points_Euro_MIREX_Meeting | Discussion Points Euro MIREX Meeting]] page. This page is open to all folks interested in helping to set the agenda for the Euro MIREX Planning Meeting. This includes '''meeting participants''' and '''anyone''' who would like to see a topic addressed by the participants. Any and all topics, suggestions, gripes, etc. related to MIREX are not only welcomed, they are '''''encouraged'''''.<br />
<br />
==Location Specifics (from Thomas Lidy)==<br />
TU Vienna<br><br />
Institute of Software Technology<br />
<br />
Favoritenstrasse 9-11<br><br />
1040 Wien<br />
<br />
The venue is best reached via the Vienna underground: ''U1 Taubstummengasse''.<br />
<br />
From the Vienna-City Holiday Inn (where Dr. Downie and many will be staying) the best way to get to the meeting is to walk. For arrival there you can take bus number 59A from the Opera.<br />
<br />
I have attached a map for orientation, where all of above is indicated (https://www.music-ir.org/mirex2007/images/7/75/MIREX_meeting.png).<br />
<br />
===Finding the building and the meeting room===<br />
Our building has huge metal letters on its front and has two entrances. <br />
Take the entrance with the glass door, search for the elevator and take <br />
the stairs next to it to the 2nd floor. There we will indicate the way <br />
to the meeting room.<br />
<br />
==Keeping In Touch In Vienna==<br />
The IMIRSEL team (i.e., Dr. Downie, Andreas Ehmann, Cameron Jones and Kris West) will be arriving in Vienna around 20:30h, 19 April, 2007. The IMIRSEL team will be leaving Vienna the morning of 25 April, 2007. The IMIRSEL team and many participants will be staying at:<br />
<br />
Holiday Inn <br><br />
VIENNA CITY<br />
<br />
MARGARETHENSTRASSE 53 <br><br />
VIENNA, A-1050 AUSTRIA<br><br />
Hotel Front Desk: 43-1-58850 | Hotel Fax: 43-1-58850 X899 <br><br />
http://tinyurl.com/2oj7u2<br />
<br />
===Personal contacts===<br />
Dr. Downie's cell phone will work in Vienna. The number is +1-217-649-3839. Note the US number: the call will go from Euroland to the States and then back to Vienna even if you are standing beside me. Probably best to TEXT me at this number. If you are not near a phone, a cell text message can get to me via: <2176493839[at]vtext.com>. Normal email is another option but will not have laptop running 24hrs a day, especially at night when we are socializing. Also, evening (CDT) 18 April and most of (GMT) 19 April, will be travel days for the IMIRSEL team so normal email contact will a problem.<br />
<br />
Thomas Lidy has graciously volunteered his contact phone number: +43 650 5594455.<br />
<br />
==Social Events==<br />
Three events will be funded by IMIRSEL. The rest will be the responsibility of the participants.<br />
<br />
#'''SATURDAY, 21 April:''' <br />
##'''Afternoon:''' Thomas Lidy suggests: 15:30h, there is a small spring "festival" with free snacks & beer at Naschmarkt Deli (outdoors I guess), which is a nice place very close to the Holiday Inn! I think we should go there!<br />
##'''Evening:''' TBA but something fun and involving MODEST amounts of imbibing (need to be fresh for Sunday!).<br />
#'''SUNDAY, April 22:'''<br />
##'''Lunch:''' China Buffet all-you-can-eat self-service (Location TBA). ''Funded by IMIRSEL.'' <br />
##'''Evening: Pseudo-Banquet:''' Salm Braeu (http://www.salmbraeu.com). 20:00h. ''Funded by IMIRSEL.''<br />
#'''MONDAY, April 23'''<br />
##'''Lunch:''' Wieden Braeu (http://www.wieden-braeu.at/) will serve us Wiener Schnitzel. Vegetarians have to tell us Sunday morning, so that we can call them, reduce the # of Schnitzel. Veggies will then be ordered separately. ''Funded by IMIRSEL.''<br />
##'''Evening:''' TBA but something fun and involving lots of imbibing and talking and imbibing.<br />
<br />
==Participant List (12 April 2007)==<br />
<csv2>participants_april12b.csv</csv2></div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Sandbox&diff=23942007:Sandbox2007-01-26T19:00:04Z<p>WikiSysop: New page: testing external link: http://www.lis.uiuc.edu/</p>
<hr />
<div>testing<br />
<br />
external link: http://www.lis.uiuc.edu/</div>WikiSysophttps://www.music-ir.org/mirex/w/index.php?title=2007:Main_Page&diff=23352007:Main Page2007-01-26T18:37:59Z<p>WikiSysop: </p>
<hr />
<div><big>'''MediaWiki has been successfully installed.'''</big><br />
<br />
Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.<br />
<br />
== Getting started ==<br />
<br />
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]<br />
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]<br />
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]<br />
<br />
Testing interwiki links<br />
* [[2005:Main_Page]]<br />
* [[2006:Main_Page]]</div>WikiSysop