https://www.music-ir.org/mirex/w/index.php?title=2018:Set_List_Identification&feed=atom&action=history2018:Set List Identification - Revision history2024-03-29T12:37:49ZRevision history for this page on the wikiMediaWiki 1.31.1https://www.music-ir.org/mirex/w/index.php?title=2018:Set_List_Identification&diff=12428&oldid=prevYun Hao: Created page with "__TOC__ ==Description== This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence in a..."2018-05-07T15:52:50Z<p>Created page with "__TOC__ ==Description== This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence in a..."</p>
<p><b>New page</b></p><div>__TOC__<br />
<br />
==Description==<br />
<br />
This task requires that algorithm identify the '''set list''' (See [http://en.wikipedia.org/wiki/Set_list Set list]). Set list is the song sequence in a live concert. It shows the order of songs will be performed in a live concert.<br />
<br />
Recently, more and more full-length live concert videos have become available on website (e.g. [https://www.youtube.com/ Youtube]). Most of them are lacking sufficient information to describe itself, such as the set list, and start/end time of each song. In this task, we collect the audio of live concerts and studio songs, applying music information retrieval techniques to answer this question -- what songs had been sung in this concert and when are the songs start and end.<br />
<br />
For the first step of this task, we assume that '''artist is known'''. In the live concert, '''the performers play their studio songs only''', however the ultimate goal is granted a full-length live concert audio and studio song database, we still can find out the set list and the start/end time of each song.<br />
<br />
here are two sub tasks in this task:<br />
<br />
===Sub task 1: Song sequence identification===<br />
*To identify the order of songs which be performed in a live concert.<br />
<br />
In this sub task, the participants known the the artist and artist's studio song collection. Assigning a live concert audio and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify the order of songs in this live concert.<br />
<br />
===Sub task 2: Time boundary identification===<br />
*To identify the start/end time of each song in song sequence<br />
<br />
In this sub task, the participants known the artist, artist's studio song collection and the '''song sequence'''. Assigning a live concert audio, song sequence and studio songs collection of a specific artist, all songs in live concert are included in studio songs collection, to identify start time and end time of each song in the live concert.<br />
<br />
== Data ==<br />
To satisfy our assessment, we pre-process all audio -- '''remove the "out of artist song" form live concert audio''' for following our assumption. (See the [https://www.music-ir.org/mirex/wiki/2016:Set_List_Identification#Description description])<br />
<br />
We provide two set for this task,participating algorithms will have to read audio in the following format.<br />
<br />
* Sample rate: 22050 Hz<br />
* Sample size: 16 bit<br />
* Number of channels: 1 (mono)<br />
* Encoding: WAV <br />
<br />
===Developing set===<br />
This set contain 3 artists and 7 live concerts, the following information would be release ([https://www.dropbox.com/sh/t83ogdrxi0f050n/AABb11MCcQUokqSjOsqhArOFa?dl=0 Dropbox])<br />
* artist<br />
* live concert name and links<br />
* studio collection list<br />
* start/end time tags<br />
<br />
We extract features for the convenience of participants, the links is the tool we used. ([https://www.dropbox.com/s/bote36k8pkmt2f8/MIREX_2015_Setlist_ID_Developing_set_chroma_fea.rar?dl=0 Dropbox])<br />
*chroma (CRP features [http://resources.mpi-inf.mpg.de/MIR/chromatoolbox/ Chroma Toolbox])<br />
<br />
Collection statistics:<br />
* 3 artists<br />
* 7 live concerts<br />
* 279 tracks<br />
<br />
=== Testing set ===<br />
This set contain 7 artists and 13 live concerts, no information would be release.<br />
<br />
Collection statistics:<br />
* 7 artists<br />
* 13 live concerts<br />
* 873 tracks<br />
<br />
== Evaluation ==<br />
<br />
For two tasks, the evaluation metrics were different.<br />
<br />
=== Sub task 1===<br />
<br />
* Edit distance (see [http://en.wikipedia.org/wiki/Edit_distance Edit distance])<br />
<br />
We evaluated the two sequence (ground truth and your result) by edit distance, there three errors included<br />
* insertion error <math>I</math><br />
* substitution error <math>S</math><br />
* deletion error <math>D</math><br />
<br />
Edit Distance: <big><math>ED = I+S+D </math> </big> <br />
<br />
Percent Correct: <big><math>Corr = \frac{N-D-S}{N}</math></big><br />
<br />
Percent Accuracy: <big><math> Acc = \frac{N-D-S-I}{N}</math></big><br />
<br />
=== Sub task 2===<br />
<br />
* average time boundary<br />
<br />
We will evaluate two time boundaries as follow: average start time boundary and average end time boundary. The evaluation function is described below:<br />
<br />
* Set list contains '''<math>N</math>''' songs<br />
<br />
''' Ground truth: '''<br />
<br />
* Start time of song '''<math>i</math>''':<math>sBD_{GT_i}</math><br />
<br />
* End time of song '''<math>i</math>''':<math>eBD_{GT_i}</math><br />
<br />
''' Identification result: ''' <br />
<br />
* Start time of song '''<math>i</math>''':<math>sBD_{ID_i}</math><br />
<br />
* End time of song '''<math>i</math>''':<math>eBD_{ID_i}</math><br />
<br />
<math> AVGsBD =\frac{\sum_{i=1}^N |sBD_{GT_i} - sBD_{ID_i}|}{N} </math>,<br />
<br />
<math> AVGeBD =\frac{\sum_{i=1}^N |eBD_{GT_i} - eBD_{ID_i}|}{N} </math>,<br />
<br />
=== Runtime performance ===<br />
In addition computation times for feature extraction and training/classification will be measured.<br />
<br />
== Submission Format ==<br />
* '''\n''' is end of line<br />
<br />
Submission to this task will have to conform to a specified format detailed below.<br />
=== Implementation details ===<br />
we recommend your submission folder construction as follow:<br />
/root_folder/... all the code you submitted<br />
/root_folder/extract_feature/... all feature your extracted<br />
/root_folder/output/... the folder to save results<br />
<br />
=== Sub task 1 ===<br />
<br />
Two inputs : live file list and studio song file list<br />
<br />
One output: song ID sequence<br />
<br />
==== Input file ====<br />
The input for studio songs list file format will be of the form:<br />
<br />
/path/to/artist_1/studio/song/001.wav\n 1st<br />
/path/to/artist_1/studio/song/002.wav\n 2nd<br />
/path/to/artist_1/studio/song/003.wav\n 3rd<br />
... <br />
<br />
The input for live concert list file format will be of the form:<br />
<br />
/path/to/artist_1/live/concert/001.wav\n<br />
<br />
==== Output file ====<br />
The output is a list file (song ID sequence), '''the song ID is the order of input list file''', not the file name of *.wav file.<br />
<br />
3\n <-- 003.wav is the first song of set list for your identification result<br />
17\n<br />
59\n<br />
...<br />
<br />
=== Sub task 2 ===<br />
<br />
Three inputs : song ID sequence list, live file list and studio song file list<br />
<br />
One output: time label of song list<br />
<br />
==== Input file ====<br />
<br />
The input is a list of song ID (song ID sequence), '''the song ID is the order of studio songs list file'''.<br />
<br />
Your system should read the *.wav file according that order and find the time boundary of the song.<br />
<br />
3\n<br />
17\n<br />
59\n<br />
...<br />
<br />
The input for live concert list file format will be of the form:<br />
<br />
/path/to/artist_1/live/concert/001.wav\n<br />
<br />
The input for studio songs list file format will be of the form:<br />
<br />
/path/to/artist_1/studio/song/001.wav\n 1st<br />
/path/to/artist_1/studio/song/002.wav\n 2nd<br />
/path/to/artist_1/studio/song/003.wav\n 3rd<br />
... <br />
<br />
==== Output file ====<br />
<br />
The output for studio songs time boundary list file format will be of the form:<br />
* please round the time boundary to millisecond<br />
* '''\t''' is tab space<br />
Start time end time<br />
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n (for input input sond ID:3)<br />
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n (for input input sond ID:17)<br />
hours.minutes.seconds.milliseconds \t hours.minutes.seconds.milliseconds\n (for input input sond ID:59)<br />
... <br />
<br />
Examples:<br />
0.7.23.521 0.13.24.512<br />
0.14.3.021 0.19.53.38<br />
0.20.9.893 0.27.15.987<br />
...<br />
...<br />
0.56.22.433 1.1.46.593<br />
1.3.51.146 1.9.21.138<br />
...<br />
<br />
=== Packaging submissions ===<br />
<br />
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).<br />
<br />
All submissions should include a README file including the following the information:<br />
<br />
* Which task you want to participate (sub task1, sub task2 or all)<br />
* Command line calling format for all executables and an example formatted set of commands<br />
* Number of threads/cores used or whether this should be specified on the command line<br />
* Expected memory footprint<br />
* Expected runtime<br />
* Any required environments (and versions), e.g. python, java, bash, matlab.<br />
<br />
== Time and hardware limits ==<br />
Due to the potentially high number of particpants in this and other audio tasks, hard limits on the runtime of submissions are specified. <br />
<br />
A hard limit of 72 hours will be imposed on runs (total feature extraction and querying times). Submissions that exceed this runtime may not receive a result.<br />
<br />
== Potential Participants ==<br />
name / email</div>Yun Hao