Difference between revisions of "2008:Query by Tapping"

From MIREX Wiki
(Task description)
(Task description)
 
(25 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
== Overview ==
 
== Overview ==
The main purpose of QBT(Query by Tapping) is to evaluate MIR system in retrieval ground-truth MIDI files by the tapping rhythm. This task provides query rhythm files both in WAV and CSV(symbolic) formate. Evaluation database and query files can be download from http://210.68.135.13/ki/QBT.rar
+
The main purpose of QBT(Query by Tapping) is to evaluate MIR system in retrieval ground-truth MIDI files by the tapping rhythm. This task provides query rhythm files in WAV formate. Each wav file with a corresponding onset file for participant to evaluate by symbolic form. Evaluation database and query files can be download from http://210.68.135.13/ki/QBT.rar (updated on 2008/8/23)
  
 
== Task description ==
 
== Task description ==
* '''Test database''': 129 ground-truth mono MIDI files.
+
* '''Test database''': 103 ground-truth mono MIDI files.
* '''WAV Quries''': 272 query files to retrieve 129 known target from collection. So far, 1~6 human assessors have listened and tapped a 15 seconds query rhythm from beginning for each target songs.
+
* '''WAV Queries''': 533 query files to retrieve 103 known target from collection. So far, 1~6 human assessors have listened and tapped a 15 seconds query rhythm from beginning for each target song. WAV files are recorded in 8k-8bit.
* '''CSV Quries''': 129 query files in CSV format which are drawn from original MIDI. It include time interval (in ms) between first 25 NoteOn event separate with comma. Rest note and time interval smller than 100ms will be discarded.
+
* '''Onset Queries''': 533 onset symbolic query files which drawn from wav files. These onset files can help participant concentrate on similarity matching instead of onset detection. All onset files can't guarantee to have perfect detection result from original wav query files.
* '''Evaluation''': Mean Reciprocal rank. Return top 10 candidates for each query file.
+
* '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise(Top-10 hit rate).
  
 
== Data processing proposal for calling formats ==
 
== Data processing proposal for calling formats ==
Line 34: Line 34:
 
The answer file for each run would look like:
 
The answer file for each run would look like:
 
<pre>
 
<pre>
Q001:0003,0567,0999,<insert X more responses>,XXXX
+
0001:0003,0567,0999,<insert X more responses>,XXXX
Q002:0103,0567,0998,<insert X more responses>,XXXX
+
0002:0103,0567,0998,<insert X more responses>,XXXX
Q00X:0002,0567,0999,<insert X more responses>,XXXX
+
000X:0002,0567,0999,<insert X more responses>,XXXX
 
</pre>
 
</pre>
 
Each line represents to each of the queries in a given task run.
 
Each line represents to each of the queries in a given task run.
Line 46: Line 46:
 
* Shu-Jen Show Hsiao(show.cs95g at nctu.edu.tw)
 
* Shu-Jen Show Hsiao(show.cs95g at nctu.edu.tw)
 
* Rainer Typke: I would be interested if the query data can also be made available in symbolic form so we can see what part of the performance comes from good onset detection from audio, and what comes from a good matching algorithm.
 
* Rainer Typke: I would be interested if the query data can also be made available in symbolic form so we can see what part of the performance comes from good onset detection from audio, and what comes from a good matching algorithm.
 +
* Hong-Ru Lee (khair at mirlab dot org)

Latest revision as of 20:06, 9 September 2008

Overview

The main purpose of QBT(Query by Tapping) is to evaluate MIR system in retrieval ground-truth MIDI files by the tapping rhythm. This task provides query rhythm files in WAV formate. Each wav file with a corresponding onset file for participant to evaluate by symbolic form. Evaluation database and query files can be download from http://210.68.135.13/ki/QBT.rar (updated on 2008/8/23)

Task description

  • Test database: 103 ground-truth mono MIDI files.
  • WAV Queries: 533 query files to retrieve 103 known target from collection. So far, 1~6 human assessors have listened and tapped a 15 seconds query rhythm from beginning for each target song. WAV files are recorded in 8k-8bit.
  • Onset Queries: 533 onset symbolic query files which drawn from wav files. These onset files can help participant concentrate on similarity matching instead of onset detection. All onset files can't guarantee to have perfect detection result from original wav query files.
  • Evaluation: Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise(Top-10 hit rate).

Data processing proposal for calling formats

Indexing the MIDIs collection

Indexing_exe <var1> <var2>

where

<var1>==<directory_of_MIDIs> 
<var2>==<indexing_files_output_and_working_directory>

Running for the query files

Running_exe <var3> <var4> <var5>

where

<var3>==<directory_of_indexed_file> 
<var4>==<directory_of_query_rhythm_files> 
<var5>==<answer_file_output_directory> 

Data processing for output answer file formats

The answer file for each run would look like:

0001:0003,0567,0999,<insert X more responses>,XXXX
0002:0103,0567,0998,<insert X more responses>,XXXX
000X:0002,0567,0999,<insert X more responses>,XXXX

Each line represents to each of the queries in a given task run.

Submission closing date

22th August 2008.

Interested Participants

  • Shu-Jen Show Hsiao(show.cs95g at nctu.edu.tw)
  • Rainer Typke: I would be interested if the query data can also be made available in symbolic form so we can see what part of the performance comes from good onset detection from audio, and what comes from a good matching algorithm.
  • Hong-Ru Lee (khair at mirlab dot org)