Difference between revisions of "2009:Query by Tapping"

From MIREX Wiki
(Jin-Da's Comments 12/08/2009)
(Data processing for output answer file formats)
Line 71: Line 71:
 
  ...
 
  ...
  
== Data processing for output answer file formats ==
+
 
The answer file for each run would look like:
 
<pre>
 
0001:0003,0567,0999,<insert X more responses>,XXXX
 
0002:0103,0567,0998,<insert X more responses>,XXXX
 
000X:0002,0567,0999,<insert X more responses>,XXXX
 
</pre>
 
Each line represents to each of the queries in a given task run.
 
  
 
== Submission closing date ==
 
== Submission closing date ==

Revision as of 11:00, 6 September 2009

Overview

The text of this section is copied from the 2008 page. Please add your comments and discussions for 2009.

The main purpose of QBT(Query by Tapping) is to evaluate MIR system in retrieval ground-truth MIDI files by the tapping rhythm. This task provides query rhythm files in WAV formate. Each wav file with a corresponding onset file for participant to evaluate by symbolic form. Evaluation database and query files can be download from http://210.68.135.13/ki/QBT.rar

Task description

Subtask 1: QBT with symbolic input

  • Test database: 103 ground-truth mono MIDI files.
  • Query files: 533 text files of onset time to retrieve 103 known target from collection. These onset files can help participant concentrate on similarity matching instead of onset detection. All onset files can't guarantee to have perfect detection result from original wav query files.
  • Evaluation: Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate).

Subtask 2: QBT with wave input

  • Test database: 103 ground-truth mono MIDI files.
  • Query files: 533 wave files of tapping recordings to retrieve 103 known target from collection. So far, 1~6 human assessors have listened and tapped a 15 seconds query rhythm from beginning for each target song. Wave files are recorded in 8k-8bit.
  • Evaluation: Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate).

Disucussions for 2009

Jin-Da's Comments 12/08/2009

We are going to have three datasets for QBT this year. Two sets are wav files (obtained from tapping on the microphone) with human labelled onset information. The other set is purely onset data obtained from computer keyboard input. Download URL of these datasets will be posted here soon.

Jin-Da's Comments 12/08/2009

I think we should have two subtasks for QBT.

  1. Subtask 1: Use the human-labeled onset time for comparison directly.
  2. Subtask 2: Use wav files for onset detection and then perform comparison.

Data processing proposal for calling formats

Indexing the MIDIs collection

Command format should look like this:

indexing %dbMidi.list% %dir_workspace_root%

where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:

./QBT/Database/00001.mid
./QBT/Database/00002.mid
./QBT/Database/00003.mid
./QBT/Database/00004.mid
...

Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)

Test the query files

The command format should be like this:

qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%

You can use %dir_workspace_root% to store any temporary indexing/database structures. If the input query files are wave format for subtask 1, then the format of %query_file_list% is like this:

./QBT_query/query_00001.wav
./QBT_query/query_00002.wav
./QBT_query/query_00003.wav 
...

If the input query files are onset files (for subtask 2), the the format of %query_file_list% is like this:

./QBT_query/query_00001.onset
./QBT_query/query_00002.onset
./QBT_query/query_00003.onset
...

The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format:

query_00001.wav: 00025 01003 02200 ... 
query_00002.wav: 01547 02313 07653 ... 
query_00003.wav: 03142 00320 00973 ... 
...


Submission closing date

TBA

Interested Participants

  • Pierre Hanna, Matthias Robine, SIMBALS, University of Bordeaux, France. hanna at labri dot fr
  • Jin-Da Chen, Jyh-Shing Roger Jang, Department of Computer Science, National Tsing-Hua University, Hsinchu, Taiwan. cdc at mirlab dot org
  • Shu-Jen Show Hsiao, Tyne Liang, Department of Computer Science, National Chiao-Tung University, Hsinchu, Taiwan, R.O.C, show.cs95g at nctu.edu.tw