Difference between revisions of "2009:Query by Tapping"
From MIREX Wiki
| (38 intermediate revisions by 6 users not shown) | |||
| Line 2: | Line 2: | ||
The text of this section is copied from the 2008 page. Please add your comments and discussions for 2009. | The text of this section is copied from the 2008 page. Please add your comments and discussions for 2009. | ||
| − | The main purpose of QBT(Query by Tapping) is to evaluate MIR system in | + | The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have two corpora for evaluation: |
| + | |||
| + | * Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBT.rar MIR-QBT]: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). | ||
| + | * Show Hsiao's [http://mirlab.org/dataSet/public/QBT_symbolic.rar QBT_symbolic]: This dataset contains only onset files (obtained from the user's tapping on keyboard). | ||
| + | |||
| + | Task deadline is September 15th. | ||
| + | |||
== Task description == | == Task description == | ||
| − | * '''Test database''': | + | |
| − | * ''' | + | === Subtask 1: QBT with symbolic input === |
| − | + | * '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT. | |
| − | * '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise(Top-10 hit rate). | + | * '''Query files''': About 800 text files of onset time to retrieve target MIDIs in MIR_QBT. These onset files can help participant concentrate on similarity matching instead of onset detection. All onset files cannot guarantee to have perfect detection result from original wav query files. |
| + | * '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). | ||
| + | |||
| + | === Subtask 2: QBT with wave input === | ||
| + | * '''Test database''': About 150 ground-truth monophonic MIDI files in MIR-QBT. | ||
| + | * '''Query files''': About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT. | ||
| + | * '''Evaluation''': Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate). | ||
== Disucussions for 2009 == | == Disucussions for 2009 == | ||
| − | + | === Jin-Da's Comments 12/08/2009 === | |
| + | We are going to have three datasets for QBT this year. Two sets are wav files (obtained from tapping on the microphone) with human labelled onset information. The other set is purely onset data obtained from computer keyboard input. Download URL of these datasets will be posted here soon. | ||
| + | === Jin-Da's Comments 12/08/2009 === | ||
| + | I think we should have two subtasks for QBT. | ||
| + | <ol> | ||
| + | <li>Subtask 1: Use the human-labeled onset time for comparison directly. | ||
| + | <li>Subtask 2: Use wav files for onset detection and then perform comparison. | ||
| + | </ol> | ||
| − | == | + | == Command formats == |
=== Indexing the MIDIs collection === | === Indexing the MIDIs collection === | ||
| − | + | Command format should look like this: | |
| − | + | ||
| + | indexing %dbMidi.list% %dir_workspace_root% | ||
| + | |||
| + | where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example: | ||
| + | |||
| + | QBT/database/00001.mid | ||
| + | QBT/database/00002.mid | ||
| + | QBT/database/00003.mid | ||
| + | QBT/database/00004.mid | ||
| + | ... | ||
| + | |||
| + | Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.) | ||
| + | |||
| + | === Test the query files === | ||
| + | The command format should be like this: | ||
| + | |||
| + | qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root% | ||
| + | |||
| + | You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this: | ||
| + | |||
| + | qbtQuery/query_00001.onset 00001.mid | ||
| + | qbtQuery/query_00002.onset 00001.mid | ||
| + | qbtQuery/query_00003.onset 00002.mid | ||
| + | ... | ||
| + | |||
| + | (Pleae refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.) | ||
| − | + | If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this: | |
| − | + | qbtQuery/query_00001.wav 00001.mid | |
| − | + | qbtQuery/query_00002.wav 00001.mid | |
| − | + | qbtQuery/query_00003.wav 00002.mid | |
| + | ... | ||
| − | + | The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1: | |
| − | |||
| − | |||
| − | + | qbtQuery/query_00001.onset: 00025 01003 02200 ... | |
| + | qbtQuery/query_00002.onset: 01547 02313 07653 ... | ||
| + | qbtQuery/query_00003.onset: 03142 00320 00973 ... | ||
| + | ... | ||
| − | + | And for subtask 2: | |
| − | |||
| − | |||
| − | |||
| − | + | qbtQuery/query_00001.wav: 00025 01003 02200 ... | |
| − | + | qbtQuery/query_00002.wav: 01547 02313 07653 ... | |
| − | + | qbtQuery/query_00003.wav: 03142 00320 00973 ... | |
| − | + | ... | |
| − | |||
| − | |||
| − | |||
| − | |||
== Submission closing date == | == Submission closing date == | ||
| − | + | Sept. 15, 2009 | |
==Interested Participants == | ==Interested Participants == | ||
| − | * | + | * Pierre Hanna, Matthias Robine, SIMBALS, University of Bordeaux, France. hanna at labri dot fr |
| + | * Jin-Da Chen, Jyh-Shing Roger Jang, Department of Computer Science, National Tsing-Hua University, Hsinchu, Taiwan. cdc at mirlab dot org | ||
| + | * Shu-Jen Show Hsiao, Tyne Liang, Department of Computer Science, National Chiao-Tung University, Hsinchu, Taiwan, R.O.C, show.cs95g at nctu.edu.tw | ||
Latest revision as of 09:33, 22 March 2010
Contents
Overview
The text of this section is copied from the 2008 page. Please add your comments and discussions for 2009.
The main purpose of QBT (Query by Tapping) is to evaluate MIR system in retrieving ground-truth MIDI files by tapping the onset of music notes to the microphone. This task provides query files in wave format as well as the corresponding human-label onset time in symbolic format. For this year's QBT task, we have two corpora for evaluation:
- Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time).
- Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard).
Task deadline is September 15th.
Task description
Subtask 1: QBT with symbolic input
- Test database: About 150 ground-truth monophonic MIDI files in MIR-QBT.
- Query files: About 800 text files of onset time to retrieve target MIDIs in MIR_QBT. These onset files can help participant concentrate on similarity matching instead of onset detection. All onset files cannot guarantee to have perfect detection result from original wav query files.
- Evaluation: Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate).
Subtask 2: QBT with wave input
- Test database: About 150 ground-truth monophonic MIDI files in MIR-QBT.
- Query files: About 800 wave files of tapping recordings to retrieve MIDIs in MIR-QBT.
- Evaluation: Return top 10 candidates for each query file. 1 point is scored for a hit in the top 10 and 0 is scored otherwise (Top-10 hit rate).
Disucussions for 2009
Jin-Da's Comments 12/08/2009
We are going to have three datasets for QBT this year. Two sets are wav files (obtained from tapping on the microphone) with human labelled onset information. The other set is purely onset data obtained from computer keyboard input. Download URL of these datasets will be posted here soon.
Jin-Da's Comments 12/08/2009
I think we should have two subtasks for QBT.
- Subtask 1: Use the human-labeled onset time for comparison directly.
- Subtask 2: Use wav files for onset detection and then perform comparison.
Command formats
Indexing the MIDIs collection
Command format should look like this:
indexing %dbMidi.list% %dir_workspace_root%
where %dbMidi.list% is the input list of database midi files named as uniq_key.mid. For example:
QBT/database/00001.mid QBT/database/00002.mid QBT/database/00003.mid QBT/database/00004.mid ...
Output indexed files are placed into %dir_workspace_root%. (Note that this step is not required unless you want to index or preprocess the midi database.)
Test the query files
The command format should be like this:
qbtProgram %dbMidi_list% %query_file_list% %resultFile% %dir_workspace_root%
You can use %dir_workspace_root% to store any temporary indexing/database structures. (You can omit %dir_workspace_root% if you do not need it at all.) If the input query files are onset files (for subtask 1), then the format of %query_file_list% is like this:
qbtQuery/query_00001.onset 00001.mid qbtQuery/query_00002.onset 00001.mid qbtQuery/query_00003.onset 00002.mid ...
(Pleae refer to the readme.txt of the downloaded MIR-QBT corpus for the format of onset files.)
If the input query files are wave files (for subtask 2), the the format of %query_file_list% is like this:
qbtQuery/query_00001.wav 00001.mid qbtQuery/query_00002.wav 00001.mid qbtQuery/query_00003.wav 00002.mid ...
The result file gives top-10 candidates for each query. For instance, for wave query file, the result file should have the following format for subtask 1:
qbtQuery/query_00001.onset: 00025 01003 02200 ... qbtQuery/query_00002.onset: 01547 02313 07653 ... qbtQuery/query_00003.onset: 03142 00320 00973 ... ...
And for subtask 2:
qbtQuery/query_00001.wav: 00025 01003 02200 ... qbtQuery/query_00002.wav: 01547 02313 07653 ... qbtQuery/query_00003.wav: 03142 00320 00973 ... ...
Submission closing date
Sept. 15, 2009
Interested Participants
- Pierre Hanna, Matthias Robine, SIMBALS, University of Bordeaux, France. hanna at labri dot fr
- Jin-Da Chen, Jyh-Shing Roger Jang, Department of Computer Science, National Tsing-Hua University, Hsinchu, Taiwan. cdc at mirlab dot org
- Shu-Jen Show Hsiao, Tyne Liang, Department of Computer Science, National Chiao-Tung University, Hsinchu, Taiwan, R.O.C, show.cs95g at nctu.edu.tw
