Difference between revisions of "2009:Query-by-Tapping Results"

From MIREX Wiki
(Task 2 Summary Results by Query Group)
Line 14: Line 14:
 
===General Legend===
 
===General Legend===
 
====Team ID====
 
====Team ID====
'''CSJ''' = [[Chun-Ta Chen and Jyh-Shing Roger Jang]]<br/>
+
'''CSJ''' = [[https://www.music-ir.org/mirex/2009/results/qbt/QbtChenJang.pdf  Chun-Ta Chen and Jyh-Shing Roger Jang]]<br/>
'''HAFR''' = [[Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine]]<br/>
+
'''HAFR''' = [[https://www.music-ir.org/mirex/2009/results/qbt/HAFR.pdf  Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine]]<br/>
'''HL''' = [[Shu-Jen Show Hsiao,Tyne Liang]]<br/>
+
'''HL''' = [[https://www.music-ir.org/mirex/2009/results/qbt/QBT_Show.pdf  Shu-Jen Show Hsiao,Tyne Liang]]<br/>
  
 
===Task 1 Results===
 
===Task 1 Results===

Revision as of 01:00, 15 October 2009

Introduction

These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

  • Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
  • Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.


Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.

General Legend

Team ID

CSJ = [Chun-Ta Chen and Jyh-Shing Roger Jang]
HAFR = [Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine]
HL = [Shu-Jen Show Hsiao,Tyne Liang]

Task 1 Results

Task 1a, Jang's Dataset Results

Task 1a Overall Results

file /nema-raid/www/mirex/results/qbt/QbtFinalTask1a.csv not found

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count: file /nema-raid/www/mirex/results/qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv not found

File:SQbtTask1aSimple Friedman Mean Ranks.pngΓÇÄ

MRR Method: file /nema-raid/www/mirex/results/qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv not found

File:SQbt1aTask2Mrr Friedman Mean Ranks.pngΓÇÄ

Task 1a Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/qbt/QbtTask1aSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbt/QbtTask1aMrrByGroup.csv not found

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]


Task 1b, Hsiao's Dataset Results

Task 1b Overall Results

file /nema-raid/www/mirex/results/qbt/QbtFinalTask1b.csv not found

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count: file /nema-raid/www/mirex/results/qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv not found

File:SQbtTask1bSimple Friedman Mean Ranks.pngΓÇÄ

MRR Method: file /nema-raid/www/mirex/results/qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv not found

File:SQbtTask1bMrr Friedman Mean Ranks.png

Task 1b Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/qbt/QbtTask1bSimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbt/QbtTask1bMrrByGroup.csv not found

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 2 Results

Task 2 Overall Results

file /nema-raid/www/mirex/results/qbt/QbtFinalTask2.csv not found

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count: file /nema-raid/www/mirex/results/qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv not found

File:SQbtTask2Simple Friedman Mean Ranks.pngΓÇÄ

MRR Method: file /nema-raid/www/mirex/results/qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv not found

File:SQbtTask2Mrr Friedman Mean Ranks.png

Task 2 Summary Results by Query Group

Simple Hit/Miss Counting file /nema-raid/www/mirex/results/qbt/QbtTask2SimpleByGroup.csv not found

MRR Method file /nema-raid/www/mirex/results/qbt/QbtTask2MrrByGroup.csv not found

Task 2 Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Runtime Results

file /nema-raid/www/mirex/results/qbsh.runtime.csv not found