2007:Query-by-Singing/Humming Results

From MIREX Wiki
Revision as of 22:45, 16 September 2007 by CameronJones (talk | contribs) (Task 2 Results)

Introduction

These are the results for the 2007 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last years. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.

Task 2 Goto Task 2 Results: In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

General Legend

Team ID

FH = Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine
CG = Carlos G├│mez, Soraya Abad-Mota, Edna Ruckhaus
RJ1 = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1
RJ2 = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2
NM = Kjell Lemström, Niko Mikkilä
XW1 = Xiao Wu, Ming Li 1
XW2 = Xiao Wu, Ming Li 2
AU1 = Alexandra L. Uitdenbogerd 1
AU2 = Alexandra L. Uitdenbogerd 2
AU3 = Alexandra L. Uitdenbogerd 3

Task 1 Results

The first subtask is the same as last years. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. The query database consists of 2797 sung queries.

Task 1 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh07_task1_sum_friedmans.csv not found file /nema-raid/www/mirex/results/qbsh07_task1_detail_friedmans.csv not found File:Qbsh07 task1 friedmans.png

Task 1 Summary Results by Query Group

file /nema-raid/www/mirex/results/qbsh07_task1_avg_per_group.csv not found

Task 2 Results

In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.

Task 2 Legend

Team ID

FH_XW = Pascal Ferraro, Pierre Hanna, Julien Allali, Matthias Robine based on XW note transcriber
CG_XW = Carlos G├│mez, Soraya Abad-Mota, Edna Ruckhaus based on XW note transcriber
RJ1_RJ = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on RJ pitch transcriber
RJ1_XW = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 1 based on XW pitch transcriber
RJ2_RJ = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on RJ pitch transcriber
RJ2_XW = J.-S. Roger Jang, Nien-Jung Lee, Chao-Ling Hsu 2 based on XW pitch transcriber
NM_XW = Kjell Lemström, Niko Mikkilä based on XW note transcriber
XW1_XW = Xiao Wu, Ming Li 1 based on XW note transcriber
XW2_XW = Xiao Wu, Ming Li 2 based on XW pitch transcriber

Task 2 Overall Results

file /nema-raid/www/mirex/results/qbsh07_task_2_overall.csv not found

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); file /nema-raid/www/mirex/results/qbsh07_task2_sum_friedmans.csv not found file /nema-raid/www/mirex/results/qbsh07_task2_detail_friedmans.csv not found File:Qbsh07 task2 friedmans.png

Task 2 Summary Results by Query Group

file /nema-raid/www/mirex/results/qbsh07_task2_avg_per_group.csv not found