Difference between revisions of "2009:Query-by-Singing/Humming Results"

From MIREX Wiki
m (Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2009:\1))
m (Robot: Automated text replacement (-<csv([^>]*)> +<csv\1>2009/))
Line 24: Line 24:
  
 
=====Task 1a Overall Results=====
 
=====Task 1a Overall Results=====
−
<csv>qbsh/QbshFinalTask1a.csv</csv>
+
<csv>2009/qbsh/QbshFinalTask1a.csv</csv>
  
 
=====Task 1a Friedman's Test for Significant Differences=====
 
=====Task 1a Friedman's Test for Significant Differences=====
Line 31: Line 31:
  
 
Simple Hit/Miss Count:
 
Simple Hit/Miss Count:
−
<csv>qbsh/friedman/Qbsh1aTask2Simple_friedman_tukeyKramerHSD.csv</csv>
+
<csv>2009/qbsh/friedman/Qbsh1aTask2Simple_friedman_tukeyKramerHSD.csv</csv>
  
 
[[Image:SQbsh1aTask2Simple_Friedman_Mean_Ranks.pngΓÇÄ]]
 
[[Image:SQbsh1aTask2Simple_Friedman_Mean_Ranks.pngΓÇÄ]]
  
 
MRR Method:
 
MRR Method:
−
<csv>qbsh/friedman/Qbsh1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
+
<csv>2009/qbsh/friedman/Qbsh1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
  
 
[[Image:SQbsh1aTask2Mrr_Friedman_Mean_Ranks.pngΓÇÄ]]
 
[[Image:SQbsh1aTask2Mrr_Friedman_Mean_Ranks.pngΓÇÄ]]
Line 42: Line 42:
 
=====Task 1a Summary Results by Query Group=====
 
=====Task 1a Summary Results by Query Group=====
 
Simple Hit/Miss Count
 
Simple Hit/Miss Count
−
<csv>qbsh/QbshTask1aSimpleByGroup.csv</csv>
+
<csv>2009/qbsh/QbshTask1aSimpleByGroup.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv>qbsh/QbshTask1aMrrByGroup.csv</csv>
+
<csv>2009/qbsh/QbshTask1aMrrByGroup.csv</csv>
  
 
====Task 1a Summary Results by Query ====
 
====Task 1a Summary Results by Query ====
Line 57: Line 57:
  
 
=====Task 1b Overall Results=====
 
=====Task 1b Overall Results=====
−
<csv>qbsh/QbshFinalTask1b.csv</csv>
+
<csv>2009/qbsh/QbshFinalTask1b.csv</csv>
  
 
=====Task 1b Friedman's Test for Significant Differences=====
 
=====Task 1b Friedman's Test for Significant Differences=====
Line 64: Line 64:
  
 
Simple Hit/Miss Count:
 
Simple Hit/Miss Count:
−
<csv>qbsh/friedman/Qbsh1bTask2Simple_friedman_tukeyKramerHSD.csv</csv>
+
<csv>2009/qbsh/friedman/Qbsh1bTask2Simple_friedman_tukeyKramerHSD.csv</csv>
  
 
[[Image:SQbsh1bTask2Simple_Friedman_Mean_Ranks.pngΓÇÄ]]
 
[[Image:SQbsh1bTask2Simple_Friedman_Mean_Ranks.pngΓÇÄ]]
  
 
MRR Method:
 
MRR Method:
−
<csv>qbsh/friedman/Qbsh1bTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
+
<csv>2009/qbsh/friedman/Qbsh1bTask2Mrr_friedman_tukeyKramerHSD.csv</csv>
  
 
[[Image:SQbsh1bTask2Mrr_Friedman_Mean_Ranks.pngΓÇÄ]]
 
[[Image:SQbsh1bTask2Mrr_Friedman_Mean_Ranks.pngΓÇÄ]]
 
=====Task 1b Summary Results by Query Group=====
 
=====Task 1b Summary Results by Query Group=====
 
Simple Hit/Miss Count
 
Simple Hit/Miss Count
−
<csv>qbsh/QbshTask1bSimpleByGroup.csv</csv>
+
<csv>2009/qbsh/QbshTask1bSimpleByGroup.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv>qbsh/QbshTask1bMrrByGroup.csv</csv>
+
<csv>2009/qbsh/QbshTask1bMrrByGroup.csv</csv>
  
 
====Task 1b Summary Results by Query ====
 
====Task 1b Summary Results by Query ====
Line 92: Line 92:
 
=====Task 1c Summary Results by Query Group=====
 
=====Task 1c Summary Results by Query Group=====
 
Simple Hit/Miss Count
 
Simple Hit/Miss Count
−
<csv>qbsh/QbshTask1cSimpleByGroup.csv</csv>
+
<csv>2009/qbsh/QbshTask1cSimpleByGroup.csv</csv>
  
 
MRR Method
 
MRR Method
−
<csv>qbsh/QbshTask1cMrrByGroup.csv</csv>
+
<csv>2009/qbsh/QbshTask1cMrrByGroup.csv</csv>
  
 
====Task 1c Summary Results by Query ====
 
====Task 1c Summary Results by Query ====
Line 107: Line 107:
  
 
=====Task 2 Overall Results=====
 
=====Task 2 Overall Results=====
−
<csv>qbsh/QbshFinalTask2.csv</csv>
+
<csv>2009/qbsh/QbshFinalTask2.csv</csv>
  
 
=====Task 2 Friedman's Test for Significant Differences=====
 
=====Task 2 Friedman's Test for Significant Differences=====
Line 113: Line 113:
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
 
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
  
−
<csv>qbsh/friedman/QbshTask2_friedman_tukeyKramerHSD.csv</csv>
+
<csv>2009/qbsh/friedman/QbshTask2_friedman_tukeyKramerHSD.csv</csv>
  
 
[[Image:SQbshTask2_Friedman_Mean_Ranks.png ΓÇÄ]]
 
[[Image:SQbshTask2_Friedman_Mean_Ranks.png ΓÇÄ]]
Line 119: Line 119:
  
 
=====Task 2 Summary Results by Query Group=====
 
=====Task 2 Summary Results by Query Group=====
−
<csv>qbsh/QbshTask2Group.csv</csv>
+
<csv>2009/qbsh/QbshTask2Group.csv</csv>
  
 
====Task 2 Summary Results by Query ====
 
====Task 2 Summary Results by Query ====
Line 126: Line 126:
 
===Runtime Results===
 
===Runtime Results===
  
−
<csv>qbsh/qbshRunTime.csv</csv>
+
<csv>2009/qbsh/qbshRunTime.csv</csv>
  
 
[[Category: Results]]
 
[[Category: Results]]

Revision as of 18:59, 13 May 2010

Introduction

These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2009:Query by Singing/Humming page.

Task Descriptions

Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:

Task 2 Goto Task 2 Results: The second subtask is the query against other humming. In the second subtask, Roger Jang's MIR-QBSH corpus has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song.

General Legend

Team ID

CSJ1 = Jyh-Shing Roger Jang matched by beginning of the sond
CSJ2 = Jyh-Shing Roger Jang matched by anywhere of the song
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine

Task 1 Results

Task 1a, Jang's dataset Results

Task 1a Overall Results
CSJ1 CSJ2 HAFR
Simple Count 0.94 0.9 0.77
MRR 0.91 0.86 0.66
Total count 4431 4431 4409

download these results as csv

Task 1a Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ1 CSJ2 0.3232 0.7812 1.2393 TRUE
CSJ1 HAFR 1.1670 1.6250 2.0830 TRUE
CSJ2 HAFR 0.3857 0.8438 1.3018 TRUE

download these results as csv

File:SQbsh1aTask2Simple Friedman Mean Ranks.pngΓÇÄ

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ1 CSJ2 0.2009 0.6667 1.1324 TRUE
CSJ1 HAFR 1.1801 1.6458 2.1116 TRUE
CSJ2 HAFR 0.5134 0.9792 1.4449 TRUE

download these results as csv

File:SQbsh1aTask2Mrr Friedman Mean Ranks.pngΓÇÄ

Task 1a Summary Results by Query Group

Simple Hit/Miss Count

CSJ1 CSJ2 HAFR
1 0.89 0.84 0.89
2 0.71 0.5 0.36
3 0.64 0.64 0.64
4 0.8 0.8 0.67
5 0.93 0.73 0.87
6 0.92 0.75 0.67
7 0.93 0.87 0.47
8 0.93 0.93 0.6
9 0.92 0.92 0.73
10 0.74 0.84 0.84
11 0.94 0.89 0.8
12 0.96 0.95 0.92
13 0.98 0.97 0.88
14 0.98 0.97 0.92
15 0.96 0.96 0.78
16 0.93 0.9 0.93
17 0.97 0.85 0.73
18 0.99 0.96 0.75
19 0.97 0.94 0.89
20 0.99 0.96 0.46
21 0.88 0.79 0.32
22 0.97 0.97 0.95
23 0.98 0.98 0.85
24 0.83 0.82 0.58
25 0.72 0.45 0.57
26 0.98 0.92 0.65
27 0.95 0.98 0.76
28 0.87 0.69 0.49
29 0.98 0.97 0.57
30 0.97 0.94 0.94
31 0.97 0.95 0.94
32 0.96 0.93 0.8
33 0.92 0.86 0.5
34 0.93 0.91 0.64
35 0.95 0.87 0.76
36 0.93 0.91 0.91
37 0.92 0.91 0.88
38 0.95 0.95 0.78
39 0.89 0.83 0.82
40 0.94 0.9 0.44
41 0.94 0.93 0.9
42 0.96 0.93 0.86
43 0.95 0.94 0.9
44 0.93 0.94 0.88
45 0.95 0.91 0.83
46 0.95 0.92 0.8
47 1 0.97 0.93
48 1 0.96 0.81

download these results as csv

MRR Method

CSJ1 CSJ2 HAFR
1 0.82 0.77 0.82
2 0.61 0.5 0.36
3 0.64 0.64 0.58
4 0.8 0.8 0.62
5 0.82 0.7 0.72
6 0.79 0.62 0.59
7 0.89 0.87 0.47
8 0.88 0.93 0.52
9 0.88 0.83 0.64
10 0.7 0.84 0.72
11 0.92 0.85 0.72
12 0.94 0.94 0.79
13 0.96 0.94 0.74
14 0.95 0.9 0.81
15 0.94 0.96 0.7
16 0.9 0.89 0.9
17 0.91 0.7 0.56
18 0.97 0.93 0.64
19 0.96 0.91 0.8
20 0.98 0.95 0.3
21 0.75 0.69 0.24
22 0.95 0.95 0.91
23 0.96 0.96 0.71
24 0.78 0.75 0.49
25 0.58 0.35 0.4
26 0.97 0.9 0.57
27 0.95 0.95 0.58
28 0.74 0.58 0.38
29 0.96 0.95 0.43
30 0.96 0.9 0.88
31 0.97 0.94 0.84
32 0.93 0.89 0.64
33 0.88 0.81 0.36
34 0.91 0.87 0.53
35 0.93 0.86 0.67
36 0.93 0.91 0.84
37 0.87 0.86 0.8
38 0.9 0.91 0.63
39 0.86 0.79 0.69
40 0.89 0.85 0.31
41 0.91 0.91 0.82
42 0.94 0.91 0.76
43 0.91 0.85 0.8
44 0.92 0.93 0.84
45 0.93 0.9 0.77
46 0.92 0.9 0.74
47 0.98 0.94 0.83
48 0.95 0.95 0.7

download these results as csv

Task 1a Summary Results by Query

Simple Hit/Miss Counting [1]

MRR Method [2]

Task 1b, ThinkIT's dataset Results

Task 1b Overall Results
CSJ1 CSJ2 HAFR
Simple Count 0.43 0.86 0.78
MRR 0.41 0.8 0.68
Total count 355 355 355

download these results as csv

Task 1b Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

Simple Hit/Miss Count:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ2 HAFR -0.0394 0.2217 0.4828 FALSE
CSJ2 CSJ1 0.7058 0.9670 1.2281 TRUE
HAFR CSJ1 0.4841 0.7453 1.0064 TRUE

download these results as csv

File:SQbsh1bTask2Simple Friedman Mean Ranks.pngΓÇÄ

MRR Method:

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ2 HAFR 0.0442 0.3208 0.5974 TRUE
CSJ2 CSJ1 0.6904 0.9670 1.2436 TRUE
HAFR CSJ1 0.3696 0.6462 0.9228 TRUE

download these results as csv

File:SQbsh1bTask2Mrr Friedman Mean Ranks.pngΓÇÄ

Task 1b Summary Results by Query Group

Simple Hit/Miss Count

CSJ1 CSJ2 HAFR
1 0.55 1 0.82
2 0.6 0.8 0.8
3 0 0 1
4 0 1 1
5 0 0 0
6 0 0.83 0.83
7 1 1 0.83
8 0.33 0.33 0
9 0 0.8 1
10 0 1 1
11 0 1 0.5
12 1 1 1
13 0 0.6 0.6
14 0 0.5 0.5
15 0 0.5 0
16 1 0.5 1
17 0 0 0
18 0.5 1 0.5
19 0 1 1
20 0 0 1
21 0 0 0.5
22 0.44 0.89 0.89
23 1 1 1
24 0 1 1
25 0.5 1 1
26 0.64 0.91 0.91
27 0.5 1 0
28 0.5 1 0.5
29 1 1 1
30 0.75 1 0.75
31 1 1 1
32 0 1 0.86
33 0 1 1
34 0.5 0.5 0
35 0 1 1
36 0 0.8 0.6
37 0 0 0
38 0.5 1 1
39 0.43 1 1
40 1 1 0.5
41 0.13 0.88 0.63
42 0.67 0.92 0.5
43 0.67 0.67 0.56
44 0.56 1 1
45 1 1 0.67
46 0.5 0.75 0.75
47 0.75 1 1
48 1 1 1
49 1 1 1
50 1 1 1
51 0.5 0.75 1
52 0.25 1 0.75
53 0.33 1 0.83
54 0.67 1 1
55 0 1 0.75
56 0.5 0.5 0.5
57 1 1 0.4
58 0 1 1
59 0 1 1
60 1 1 0.5
61 0 1 1
62 0.33 1 1
63 1 1 1
64 0 1 0
65 0 0.67 1
66 0.5 0.75 1
67 0 1 0.67
68 1 1 1
69 0 1 1
70 1 1 1
71 0 0.67 1
72 1 1 1
73 1 1 0.5
74 1 1 1
75 0.33 0.67 0.67
76 0 1 0.5
77 1 1 1
78 1 1 1
79 0.5 1 1
80 0.33 1 1
81 0.33 0.5 1
82 0 1 1
83 0 1 1
84 0 0.25 0.5
85 0.5 1 0.75
86 1 1 1
87 0 1 0.33
88 0 1 1
89 0.43 0.86 1
90 0 1 1
91 0 1 0.5
92 0.5 1 1
93 0.5 1 1
94 0 1 1
95 0 1 0
96 1 0 1
97 0 1 1
98 0 1 1
99 0.75 1 1
100 0.33 1 1
101 0 0 0
102 0 1 1
103 0.38 0.75 0.63
104 0 1 1
105 1 0 1
106 0 0 0

download these results as csv

MRR Method

CSJ1 CSJ2 HAFR
1 0.42 0.95 0.82
2 0.6 0.8 0.55
3 0 0 1
4 0 0.67 0.5
5 0 0 0
6 0 0.83 0.83
7 1 0.92 0.83
8 0.04 0.33 0
9 0 0.8 0.85
10 0 1 1
11 0 0.75 0.17
12 1 1 1
13 0 0.6 0.3
14 0 0.5 0.5
15 0 0.5 0
16 1 0.5 1
17 0 0 0
18 0.5 1 0.5
19 0 1 1
20 0 0 0.57
21 0 0 0.5
22 0.44 0.81 0.65
23 1 1 1
24 0 1 1
25 0.5 1 0.58
26 0.64 0.66 0.82
27 0.38 0.88 0
28 0.5 0.78 0.5
29 1 1 1
30 0.75 1 0.75
31 1 1 1
32 0 1 0.79
33 0 1 1
34 0.17 0.25 0
35 0 1 1
36 0 0.8 0.6
37 0 0 0
38 0.5 1 1
39 0.43 0.93 0.7
40 1 1 0.5
41 0.13 0.76 0.56
42 0.67 0.92 0.42
43 0.57 0.67 0.46
44 0.56 1 0.85
45 1 1 0.67
46 0.5 0.5 0.5
47 0.75 1 1
48 1 1 1
49 1 1 0.5
50 1 1 1
51 0.5 0.54 1
52 0.25 1 0.75
53 0.33 0.85 0.83
54 0.58 1 1
55 0 1 0.36
56 0.5 0.5 0.5
57 1 1 0.23
58 0 1 1
59 0 1 0.46
60 1 1 0.5
61 0 1 1
62 0.33 1 0.83
63 1 0.75 1
64 0 1 0
65 0 0.67 1
66 0.5 0.75 1
67 0 1 0.37
68 1 1 0.63
69 0 1 1
70 1 1 1
71 0 0.42 1
72 1 1 1
73 1 1 0.5
74 1 1 1
75 0.33 0.4 0.67
76 0 1 0.5
77 1 1 0.78
78 1 1 0.2
79 0.5 1 1
80 0.33 1 1
81 0.33 0.35 1
82 0 1 1
83 0 0.71 1
84 0 0.08 0.09
85 0.5 1 0.63
86 1 1 1
87 0 1 0.33
88 0 1 1
89 0.43 0.59 0.87
90 0 1 0.29
91 0 0.58 0.5
92 0.5 1 1
93 0.5 1 1
94 0 1 1
95 0 0.25 0
96 0.5 0 1
97 0 1 1
98 0 1 0.33
99 0.75 0.68 0.78
100 0.33 1 1
101 0 0 0
102 0 0.33 1
103 0.38 0.75 0.46
104 0 1 1
105 0.1 0 0.14
106 0 0 0

download these results as csv

Task 1b Summary Results by Query

Simple Hit/Miss Counting [3]

MRR Method [4]

Task 1c, IOACAS2's dataset Results

As argued by Jang, there are some abnormal in the IOACAS2 data set. Therefore, we just provide the raw result without any further analysis.


Task 1c Summary Results by Query Group

Simple Hit/Miss Count

CSJ1 CSJ2 HAFR
1 0 1 0
2 0 0 1
3 0 1 1
4 0 0 0.33
5 0.27 0.27 0.36
6 0 0 0
7 0.5 0 0.5
8 0 0 0
9 0 0 1
10 1 1 1
11 0 0.5 0.75
12 0 1 1
13 0.14 0.14 0.43
14 0 0 0.4
15 0 0 0
16 1 1 1
17 0 0 1
18 0 1 0
19 0.5 0 0
20 0.25 0 0.5
21 0 0 1
22 0 0 0
23 0 0 0
24 0 0.14 0.14
25 0 0 0
26 0 0.25 1
27 0 1 0
28 0 0 0
29 0.5 0.38 0.63
30 0 0 1
31 0 0 0
32 0 0 1
33 0.33 0 0.33
34 0 1 1
35 0 1 0
36 0 0 1
37 0.11 0.33 0.89
38 0.5 0.5 0.5
39 0 1 1
40 0.5 0.5 1
41 0 1 1
42 0 0 1
43 0 0 1
44 0.5 1 1
45 0 0 0
46 0 0 1
47 0 1 1
48 0 1 0
49 0 0 0
50 0.13 0 0.88
51 0.25 0.5 0.75
52 0.33 0.33 0.67
53 0 0 0.5
54 0 0 0.5
55 0 0 1
56 0.33 0.33 0
57 0 0.33 1
58 0 0 0.29
59 1 1 0
60 0 1 1
61 0 0.5 0.5
62 0 0 0
63 0 0 0
64 0.44 0.33 0.33
65 0 0 0
66 0 0 0
67 0 0 0.5
68 0.6 0.7 0.5
69 0 0 0.33
70 0 0.6 0.3
71 0 0 1
72 0.5 0.33 0.33
73 0 0 0
74 0 0 0
75 0 0 1
76 0 0 0
77 1 1 1
78 0.83 0 0.17
79 0 0 1
80 0.54 0.62 0.85
81 0 0 1
82 0 0 0
83 0.43 0.14 1
84 0 0 0
85 0 0 1
86 0 1 0.25
87 1 1 1
88 0.25 0.75 0.75
89 0 1 1
90 0 0 0
91 0.5 0.5 1
92 0.5 1 1
93 0.4 0.6 0.8
94 0 0 0
95 0 0 1
96 0.33 0.33 1
97 0 0.5 1
98 1 0.5 0.5
99 0 0 0
100 0 0 1
101 0 1 1
102 0.5 0.5 0.5
103 0 1 0
104 1 0 1
105 0 1 0
106 0 0 0
107 0 1 1
108 0 1 1
109 1 0.5 1
110 0 0 1
111 0 0 0
112 0 0 0.5
113 0 0 0
114 1 1 1
115 0 0 0.5
116 0 0 0.5
117 0 0 0
118 1 0 1
119 0 0.2 0.4
120 0 0 1
121 0 0 0
122 1 0 1
123 0 1 1
124 0 0 0
125 0 1 1
126 0 0 0
127 0 0 0
128 0 0 0
129 0 1 1
130 0 0 0
131 1 1 1
132 0 0 0.5
133 1 1 0
134 1 1 1
135 0 0 0
136 0 1 1
137 0 1 1
138 0 1 1
139 1 1 1
140 1 0 1
141 1 1 1
142 0 0.5 1
143 0 0 1
144 0 0 1
145 0.33 0.33 1
146 0 0 0
147 0 1 0
148 0.5 1 0.5
149 1 1 1
150 0.5 0.5 1
151 0 1 1
152 0.33 0.33 1
153 0 0 1
154 0 1 0
155 0 0 1
156 1 0 1
157 1 1 1
158 0 0 0
159 0 0 0
160 0 0 0
161 0 0 0
162 1 1 0
163 0 0 1
164 0 1 0
165 0 1 1
166 0 0 0
167 1 1 1
168 1 1 0
169 1 1 1
170 1 1 1
171 0 0 1
172 1 1 1
173 1 0 1
174 1 0 1
175 0 1 1
176 0 1 1
177 0 0 1
178 1 0.5 1
179 1 0 1
180 0 0 1
181 1 1 1
182 0 0 1
183 0 1 1
184 0 1 0
185 0 1 1
186 0 0.33 0.33
187 0 1 1
188 0 0 1
189 0 1 1
190 0 0.5 1
191 1 1 1
192 0 1 0

download these results as csv

MRR Method

CSJ1 CSJ2 HAFR
1 0 0.17 0
2 0 0 0.58
3 0 1 1
4 0 0 0.33
5 0.15 0.2 0.19
6 0 0 0
7 0.17 0 0.5
8 0 0 0
9 0 0 1
10 1 1 1
11 0 0.28 0.63
12 0 1 1
13 0.02 0.04 0.43
14 0 0 0.4
15 0 0 0
16 1 0.25 1
17 0 0 0.63
18 0 0.17 0
19 0.06 0 0
20 0.03 0 0.5
21 0 0 1
22 0 0 0
23 0 0 0
24 0 0.02 0.01
25 0 0 0
26 0 0.13 0.57
27 0 0.25 0
28 0 0 0
29 0.39 0.26 0.41
30 0 0 1
31 0 0 0
32 0 0 0.75
33 0.33 0 0.17
34 0 0.2 1
35 0 0.14 0
36 0 0 0.75
37 0.01 0.08 0.75
38 0.25 0.25 0.5
39 0 0.13 1
40 0.5 0.5 1
41 0 1 1
42 0 0 1
43 0 0 0.78
44 0.25 0.6 1
45 0 0 0
46 0 0 1
47 0 0.25 0.33
48 0 1 0
49 0 0 0
50 0.03 0 0.66
51 0.03 0.29 0.53
52 0.33 0.17 0.67
53 0 0 0.13
54 0 0 0.5
55 0 0 0.14
56 0.06 0.08 0
57 0 0.08 0.16
58 0 0 0.29
59 1 0.14 0
60 0 1 0.33
61 0 0.25 0.25
62 0 0 0
63 0 0 0
64 0.36 0.22 0.2
65 0 0 0
66 0 0 0
67 0 0 0.07
68 0.55 0.36 0.43
69 0 0 0.33
70 0 0.6 0.3
71 0 0 1
72 0.5 0.13 0.13
73 0 0 0
74 0 0 0
75 0 0 0.6
76 0 0 0
77 1 0.5 1
78 0.42 0 0.02
79 0 0 1
80 0.4 0.43 0.55
81 0 0 0.8
82 0 0 0
83 0.17 0.02 0.81
84 0 0 0
85 0 0 0.5
86 0 0.79 0.13
87 0.5 1 1
88 0.06 0.63 0.75
89 0 1 1
90 0 0 0
91 0.07 0.5 1
92 0.5 1 1
93 0.25 0.43 0.7
94 0 0 0
95 0 0 1
96 0.33 0.17 0.78
97 0 0.5 1
98 0.57 0.5 0.5
99 0 0 0
100 0 0 0.67
101 0 1 0.25
102 0.5 0.5 0.44
103 0 1 0
104 0.2 0 0.14
105 0 1 0
106 0 0 0
107 0 0.67 0.67
108 0 1 1
109 0.63 0.5 0.56
110 0 0 1
111 0 0 0
112 0 0 0.5
113 0 0 0
114 1 1 1
115 0 0 0.5
116 0 0 0.05
117 0 0 0
118 0.5 0 1
119 0 0.2 0.13
120 0 0 1
121 0 0 0
122 0.5 0 0.14
123 0 0.2 1
124 0 0 0
125 0 1 1
126 0 0 0
127 0 0 0
128 0 0 0
129 0 0.1 1
130 0 0 0
131 1 1 1
132 0 0 0.5
133 1 1 0
134 1 1 0.33
135 0 0 0
136 0 1 1
137 0 0.14 0.33
138 0 0.61 1
139 1 1 1
140 0.14 0 1
141 1 0.33 1
142 0 0.12 0.58
143 0 0 0.11
144 0 0 0.5
145 0.04 0.17 1
146 0 0 0
147 0 0.11 0
148 0.5 1 0.5
149 1 1 1
150 0.5 0.25 0.63
151 0 0.33 1
152 0.03 0.33 0.83
153 0 0 1
154 0 0.5 0
155 0 0 1
156 0.2 0 0.33
157 1 1 1
158 0 0 0
159 0 0 0
160 0 0 0
161 0 0 0
162 1 0.5 0
163 0 0 0.25
164 0 0.25 0
165 0 0.2 1
166 0 0 0
167 1 1 1
168 1 0.17 0
169 1 1 1
170 0.2 0.5 1
171 0 0 0.11
172 1 1 0.5
173 0.17 0 1
174 0.2 0 0.25
175 0 0.5 1
176 0 0.13 1
177 0 0 1
178 1 0.1 0.63
179 1 0 0.33
180 0 0 1
181 1 0.5 1
182 0 0 1
183 0 1 0.25
184 0 1 0
185 0 1 1
186 0 0.33 0.03
187 0 1 0.25
188 0 0 1
189 0 1 1
190 0 0.06 1
191 1 0.17 1
192 0 1 0

download these results as csv

Task 1c Summary Results by Query

Simple Hit/Miss Counting [5]

MRR Method [6]

Task 2 Results

Task 2 Overall Results
CSJ1 CSJ2 HAFR
Simple Count 8.48 6.97 6.98
Total count 2040 2040 2020

download these results as csv

Task 2 Friedman's Test for Significant Differences

The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);

TeamID TeamID Lowerbound Mean Upperbound Significance
CSJ1 CSJ2 0.8116 1.2812 1.7509 TRUE
CSJ1 HAFR 0.8741 1.3438 1.8134 TRUE
CSJ2 HAFR -0.4071 0.0625 0.5321 FALSE

download these results as csv

File:SQbshTask2 Friedman Mean Ranks.png ΓÇÄ


Task 2 Summary Results by Query Group
CSJ1 CSJ2 HAFR
1 4.56 3.67 4.11
2 2 1.75 0.25
3 0 0 0
4 3.4 3.4 2.6
5 6.6 6.6 4.8
6 4 2.5 1.5
7 5.8 4.4 2.2
8 5.4 6.2 4
9 2 1.5 2.5
10 4.33 2.56 3
11 8.3 7.56 6.33
12 9.25 7.54 6.93
13 9.22 7.57 7.35
14 8.63 7.79 8.69
15 4.36 3.93 3.93
16 8.43 6.35 7.38
17 8.67 7.15 5.41
18 8.39 6.03 8.14
19 9.22 7.94 7.69
20 9.1 8.12 8.27
21 8.91 6 3.26
22 9.17 7.82 8.1
23 8.29 6.87 4.93
24 8.9 7.61 6.86
25 8.1 7.39 7.45
26 7.52 4.86 6
27 7.93 6.42 4.76
28 8.98 7.93 7.07
29 9.04 8.03 7.31
30 9.42 8.03 9.05
31 9.57 8.7 7.83
32 9.59 8.08 7.84
33 7.59 6.1 6.52
34 9.29 7.51 8.38
35 8.74 6.43 7.5
36 7.71 5.79 4.33
37 7.43 5.21 6.25
38 8.28 7.11 7.7
39 6.97 5.91 7.19
40 8.81 6.96 6.44
41 9.15 7.14 7.59
42 6.86 6.12 5.71
43 8.2 5.88 6.68
44 9.19 8.11 6.56
45 7.11 5.13 5.09
46 7.66 5.42 6.7
47 7.92 5.69 3.38
48 8.18 6 7.59

download these results as csv

Task 2 Summary Results by Query

[7]

Runtime Results

Participant Task Runtime (min) Machine
CSJ1 1a ~55 BIGWIN
CSJ1 1b 7 BIGWIN
CSJ1 2 ~740 BIGWIN
CSJ2 1a ~1800 BIGWIN
CSJ2 1b ~210 BIGWIN
CSJ2 2 ~730 BIGWIN
HAFR 1a 1689 BEER 2
HAFR 1b 247 BEER 2
HAFR 2 242 BEER 2

download these results as csv

Retrieved from "https://www.music-ir.org/mirex/w/index.php?title=2009:Query-by-Singing/Humming_Results&oldid=6571"