Difference between revisions of "2009:Query-by-Tapping Results"
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-\[\[([A-Z][^:]+)\]\] +2009:\1)) |
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-<csv([^>]*)> +<csv\1>2009/)) |
||
Line 24: | Line 24: | ||
=====Task 1a Overall Results===== | =====Task 1a Overall Results===== | ||
− | <csv>qbt/QbtFinalTask1a.csv</csv> | + | <csv>2009/qbt/QbtFinalTask1a.csv</csv> |
====Task 1a Friedman's Test for Significant Differences==== | ====Task 1a Friedman's Test for Significant Differences==== | ||
Line 31: | Line 31: | ||
Simple Hit/Miss Count: | Simple Hit/Miss Count: | ||
− | <csv>qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbt/friedman/QbtTask1aSimple_friedman_tukeyKramerHSD.csv</csv> |
[[Image:SQbtTask1aSimple_Friedman_Mean_Ranks.pngΓÇÄ]] | [[Image:SQbtTask1aSimple_Friedman_Mean_Ranks.pngΓÇÄ]] | ||
MRR Method: | MRR Method: | ||
− | <csv>qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbt/friedman/Qbt1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv> |
[[Image:SQbt1aTask2Mrr_Friedman_Mean_Ranks.pngΓÇÄ]] | [[Image:SQbt1aTask2Mrr_Friedman_Mean_Ranks.pngΓÇÄ]] | ||
Line 42: | Line 42: | ||
====Task 1a Summary Results by Query Group==== | ====Task 1a Summary Results by Query Group==== | ||
Simple Hit/Miss Counting | Simple Hit/Miss Counting | ||
− | <csv p=2>qbt/QbtTask1aSimpleByGroup.csv</csv> | + | <csv p=2>2009/qbt/QbtTask1aSimpleByGroup.csv</csv> |
MRR Method | MRR Method | ||
− | <csv p=2>qbt/QbtTask1aMrrByGroup.csv</csv> | + | <csv p=2>2009/qbt/QbtTask1aMrrByGroup.csv</csv> |
====Task 1a Summary Results by Query ==== | ====Task 1a Summary Results by Query ==== | ||
Line 58: | Line 58: | ||
=====Task 1b Overall Results===== | =====Task 1b Overall Results===== | ||
− | <csv p=2>qbt/QbtFinalTask1b.csv</csv> | + | <csv p=2>2009/qbt/QbtFinalTask1b.csv</csv> |
====Task 1b Friedman's Test for Significant Differences==== | ====Task 1b Friedman's Test for Significant Differences==== | ||
Line 65: | Line 65: | ||
Simple Hit/Miss Count: | Simple Hit/Miss Count: | ||
− | <csv>qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbt/friedman/QbtTask1bSimple_friedman_tukeyKramerHSD.csv</csv> |
[[Image:SQbtTask1bSimple_Friedman_Mean_Ranks.pngΓÇÄ]] | [[Image:SQbtTask1bSimple_Friedman_Mean_Ranks.pngΓÇÄ]] | ||
MRR Method: | MRR Method: | ||
− | <csv>qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbt/friedman/QbtTask1bMrr_friedman_tukeyKramerHSD.csv</csv> |
[[Image:SQbtTask1bMrr_Friedman_Mean_Ranks.png ]] | [[Image:SQbtTask1bMrr_Friedman_Mean_Ranks.png ]] | ||
Line 76: | Line 76: | ||
====Task 1b Summary Results by Query Group==== | ====Task 1b Summary Results by Query Group==== | ||
Simple Hit/Miss Counting | Simple Hit/Miss Counting | ||
− | <csv p=2>qbt/QbtTask1bSimpleByGroup.csv</csv> | + | <csv p=2>2009/qbt/QbtTask1bSimpleByGroup.csv</csv> |
MRR Method | MRR Method | ||
− | <csv p=2>qbt/QbtTask1bMrrByGroup.csv</csv> | + | <csv p=2>2009/qbt/QbtTask1bMrrByGroup.csv</csv> |
====Task 1b Summary Results by Query ==== | ====Task 1b Summary Results by Query ==== | ||
Line 92: | Line 92: | ||
=====Task 2 Overall Results===== | =====Task 2 Overall Results===== | ||
− | <csv>qbt/QbtFinalTask2.csv</csv> | + | <csv>2009/qbt/QbtFinalTask2.csv</csv> |
====Task 2 Friedman's Test for Significant Differences==== | ====Task 2 Friedman's Test for Significant Differences==== | ||
Line 99: | Line 99: | ||
Simple Hit/Miss Count: | Simple Hit/Miss Count: | ||
− | <csv>qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbt/friedman/QbtTask2Simple_friedman_tukeyKramerHSD.csv</csv> |
[[Image:SQbtTask2Simple_Friedman_Mean_Ranks.pngΓÇÄ]] | [[Image:SQbtTask2Simple_Friedman_Mean_Ranks.pngΓÇÄ]] | ||
MRR Method: | MRR Method: | ||
− | <csv>qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbt/friedman/QbtTask2Mrr_friedman_tukeyKramerHSD.csv</csv> |
[[Image:SQbtTask2Mrr_Friedman_Mean_Ranks.png ]] | [[Image:SQbtTask2Mrr_Friedman_Mean_Ranks.png ]] | ||
Line 110: | Line 110: | ||
====Task 2 Summary Results by Query Group==== | ====Task 2 Summary Results by Query Group==== | ||
Simple Hit/Miss Counting | Simple Hit/Miss Counting | ||
− | <csv p=2>qbt/QbtTask2SimpleByGroup.csv</csv> | + | <csv p=2>2009/qbt/QbtTask2SimpleByGroup.csv</csv> |
MRR Method | MRR Method | ||
− | <csv p=2>qbt/QbtTask2MrrByGroup.csv</csv> | + | <csv p=2>2009/qbt/QbtTask2MrrByGroup.csv</csv> |
====Task 2 Summary Results by Query ==== | ====Task 2 Summary Results by Query ==== | ||
Line 124: | Line 124: | ||
===Runtime Results=== | ===Runtime Results=== | ||
− | <csv>qbt/qbtRunTime.csv</csv> | + | <csv>2009/qbt/qbtRunTime.csv</csv> |
[[Category: Results]] | [[Category: Results]] |
Revision as of 19:59, 13 May 2010
Contents
Introduction
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2009:Query by Singing/Humming page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
- Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
- Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.
Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.
General Legend
Team ID
CSJ = Chun-Ta Chen and Jyh-Shing Roger Jang
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine
HL = Shu-Jen Show Hsiao,Tyne Liang
Task 1 Results
Task 1a, Jang's Dataset Results
Task 1a Overall Results
CSJ | HAFR | HL | |
---|---|---|---|
Simple Count | 0.87 | 0.88 | 0.89 |
MRR | 0.78 | 0.77 | 0.79 |
Total count | 890 | 890 | 890 |
Task 1a Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
HL | CSJ | -0.1863 | -0.0184 | 0.1496 | FALSE |
HL | HAFR | -0.1716 | -0.0037 | 0.1643 | FALSE |
CSJ | HAFR | -0.1533 | 0.0147 | 0.1827 | FALSE |
File:SQbtTask1aSimple Friedman Mean Ranks.pngΓÇÄ
MRR Method:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
HL | CSJ | -0.2337 | -0.0074 | 0.2190 | FALSE |
HL | HAFR | -0.0646 | 0.1618 | 0.3881 | FALSE |
CSJ | HAFR | -0.0572 | 0.1691 | 0.3955 | FALSE |
File:SQbt1aTask2Mrr Friedman Mean Ranks.pngΓÇÄ
Task 1a Summary Results by Query Group
Simple Hit/Miss Counting
CSJ | HAFR | HL | |
---|---|---|---|
1 | 1 | 0.92 | 0.92 |
2 | 1 | 1 | 1 |
3 | 1 | 1 | 1 |
4 | 1 | 0 | 1 |
5 | 1 | 1 | 1 |
6 | 1 | 1 | 0.50 |
7 | 1 | 1 | 1 |
8 | 1 | 1 | 1 |
9 | 1 | 1 | 1 |
10 | 1 | 1 | 1 |
11 | 0.50 | 1 | 0.50 |
12 | 1 | 1 | 1 |
13 | 1 | 1 | 1 |
14 | 1 | 1 | 1 |
15 | 1 | 1 | 1 |
16 | 1 | 1 | 0.95 |
17 | 0.87 | 0.93 | 1 |
18 | 0.77 | 0.85 | 0.69 |
19 | 0.38 | 0.50 | 1 |
20 | 1 | 1 | 1 |
21 | 1 | 1 | 1 |
22 | 1 | 0.50 | 0.50 |
23 | 0 | 0 | 0 |
24 | 1 | 1 | 1 |
25 | 1 | 1 | 1 |
26 | 0 | 0 | 1 |
27 | 1 | 0.94 | 1 |
28 | 1 | 1 | 1 |
29 | 1 | 1 | 1 |
30 | 0.58 | 0.75 | 0.67 |
31 | 1 | 1 | 1 |
32 | 1 | 1 | 0.67 |
33 | 0.78 | 0.75 | 0.94 |
34 | 0.92 | 0.92 | 0.92 |
35 | 1 | 1 | 1 |
36 | 1 | 1 | 1 |
37 | 0 | 1 | 1 |
38 | 1 | 1 | 1 |
39 | 1 | 1 | 1 |
40 | 1 | 1 | 1 |
41 | 0.83 | 0.75 | 0.92 |
42 | 1 | 1 | 1 |
43 | 0.80 | 1 | 1 |
44 | 0.75 | 0.75 | 0.50 |
45 | 1 | 1 | 0.50 |
46 | 1 | 1 | 1 |
47 | 1 | 1 | 1 |
48 | 1 | 1 | 1 |
49 | 1 | 1 | 1 |
50 | 1 | 1 | 1 |
51 | 0.60 | 0.80 | 0.60 |
52 | 1 | 1 | 1 |
53 | 0.80 | 0.60 | 0.60 |
54 | 1 | 1 | 1 |
55 | 1 | 0.98 | 1 |
56 | 1 | 1 | 1 |
57 | 1 | 0.80 | 1 |
58 | 1 | 1 | 1 |
59 | 0.20 | 0.60 | 0.20 |
60 | 0.43 | 1 | 0.71 |
61 | 1 | 1 | 1 |
62 | 1 | 1 | 1 |
63 | 1 | 1 | 1 |
64 | 0.50 | 0.50 | 0.50 |
65 | 1 | 1 | 1 |
66 | 1 | 1 | 0.50 |
67 | 1 | 1 | 1 |
68 | 0.67 | 0.78 | 0.61 |
69 | 0.82 | 0.82 | 0.94 |
70 | 1 | 1 | 0.75 |
71 | 1 | 0 | 1 |
72 | 0.83 | 0.92 | 0.75 |
73 | 0.93 | 0.86 | 0.93 |
74 | 1 | 1 | 1 |
75 | 0.83 | 1 | 0.83 |
76 | 1 | 1 | 1 |
77 | 0 | 0 | 1 |
78 | 1 | 0 | 0 |
79 | 1 | 0.50 | 0.75 |
80 | 1 | 1 | 1 |
81 | 1 | 1 | 1 |
82 | 1 | 1 | 1 |
83 | 1 | 1 | 1 |
84 | 1 | 1 | 1 |
85 | 0.86 | 0.86 | 0.86 |
86 | 1 | 1 | 1 |
87 | 0.58 | 0.75 | 0.75 |
88 | 1 | 1 | 1 |
89 | 1 | 0.67 | 1 |
90 | 0.50 | 1 | 1 |
91 | 1 | 1 | 1 |
92 | 0.25 | 0.25 | 0.50 |
93 | 1 | 1 | 1 |
94 | 0.50 | 0.67 | 0.67 |
95 | 1 | 1 | 1 |
96 | 1 | 1 | 1 |
97 | 1 | 1 | 1 |
98 | 0.50 | 0.50 | 0.50 |
99 | 0.88 | 0.88 | 0.81 |
100 | 1 | 1 | 1 |
101 | 0.83 | 1 | 0.83 |
102 | 0.88 | 1 | 1 |
103 | 1 | 1 | 1 |
104 | 1 | 1 | 1 |
105 | 1 | 1 | 0.96 |
106 | 0.73 | 0.68 | 0.50 |
107 | 0.94 | 0.94 | 1 |
108 | 0.69 | 0.56 | 0.88 |
109 | 0.94 | 0.81 | 0.94 |
110 | 1 | 1 | 1 |
111 | 1 | 1 | 0.78 |
112 | 1 | 1 | 1 |
113 | 1 | 1 | 1 |
114 | 1 | 1 | 1 |
115 | 0.80 | 0.60 | 0.60 |
116 | 0.75 | 0.75 | 0.75 |
117 | 1 | 1 | 1 |
118 | 1 | 1 | 1 |
119 | 0.67 | 0.67 | 0.67 |
120 | 0.67 | 0.67 | 1 |
121 | 0.33 | 0.67 | 0.33 |
122 | 0.33 | 0.67 | 0.67 |
123 | 0.67 | 0.33 | 0.33 |
124 | 0.33 | 0.33 | 0.33 |
125 | 1 | 0.33 | 0.67 |
126 | 0.67 | 0.67 | 1 |
127 | 1 | 1 | 1 |
128 | 0.67 | 0.67 | 0.67 |
129 | 0.50 | 0.50 | 0 |
130 | 1 | 0.50 | 1 |
131 | 1 | 1 | 1 |
132 | 1 | 1 | 1 |
133 | 1 | 1 | 1 |
134 | 1 | 1 | 1 |
135 | 1 | 1 | 1 |
136 | 1 | 1 | 1 |
MRR Method
CSJ | HAFR | HL | |
---|---|---|---|
1 | 0.94 | 0.88 | 0.88 |
2 | 1 | 1 | 1 |
3 | 1 | 1 | 1 |
4 | 1 | 0 | 1 |
5 | 1 | 1 | 1 |
6 | 0.78 | 0.88 | 0.50 |
7 | 1 | 1 | 1 |
8 | 0.75 | 0.75 | 0.75 |
9 | 1 | 0.25 | 1 |
10 | 1 | 1 | 1 |
11 | 0.50 | 0.60 | 0.50 |
12 | 1 | 1 | 1 |
13 | 1 | 1 | 0.50 |
14 | 1 | 0.94 | 0.94 |
15 | 1 | 1 | 1 |
16 | 0.93 | 0.82 | 0.35 |
17 | 0.76 | 0.93 | 1 |
18 | 0.71 | 0.64 | 0.59 |
19 | 0.28 | 0.30 | 0.50 |
20 | 1 | 1 | 1 |
21 | 0.56 | 1 | 1 |
22 | 1 | 0.50 | 0.50 |
23 | 0 | 0 | 0 |
24 | 1 | 0.50 | 1 |
25 | 0.90 | 0.90 | 0.89 |
26 | 0 | 0 | 0.14 |
27 | 0.97 | 0.76 | 0.88 |
28 | 0.83 | 1 | 1 |
29 | 1 | 1 | 1 |
30 | 0.52 | 0.75 | 0.53 |
31 | 1 | 0.33 | 1 |
32 | 1 | 1 | 0.67 |
33 | 0.50 | 0.72 | 0.91 |
34 | 0.92 | 0.92 | 0.92 |
35 | 1 | 1 | 1 |
36 | 0.88 | 0.78 | 0.88 |
37 | 0 | 1 | 0.25 |
38 | 1 | 1 | 1 |
39 | 1 | 1 | 1 |
40 | 0.96 | 0.96 | 1 |
41 | 0.63 | 0.58 | 0.60 |
42 | 1 | 1 | 1 |
43 | 0.80 | 0.88 | 0.84 |
44 | 0.75 | 0.63 | 0.50 |
45 | 0.60 | 0.42 | 0.17 |
46 | 1 | 0.71 | 0.92 |
47 | 1 | 0.57 | 1 |
48 | 0.41 | 1 | 0.78 |
49 | 1 | 1 | 0.75 |
50 | 0.73 | 0.44 | 0.78 |
51 | 0.33 | 0.63 | 0.27 |
52 | 1 | 1 | 1 |
53 | 0.67 | 0.60 | 0.60 |
54 | 1 | 0.50 | 1 |
55 | 0.98 | 0.96 | 0.98 |
56 | 1 | 1 | 1 |
57 | 0.90 | 0.80 | 1 |
58 | 1 | 1 | 0.89 |
59 | 0.03 | 0.43 | 0.20 |
60 | 0.43 | 0.59 | 0.52 |
61 | 1 | 1 | 1 |
62 | 1 | 0.96 | 0.94 |
63 | 0.33 | 0.11 | 1 |
64 | 0.38 | 0.28 | 0.38 |
65 | 1 | 1 | 1 |
66 | 1 | 1 | 0.50 |
67 | 1 | 1 | 0.50 |
68 | 0.56 | 0.55 | 0.56 |
69 | 0.77 | 0.67 | 0.77 |
70 | 1 | 0.78 | 0.50 |
71 | 1 | 0 | 0.50 |
72 | 0.69 | 0.82 | 0.68 |
73 | 0.87 | 0.82 | 0.89 |
74 | 1 | 0.80 | 1 |
75 | 0.71 | 0.85 | 0.83 |
76 | 1 | 1 | 1 |
77 | 0 | 0 | 0.12 |
78 | 1 | 0 | 0 |
79 | 0.65 | 0.38 | 0.75 |
80 | 0.73 | 1 | 1 |
81 | 1 | 1 | 1 |
82 | 1 | 1 | 1 |
83 | 1 | 0.88 | 1 |
84 | 1 | 0.50 | 1 |
85 | 0.63 | 0.16 | 0.86 |
86 | 1 | 1 | 1 |
87 | 0.33 | 0.69 | 0.45 |
88 | 1 | 1 | 1 |
89 | 0.83 | 0.42 | 1 |
90 | 0.25 | 1 | 1 |
91 | 1 | 1 | 1 |
92 | 0.25 | 0.13 | 0.13 |
93 | 1 | 1 | 1 |
94 | 0.42 | 0.29 | 0.36 |
95 | 0.91 | 0.94 | 1 |
96 | 1 | 1 | 1 |
97 | 1 | 1 | 0.88 |
98 | 0.50 | 0.50 | 0.50 |
99 | 0.88 | 0.79 | 0.70 |
100 | 1 | 1 | 1 |
101 | 0.83 | 0.57 | 0.83 |
102 | 0.88 | 1 | 1 |
103 | 1 | 1 | 1 |
104 | 0.98 | 1 | 1 |
105 | 0.94 | 0.94 | 0.96 |
106 | 0.60 | 0.59 | 0.48 |
107 | 0.73 | 0.68 | 0.82 |
108 | 0.48 | 0.46 | 0.69 |
109 | 0.90 | 0.65 | 0.89 |
110 | 1 | 0.97 | 1 |
111 | 0.89 | 0.82 | 0.78 |
112 | 0.78 | 0.94 | 0.94 |
113 | 1 | 1 | 0.90 |
114 | 0.80 | 0.80 | 0.90 |
115 | 0.48 | 0.31 | 0.25 |
116 | 0.53 | 0.54 | 0.63 |
117 | 1 | 1 | 1 |
118 | 0.47 | 1 | 1 |
119 | 0.38 | 0.67 | 0.67 |
120 | 0.67 | 0.67 | 1 |
121 | 0.33 | 0.40 | 0.17 |
122 | 0.08 | 0.50 | 0.39 |
123 | 0.38 | 0.33 | 0.33 |
124 | 0.33 | 0.33 | 0.33 |
125 | 0.70 | 0.17 | 0.67 |
126 | 0.23 | 0.67 | 0.67 |
127 | 1 | 1 | 1 |
128 | 0.67 | 0.67 | 0.67 |
129 | 0.50 | 0.25 | 0 |
130 | 0.58 | 0.50 | 1 |
131 | 0.67 | 1 | 1 |
132 | 0.75 | 1 | 0.67 |
133 | 0.17 | 0.14 | 0.20 |
134 | 1 | 1 | 1 |
135 | 1 | 1 | 1 |
136 | 0.50 | 1 | 1 |
Task 1a Summary Results by Query
Simple Hit/Miss Counting [1]
MRR Method [2]
Task 1b, Hsiao's Dataset Results
Task 1b Overall Results
CSJ | HAFR | HL | |
---|---|---|---|
Simple Count | 0.82 | 0.88 | 0.90 |
MRR | 0.66 | 0.70 | 0.69 |
Total count | 410.00 | 410.00 | 410.00 |
Task 1b Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
HL | HAFR | -0.1073 | 0.0352 | 0.1776 | FALSE |
HL | CSJ | 0.0451 | 0.1875 | 0.3299 | TRUE |
HAFR | CSJ | 0.0099 | 0.1523 | 0.2948 | TRUE |
File:SQbtTask1bSimple Friedman Mean Ranks.pngΓÇÄ
MRR Method:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
HAFR | HL | -0.1002 | 0.1328 | 0.3659 | FALSE |
HAFR | CSJ | -0.0260 | 0.2070 | 0.4401 | FALSE |
HL | CSJ | -0.1588 | 0.0742 | 0.3073 | FALSE |
File:SQbtTask1bMrr Friedman Mean Ranks.png
Task 1b Summary Results by Query Group
Simple Hit/Miss Counting
CSJ | HAFR | HL | |
---|---|---|---|
1 | 1 | 1 | 1 |
2 | NaN | NaN | NaN |
3 | 1 | 1 | 1 |
4 | 1 | 1 | 1 |
5 | 1 | 1 | 1 |
6 | 1 | 1 | 1 |
7 | 1 | 1 | 1 |
8 | 1 | 1 | 1 |
9 | 1 | 1 | 1 |
10 | 0.67 | 0.67 | 0.67 |
11 | 1 | 1 | 1 |
12 | 0.75 | 1 | 0.50 |
13 | 1 | 1 | 1 |
14 | 1 | 1 | 1 |
15 | NaN | NaN | NaN |
16 | 1 | 1 | 1 |
17 | 1 | 1 | 1 |
18 | 1 | 1 | 1 |
19 | 1 | 1 | 1 |
20 | 1 | 1 | 1 |
21 | 0.67 | 1 | 1 |
22 | 1 | 1 | 1 |
23 | 1 | 0.83 | 0.83 |
24 | 0.67 | 0.67 | 0.67 |
25 | 0 | 1 | 1 |
26 | 0 | 0 | 0 |
27 | 1 | 1 | 1 |
28 | NaN | NaN | NaN |
29 | 1 | 1 | 1 |
30 | 1 | 1 | 1 |
31 | NaN | NaN | NaN |
32 | 1 | 1 | 1 |
33 | 1 | 1 | 1 |
34 | 1 | 1 | 1 |
35 | NaN | NaN | NaN |
36 | 0 | 0 | 0 |
37 | 0.50 | 0 | 0.50 |
38 | 1 | 0.86 | 1 |
39 | 1 | 1 | 1 |
40 | 0 | 0.50 | 0.50 |
41 | 1 | 1 | 1 |
42 | 0.67 | 1 | 1 |
43 | 1 | 1 | 1 |
44 | 0 | 1 | 0.50 |
45 | 1 | 1 | 1 |
46 | 1 | 1 | 1 |
47 | 1 | 1 | 1 |
48 | 1 | 1 | 1 |
49 | 1 | 0.67 | 1 |
50 | 1 | 1 | 1 |
51 | 1 | 1 | 1 |
52 | 1 | 1 | 1 |
53 | 0.89 | 0.89 | 0.89 |
54 | 0.33 | 1 | 1 |
55 | 0.50 | 1 | 1 |
56 | 1 | 0.50 | 1 |
57 | 1 | 1 | 1 |
58 | NaN | NaN | NaN |
59 | 1 | 1 | 1 |
60 | 1 | 1 | 1 |
61 | 0.75 | 0.75 | 1 |
62 | 0 | 0.50 | 0.50 |
63 | 1 | 1 | 1 |
64 | NaN | NaN | NaN |
65 | 1 | 1 | 1 |
66 | NaN | NaN | NaN |
67 | 1 | 1 | 1 |
68 | NaN | NaN | NaN |
69 | 0.40 | 1 | 1 |
70 | 0.20 | 1 | 1 |
71 | NaN | NaN | NaN |
72 | 1 | 1 | 1 |
73 | 1 | 1 | 0.50 |
74 | NaN | NaN | NaN |
75 | 1 | 1 | 1 |
76 | 0.40 | 1 | 0.40 |
77 | NaN | NaN | NaN |
78 | 1 | 1 | 1 |
79 | 0.50 | 1 | 1 |
80 | 1 | 1 | 0 |
81 | 0 | 1 | 1 |
82 | 0 | 1 | 1 |
83 | 1 | 1 | 1 |
84 | 1 | 1 | 1 |
85 | 0 | 0 | 1 |
86 | 1 | 1 | 1 |
87 | 1 | 1 | 1 |
88 | 0.80 | 0.60 | 0.80 |
89 | 0.80 | 1 | 0.20 |
90 | 1 | 1 | 1 |
91 | 1 | 1 | 1 |
92 | 1 | 1 | 0.80 |
93 | 0.50 | 1 | 1 |
94 | 1 | 0.75 | 1 |
95 | 0.67 | 0.67 | 0.67 |
96 | 1 | 1 | 1 |
97 | 0 | 0 | 0 |
98 | 1 | 1 | 1 |
99 | 1 | 1 | 1 |
100 | 1 | 1 | 1 |
101 | 1 | 1 | 1 |
102 | 1 | 1 | 1 |
103 | NaN | NaN | NaN |
104 | 1 | 1 | 1 |
105 | 1 | 1 | 1 |
106 | 0 | 0.33 | 0 |
107 | NaN | NaN | NaN |
108 | 1 | 1 | 1 |
109 | 1 | 1 | 1 |
110 | 0 | 1 | 1 |
111 | 1 | 1 | 1 |
112 | 0.67 | 1 | 0.67 |
113 | 1 | 1 | 1 |
114 | 0.50 | 0.50 | 0.50 |
115 | NaN | NaN | NaN |
116 | 0 | 0 | 1 |
117 | 1 | 1 | 1 |
118 | 1 | 1 | 1 |
119 | 0 | 0 | 0 |
120 | 0.67 | 0.67 | 0.67 |
121 | 1 | 0.67 | 1 |
122 | 1 | 1 | 1 |
123 | 1 | 1 | 1 |
124 | 1 | 1 | 1 |
125 | 1 | 0 | 1 |
126 | 1 | 1 | 1 |
127 | 1 | 1 | 1 |
128 | 1 | 1 | 1 |
129 | 1 | 1 | 1 |
130 | 0 | 0 | 0 |
131 | 0 | 0 | 1 |
132 | 1 | 1 | 1 |
133 | 0.33 | 1 | 0.67 |
134 | 0.50 | 0.50 | 0.50 |
135 | 1 | 1 | 1 |
136 | 1 | 1 | 1 |
137 | 1 | 1 | 1 |
138 | 1 | 1 | 1 |
139 | 0.60 | 1 | 1 |
140 | 0 | 0 | 1 |
141 | 1 | 1 | 1 |
142 | 1 | 1 | 1 |
143 | 1 | 1 | 1 |
MRR Method
CSJ | HAFR | HL | |
---|---|---|---|
1 | 0.56 | 1 | 1 |
2 | NaN | NaN | NaN |
3 | 0.88 | 0.83 | 0.83 |
4 | 0.50 | 0.75 | 0.50 |
5 | 1 | 1 | 1 |
6 | 1 | 0.50 | 1 |
7 | 1 | 1 | 1 |
8 | 1 | 1 | 1 |
9 | 1 | 1 | 1 |
10 | 0.44 | 0.50 | 0.67 |
11 | 1 | 1 | 1 |
12 | 0.75 | 0.81 | 0.50 |
13 | 1 | 1 | 1 |
14 | 1 | 0.42 | 1 |
15 | NaN | NaN | NaN |
16 | 1 | 1 | 1 |
17 | 1 | 1 | 1 |
18 | 1 | 1 | 1 |
19 | 1 | 1 | 1 |
20 | 0.83 | 1 | 0.61 |
21 | 0.45 | 1 | 1 |
22 | 1 | 1 | 0.88 |
23 | 0.88 | 0.47 | 0.64 |
24 | 0.67 | 0.67 | 0.67 |
25 | 0 | 0.75 | 0.15 |
26 | 0 | 0 | 0 |
27 | 0.50 | 0.67 | 0.44 |
28 | NaN | NaN | NaN |
29 | 1 | 1 | 0.75 |
30 | 1 | 1 | 0.75 |
31 | NaN | NaN | NaN |
32 | 1 | 0.50 | 1 |
33 | 0.33 | 0.56 | 0.50 |
34 | 0.78 | 0.61 | 0.58 |
35 | NaN | NaN | NaN |
36 | 0 | 0 | 0 |
37 | 0.13 | 0 | 0.25 |
38 | 0.89 | 0.26 | 0.90 |
39 | 0.18 | 1 | 1 |
40 | 0 | 0.07 | 0.08 |
41 | 1 | 1 | 0.50 |
42 | 0.50 | 0.83 | 0.75 |
43 | 0.75 | 0.75 | 0.88 |
44 | 0 | 0.42 | 0.10 |
45 | 0.33 | 0.56 | 0.22 |
46 | 0.78 | 1 | 1 |
47 | 1 | 1 | 0.75 |
48 | 1 | 1 | 1 |
49 | 0.53 | 0.14 | 0.28 |
50 | 1 | 1 | 1 |
51 | 0.10 | 0.13 | 0.17 |
52 | 1 | 1 | 1 |
53 | 0.89 | 0.83 | 0.89 |
54 | 0.11 | 0.37 | 0.32 |
55 | 0.06 | 1 | 0.75 |
56 | 0.42 | 0.25 | 0.67 |
57 | 0.80 | 0.90 | 0.60 |
58 | NaN | NaN | NaN |
59 | 1 | 0.50 | 0.50 |
60 | 1 | 0.63 | 1 |
61 | 0.50 | 0.75 | 0.54 |
62 | 0 | 0.50 | 0.50 |
63 | 1 | 0.67 | 0.50 |
64 | NaN | NaN | NaN |
65 | 0.69 | 0.49 | 1 |
66 | NaN | NaN | NaN |
67 | 0.58 | 0.20 | 0.67 |
68 | NaN | NaN | NaN |
69 | 0.05 | 0.85 | 1 |
70 | 0.20 | 0.41 | 0.90 |
71 | NaN | NaN | NaN |
72 | 1 | 1 | 0.60 |
73 | 1 | 1 | 0.50 |
74 | NaN | NaN | NaN |
75 | 1 | 1 | 1 |
76 | 0.20 | 0.85 | 0.30 |
77 | NaN | NaN | NaN |
78 | 0.67 | 0.83 | 0.67 |
79 | 0.13 | 0.58 | 0.88 |
80 | 0.13 | 0.33 | 0 |
81 | 0 | 0.35 | 0.29 |
82 | 0 | 0.50 | 0.25 |
83 | 1 | 1 | 1 |
84 | 1 | 0.63 | 1 |
85 | 0 | 0 | 0.25 |
86 | 1 | 1 | 1 |
87 | 1 | 1 | 1 |
88 | 0.67 | 0.43 | 0.47 |
89 | 0.38 | 0.69 | 0.02 |
90 | 0.83 | 0.78 | 0.53 |
91 | 0.88 | 1 | 0.88 |
92 | 1 | 1 | 0.80 |
93 | 0.50 | 1 | 1 |
94 | 0.33 | 0.16 | 0.29 |
95 | 0.67 | 0.67 | 0.67 |
96 | 1 | 1 | 1 |
97 | 0 | 0 | 0 |
98 | 1 | 0.75 | 1 |
99 | 1 | 1 | 1 |
100 | 1 | 1 | 1 |
101 | 1 | 1 | 0.80 |
102 | 1 | 1 | 1 |
103 | NaN | NaN | NaN |
104 | 0.33 | 1 | 1 |
105 | 0.35 | 0.33 | 0.40 |
106 | 0 | 0.11 | 0 |
107 | NaN | NaN | NaN |
108 | 1 | 1 | 1 |
109 | 0.33 | 0.33 | 0.33 |
110 | 0 | 0.10 | 0.17 |
111 | 0.50 | 0.50 | 0.13 |
112 | 0.50 | 0.78 | 0.20 |
113 | 1 | 1 | 1 |
114 | 0.50 | 0.17 | 0.50 |
115 | NaN | NaN | NaN |
116 | 0 | 0 | 0.60 |
117 | 0.75 | 0.75 | 0.75 |
118 | 0.83 | 0.67 | 0.33 |
119 | 0 | 0 | 0 |
120 | 0.21 | 0.50 | 0.20 |
121 | 0.56 | 0.67 | 0.71 |
122 | 0.65 | 0.88 | 0.43 |
123 | 0.79 | 0.78 | 1 |
124 | 0.70 | 0.70 | 0.70 |
125 | 0.19 | 0 | 0.23 |
126 | 1 | 1 | 1 |
127 | 0.50 | 0.88 | 0.50 |
128 | 0.56 | 0.78 | 0.50 |
129 | 1 | 1 | 1 |
130 | 0 | 0 | 0 |
131 | 0 | 0 | 0.29 |
132 | 1 | 0.75 | 0.63 |
133 | 0.03 | 0.48 | 0.15 |
134 | 0.50 | 0.06 | 0.07 |
135 | 1 | 1 | 0.78 |
136 | 0.56 | 1 | 1 |
137 | 1 | 1 | 1 |
138 | 0.23 | 0.38 | 0.33 |
139 | 0.35 | 0.44 | 0.80 |
140 | 0 | 0 | 0.50 |
141 | 0.56 | 0.69 | 0.42 |
142 | 0.50 | 0.50 | 0.39 |
143 | 1 | 1 | 1 |
Task 1b Summary Results by Query
Simple Hit/Miss Counting [3]
MRR Method [4]
Task 2 Results
Task 2 Overall Results
CSJ | HAFR | HL | |
---|---|---|---|
Simple Count | 0.86 | 0.79 | 0.84 |
MRR | 0.77 | 0.66 | 0.73 |
Total count | 890 | 890 | 880 |
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ | HL | -0.1404 | 0.0515 | 0.2434 | FALSE |
CSJ | HAFR | 0.0103 | 0.2022 | 0.3941 | TRUE |
HL | HAFR | -0.0412 | 0.1507 | 0.3426 | FALSE |
File:SQbtTask2Simple Friedman Mean Ranks.pngΓÇÄ
MRR Method:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ | HL | -0.1350 | 0.1066 | 0.3482 | FALSE |
CSJ | HAFR | 0.1702 | 0.4118 | 0.6533 | TRUE |
HL | HAFR | 0.0636 | 0.3051 | 0.5467 | TRUE |
File:SQbtTask2Mrr Friedman Mean Ranks.png
Task 2 Summary Results by Query Group
Simple Hit/Miss Counting
CSJ | HAFR | HL | |
---|---|---|---|
1 | 1 | 0.69 | 0.77 |
2 | 1 | 1 | 1 |
3 | 1 | 1 | 1 |
4 | 1 | 0 | 1 |
5 | 1 | 1 | 1 |
6 | 1 | 1 | 0.75 |
7 | 1 | 1 | 1 |
8 | 1 | 1 | 1 |
9 | 1 | 1 | 1 |
10 | 1 | 1 | 1 |
11 | 0.50 | 1 | 1 |
12 | 1 | 1 | 1 |
13 | 1 | 1 | 1 |
14 | 1 | 1 | 1 |
15 | 1 | 1 | 1 |
16 | 1 | 0.90 | 0.85 |
17 | 0.80 | 0.93 | 1 |
18 | 0.77 | 0.62 | 0.69 |
19 | 0.38 | 0.75 | 1 |
20 | 1 | 1 | 1 |
21 | 1 | 1 | 1 |
22 | 1 | 1 | 0.50 |
23 | 0 | 0 | 0 |
24 | 1 | 1 | 1 |
25 | 1 | 1 | 1 |
26 | 0 | 0 | 1 |
27 | 1 | 0.71 | 0.88 |
28 | 1 | 0.67 | 1 |
29 | 1 | 1 | 1 |
30 | 0.67 | 0.67 | 0.75 |
31 | 1 | 0 | 1 |
32 | 1 | 1 | 1 |
33 | 0.71 | 0.59 | 0.82 |
34 | 0.83 | 0.92 | 0.92 |
35 | 1 | 1 | 1 |
36 | 1 | 1 | 1 |
37 | 0 | 1 | 1 |
38 | 1 | 1 | 1 |
39 | 1 | 1 | 1 |
40 | 1 | 0.86 | 0.93 |
41 | 0.75 | 0.83 | 0.92 |
42 | 1 | 1 | 1 |
43 | 0.80 | 0.90 | 0.90 |
44 | 0.75 | 0.75 | 0.50 |
45 | 1 | 1 | 0.50 |
46 | 1 | 0.83 | 1 |
47 | 1 | 1 | 1 |
48 | 1 | 1 | 1 |
49 | 1 | 1 | 0.50 |
50 | 1 | 1 | 1 |
51 | 0.60 | 0.80 | 0.80 |
52 | 1 | 1 | 1 |
53 | 0.80 | 0.20 | 0.60 |
54 | 1 | 1 | 1 |
55 | 1 | 0.84 | 0.98 |
56 | 1 | 1 | 1 |
57 | 1 | 0.80 | 0.80 |
58 | 1 | 1 | 1 |
59 | 0.20 | 0.40 | 0.20 |
60 | 0.43 | 0.57 | 0.57 |
61 | 1 | 0.82 | 0.91 |
62 | 0.92 | 0.75 | 0.83 |
63 | 1 | 0 | 1 |
64 | 0.50 | 0.50 | 0.50 |
65 | 1 | 1 | 1 |
66 | 1 | 1 | 0.50 |
67 | 1 | 1 | 1 |
68 | 0.56 | 0.56 | 0.61 |
69 | 0.82 | 0.64 | 0.91 |
70 | 1 | 0.75 | 0.75 |
71 | 1 | 0 | 1 |
72 | 0.83 | 0.92 | 0.75 |
73 | 0.86 | 0.71 | 0.93 |
74 | 1 | 1 | 1 |
75 | 0.83 | 0.83 | 0.83 |
76 | 1 | 1 | 1 |
77 | 0 | 0 | 1 |
78 | 1 | 0 | 0 |
79 | 1 | 0.75 | 0.75 |
80 | 1 | 1 | 1 |
81 | 1 | 1 | 1 |
82 | 1 | 1 | 1 |
83 | 1 | 1 | 1 |
84 | 1 | 1 | 1 |
85 | 0.86 | 0.43 | 0.71 |
86 | 1 | 1 | 1 |
87 | 0.58 | 0.67 | 0.67 |
88 | 1 | 1 | 1 |
89 | 1 | 1 | 1 |
90 | 0.50 | 1 | 1 |
91 | 1 | 1 | 1 |
92 | 0.25 | 0.25 | 0.25 |
93 | 1 | 1 | 1 |
94 | 0.50 | 0.83 | 0.67 |
95 | 1 | 0.88 | 1 |
96 | 1 | 1 | 1 |
97 | 1 | 1 | 1 |
98 | 0.50 | 0.50 | 0.50 |
99 | 0.81 | 0.88 | 0.75 |
100 | 1 | 1 | 1 |
101 | 0.83 | 1 | 0.83 |
102 | 0.88 | 1 | 1 |
103 | 1 | 1 | 1 |
104 | 0.96 | 0.89 | 0.89 |
105 | 1 | 0.81 | 0.89 |
106 | 0.64 | 0.73 | 0.45 |
107 | 0.94 | 0.72 | 0.89 |
108 | 0.69 | 0.44 | 0.81 |
109 | 0.88 | 0.81 | 0.75 |
110 | 0.93 | 0.93 | 0.87 |
111 | 1 | 1 | 1 |
112 | 1 | 1 | 0.88 |
113 | 0.86 | 0.71 | 0.86 |
114 | 1 | 0.60 | 0.60 |
115 | 0.80 | 0.60 | 0.40 |
116 | 0.50 | 0.75 | 0.75 |
117 | 1 | 1 | 1 |
118 | 1 | 1 | 1 |
119 | 0.67 | 0.67 | 1 |
120 | 1 | 0.67 | 1 |
121 | 1 | 1 | 1 |
122 | 0.33 | 0.67 | 1 |
123 | 0.67 | 0.67 | 1 |
124 | 0.33 | 0.33 | 0.50 |
125 | 0.67 | 1 | 1 |
126 | 1 | 1 | 1 |
127 | 1 | 1 | 0.67 |
128 | 0.67 | 0.67 | 0.67 |
129 | 0.50 | 0.50 | 0 |
130 | 1 | 0.50 | 1 |
131 | 1 | 1 | 1 |
132 | 1 | 1 | 1 |
133 | 1 | 0 | 0 |
134 | 1 | 1 | 1 |
135 | 1 | 1 | 1 |
136 | 1 | 0 | 0 |
MRR Method
CSJ | HAFR | HL | |
---|---|---|---|
1 | 0.94 | 0.69 | 0.77 |
2 | 1 | 1 | 1 |
3 | 1 | 1 | 1 |
4 | 1 | 0 | 1 |
5 | 1 | 0.67 | 1 |
6 | 0.78 | 0.83 | 0.75 |
7 | 1 | 1 | 1 |
8 | 0.75 | 1 | 0.50 |
9 | 1 | 0.10 | 1 |
10 | 1 | 1 | 1 |
11 | 0.50 | 0.67 | 0.63 |
12 | 1 | 1 | 1 |
13 | 1 | 1 | 0.50 |
14 | 1 | 0.89 | 0.94 |
15 | 1 | 1 | 1 |
16 | 0.91 | 0.70 | 0.32 |
17 | 0.74 | 0.79 | 0.81 |
18 | 0.71 | 0.34 | 0.59 |
19 | 0.28 | 0.33 | 0.44 |
20 | 1 | 1 | 1 |
21 | 0.55 | 1 | 1 |
22 | 1 | 0.55 | 0.50 |
23 | 0 | 0 | 0 |
24 | 1 | 0.50 | 1 |
25 | 0.90 | 0.93 | 0.89 |
26 | 0 | 0 | 0.14 |
27 | 0.94 | 0.63 | 0.68 |
28 | 0.83 | 0.37 | 1 |
29 | 1 | 1 | 1 |
30 | 0.60 | 0.53 | 0.61 |
31 | 1 | 0 | 1 |
32 | 1 | 1 | 0.72 |
33 | 0.43 | 0.56 | 0.76 |
34 | 0.83 | 0.92 | 0.92 |
35 | 1 | 1 | 1 |
36 | 0.88 | 0.58 | 0.88 |
37 | 0 | 1 | 0.25 |
38 | 1 | 1 | 1 |
39 | 1 | 1 | 1 |
40 | 0.96 | 0.82 | 0.93 |
41 | 0.52 | 0.47 | 0.60 |
42 | 1 | 1 | 1 |
43 | 0.73 | 0.78 | 0.83 |
44 | 0.75 | 0.75 | 0.50 |
45 | 0.58 | 0.33 | 0.17 |
46 | 1 | 0.63 | 0.92 |
47 | 1 | 1 | 1 |
48 | 0.42 | 0.43 | 0.78 |
49 | 1 | 1 | 0.25 |
50 | 0.73 | 0.51 | 0.78 |
51 | 0.32 | 0.70 | 0.47 |
52 | 1 | 1 | 0.63 |
53 | 0.67 | 0.20 | 0.60 |
54 | 1 | 0.33 | 1 |
55 | 0.98 | 0.75 | 0.98 |
56 | 1 | 1 | 1 |
57 | 1 | 0.80 | 0.80 |
58 | 1 | 0.91 | 0.89 |
59 | 0.03 | 0.09 | 0.20 |
60 | 0.43 | 0.40 | 0.50 |
61 | 1 | 0.82 | 0.91 |
62 | 0.92 | 0.69 | 0.79 |
63 | 0.33 | 0 | 1 |
64 | 0.38 | 0.50 | 0.38 |
65 | 1 | 1 | 1 |
66 | 1 | 1 | 0.50 |
67 | 1 | 0.25 | 0.50 |
68 | 0.50 | 0.44 | 0.56 |
69 | 0.76 | 0.49 | 0.75 |
70 | 1 | 0.54 | 0.50 |
71 | 1 | 0 | 0.50 |
72 | 0.69 | 0.80 | 0.61 |
73 | 0.86 | 0.61 | 0.93 |
74 | 1 | 0.90 | 1 |
75 | 0.71 | 0.83 | 0.75 |
76 | 1 | 1 | 1 |
77 | 0 | 0 | 0.12 |
78 | 1 | 0 | 0 |
79 | 0.65 | 0.38 | 0.75 |
80 | 0.73 | 1 | 1 |
81 | 1 | 1 | 1 |
82 | 1 | 1 | 1 |
83 | 1 | 0.88 | 1 |
84 | 1 | 0.50 | 1 |
85 | 0.66 | 0.22 | 0.62 |
86 | 1 | 1 | 1 |
87 | 0.33 | 0.50 | 0.37 |
88 | 1 | 0.75 | 1 |
89 | 0.83 | 0.58 | 0.83 |
90 | 0.25 | 0.56 | 1 |
91 | 1 | 0.87 | 1 |
92 | 0.25 | 0.06 | 0.08 |
93 | 1 | 1 | 1 |
94 | 0.42 | 0.49 | 0.36 |
95 | 0.91 | 0.70 | 1 |
96 | 1 | 0.50 | 1 |
97 | 1 | 0.79 | 0.88 |
98 | 0.50 | 0.50 | 0.50 |
99 | 0.81 | 0.79 | 0.69 |
100 | 1 | 1 | 1 |
101 | 0.83 | 0.52 | 0.83 |
102 | 0.88 | 1 | 1 |
103 | 1 | 0.89 | 1 |
104 | 0.90 | 0.78 | 0.85 |
105 | 0.97 | 0.81 | 0.89 |
106 | 0.58 | 0.58 | 0.40 |
107 | 0.78 | 0.67 | 0.66 |
108 | 0.47 | 0.33 | 0.57 |
109 | 0.83 | 0.62 | 0.65 |
110 | 0.88 | 0.90 | 0.87 |
111 | 1 | 0.81 | 1 |
112 | 0.88 | 1 | 0.88 |
113 | 0.76 | 0.71 | 0.64 |
114 | 0.80 | 0.44 | 0.37 |
115 | 0.30 | 0.35 | 0.22 |
116 | 0.50 | 0.58 | 0.35 |
117 | 1 | 0.75 | 1 |
118 | 0.47 | 1 | 1 |
119 | 0.38 | 0.50 | 1 |
120 | 0.83 | 0.67 | 1 |
121 | 0.83 | 0.83 | 0.75 |
122 | 0.08 | 0.25 | 0.25 |
123 | 0.40 | 0.44 | 0.57 |
124 | 0.33 | 0.17 | 0.50 |
125 | 0.67 | 0.28 | 0.75 |
126 | 0.48 | 1 | 0.75 |
127 | 1 | 1 | 0.67 |
128 | 0.67 | 0.67 | 0.67 |
129 | 0.50 | 0.05 | 0 |
130 | 1 | 0.50 | 1 |
131 | 1 | 1 | 1 |
132 | 0.75 | 0.60 | 1 |
133 | 0.14 | 0 | 0 |
134 | 1 | 1 | 1 |
135 | 1 | 1 | 1 |
136 | 0.20 | 0 | 0 |
Task 2 Summary Results by Query
Simple Hit/Miss Counting [5]
MRR Method [6]
Runtime Results
Participant | Task | Runtime (min) | Machine |
---|---|---|---|
CSJ | 1a | ~30 | BEER 2 |
CSJ | 1b | 7 | BEER 2 |
CSJ | 2 | ~35 | BEER 2 |
HAFR | 1a | 15 | BEER 2 |
HAFR | 1b | 5 | BEER 2 |
HAFR | 2 | 35 | BEER 2 |
HL | 1a | ~95 | FAST |
HL | 1b | ~40 | FAST |
HL | 2 | ~95 | FAST |