Difference between revisions of "2009:Query-by-Singing/Humming Results"
IMIRSELBot (talk | contribs) m (Robot: Automated text replacement (-mirex/2009/results/ +mirex/results/2009/)) |
|||
(10 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
− | These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[Query by Singing/Humming]] page. | + | These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[2009:Query by Singing/Humming]] page. |
===Task Descriptions=== | ===Task Descriptions=== | ||
Line 8: | Line 8: | ||
*[[#Task 1a, Jang's dataset Results|Jang's Dataset Results]] Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] with 48 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 4431 humming. All queries are from the beginning of references | *[[#Task 1a, Jang's dataset Results|Jang's Dataset Results]] Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] with 48 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 4431 humming. All queries are from the beginning of references | ||
*[[#Task 1b, ThinkIT's dataset Results|ThinkIT's Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] data set with 106 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 355 humming. There are no "singing from beginning" gurantee. | *[[#Task 1b, ThinkIT's dataset Results|ThinkIT's Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus1.rar IOACAS corpus 1] data set with 106 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 355 humming. There are no "singing from beginning" gurantee. | ||
− | *[[#Task 1c, IOACAS2's dataset Results| | + | *[[#Task 1c, IOACAS2's dataset Results|IOACAS's 2nd Dataset Results]] [http://mirlab.org/dataSet/public/IOACAS_QBH_Coprus2.rar IOACAS corpus 2] data set with 192 songs as ground truth + 2000 Essen Collection MIDI noise files. See [http://www.esac-data.org/ ESAC Data Homepage] for more information about the Essen Collection. The queries consists of 404 humming. There are no "singing from beginning" gurantee. |
− | '''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': The second subtask is the query against other humming. In the second subtask, Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song. | + | '''Task 2 [[#Task 2 Results|Goto Task 2 Results]]''': The second subtask is the query against other humming. In the second subtask, Roger Jang's [http://mirlab.org/dataSet/public/MIR-QBSH-corpus.rar MIR-QBSH corpus] has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song. |
===General Legend=== | ===General Legend=== | ||
====Team ID==== | ====Team ID==== | ||
− | '''CSJ1''' = [ | + | '''CSJ1''' = [https://www.music-ir.org/mirex/results/2009/qbt/QbshJang.pdf Jyh-Shing Roger Jang] matched by beginning of the sond<br /> |
− | '''CSJ2''' = [ | + | '''CSJ2''' = [https://www.music-ir.org/mirex/results/2009/qbt/QbshJang.pdf Jyh-Shing Roger Jang] matched by anywhere of the song<br /> |
− | '''HAFR''' = [ | + | '''HAFR''' = [https://www.music-ir.org/mirex/results/2009/qbt/HAFR.pdf Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine]<br /> |
===Task 1 Results=== | ===Task 1 Results=== | ||
Line 24: | Line 24: | ||
=====Task 1a Overall Results===== | =====Task 1a Overall Results===== | ||
− | <csv>qbsh/QbshFinalTask1a.csv</csv> | + | <csv>2009/qbsh/QbshFinalTask1a.csv</csv> |
=====Task 1a Friedman's Test for Significant Differences===== | =====Task 1a Friedman's Test for Significant Differences===== | ||
Line 31: | Line 31: | ||
Simple Hit/Miss Count: | Simple Hit/Miss Count: | ||
− | <csv>qbsh/friedman/Qbsh1aTask2Simple_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbsh/friedman/Qbsh1aTask2Simple_friedman_tukeyKramerHSD.csv</csv> |
− | [[Image: | + | [[Image:2009_sqbsh1atask2simple_friedman_mean_ranks.pngγçä]] |
MRR Method: | MRR Method: | ||
− | <csv>qbsh/friedman/Qbsh1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbsh/friedman/Qbsh1aTask2Mrr_friedman_tukeyKramerHSD.csv</csv> |
− | [[Image: | + | [[Image:2009_sqbsh1atask2mrr_friedman_mean_ranks.pngγçä]] |
=====Task 1a Summary Results by Query Group===== | =====Task 1a Summary Results by Query Group===== | ||
Simple Hit/Miss Count | Simple Hit/Miss Count | ||
− | <csv>qbsh/QbshTask1aSimpleByGroup.csv</csv> | + | <csv>2009/qbsh/QbshTask1aSimpleByGroup.csv</csv> |
MRR Method | MRR Method | ||
− | <csv>qbsh/QbshTask1aMrrByGroup.csv</csv> | + | <csv>2009/qbsh/QbshTask1aMrrByGroup.csv</csv> |
====Task 1a Summary Results by Query ==== | ====Task 1a Summary Results by Query ==== | ||
Simple Hit/Miss Counting | Simple Hit/Miss Counting | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask1aSimpleByQuery.csv] |
MRR Method | MRR Method | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask1aMrrByQuery.csv] |
====Task 1b, ThinkIT's dataset Results==== | ====Task 1b, ThinkIT's dataset Results==== | ||
=====Task 1b Overall Results===== | =====Task 1b Overall Results===== | ||
− | <csv>qbsh/QbshFinalTask1b.csv</csv> | + | <csv>2009/qbsh/QbshFinalTask1b.csv</csv> |
=====Task 1b Friedman's Test for Significant Differences===== | =====Task 1b Friedman's Test for Significant Differences===== | ||
Line 64: | Line 64: | ||
Simple Hit/Miss Count: | Simple Hit/Miss Count: | ||
− | <csv>qbsh/friedman/Qbsh1bTask2Simple_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbsh/friedman/Qbsh1bTask2Simple_friedman_tukeyKramerHSD.csv</csv> |
− | [[Image: | + | [[Image:2009_sqbsh1btask2simple_friedman_mean_ranks.pngγçä]] |
MRR Method: | MRR Method: | ||
− | <csv>qbsh/friedman/Qbsh1bTask2Mrr_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbsh/friedman/Qbsh1bTask2Mrr_friedman_tukeyKramerHSD.csv</csv> |
− | [[Image: | + | [[Image:2009_sqbsh1btask2mrr_friedman_mean_ranks.pngγçä]] |
=====Task 1b Summary Results by Query Group===== | =====Task 1b Summary Results by Query Group===== | ||
Simple Hit/Miss Count | Simple Hit/Miss Count | ||
− | <csv>qbsh/QbshTask1bSimpleByGroup.csv</csv> | + | <csv>2009/qbsh/QbshTask1bSimpleByGroup.csv</csv> |
MRR Method | MRR Method | ||
− | <csv>qbsh/QbshTask1bMrrByGroup.csv</csv> | + | <csv>2009/qbsh/QbshTask1bMrrByGroup.csv</csv> |
====Task 1b Summary Results by Query ==== | ====Task 1b Summary Results by Query ==== | ||
Simple Hit/Miss Counting | Simple Hit/Miss Counting | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask1bSimpleByQuery.csv] |
MRR Method | MRR Method | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask1bMrrByQuery.csv] |
====Task 1c, IOACAS2's dataset Results==== | ====Task 1c, IOACAS2's dataset Results==== | ||
Line 92: | Line 92: | ||
=====Task 1c Summary Results by Query Group===== | =====Task 1c Summary Results by Query Group===== | ||
Simple Hit/Miss Count | Simple Hit/Miss Count | ||
− | <csv>qbsh/QbshTask1cSimpleByGroup.csv</csv> | + | <csv>2009/qbsh/QbshTask1cSimpleByGroup.csv</csv> |
MRR Method | MRR Method | ||
− | <csv>qbsh/QbshTask1cMrrByGroup.csv</csv> | + | <csv>2009/qbsh/QbshTask1cMrrByGroup.csv</csv> |
====Task 1c Summary Results by Query ==== | ====Task 1c Summary Results by Query ==== | ||
Simple Hit/Miss Counting | Simple Hit/Miss Counting | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask1cSimpleByQuery.csv] |
MRR Method | MRR Method | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask1cMrrByQuery.csv] |
===Task 2 Results=== | ===Task 2 Results=== | ||
=====Task 2 Overall Results===== | =====Task 2 Overall Results===== | ||
− | <csv>qbsh/QbshFinalTask2.csv</csv> | + | <csv>2009/qbsh/QbshFinalTask2.csv</csv> |
=====Task 2 Friedman's Test for Significant Differences===== | =====Task 2 Friedman's Test for Significant Differences===== | ||
Line 113: | Line 113: | ||
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
− | <csv>qbsh/friedman/QbshTask2_friedman_tukeyKramerHSD.csv</csv> | + | <csv>2009/qbsh/friedman/QbshTask2_friedman_tukeyKramerHSD.csv</csv> |
− | [[Image: | + | [[Image:2009_sqbshtask2_friedman_mean_ranks.png γçä]] |
=====Task 2 Summary Results by Query Group===== | =====Task 2 Summary Results by Query Group===== | ||
− | <csv>qbsh/QbshTask2Group.csv</csv> | + | <csv>2009/qbsh/QbshTask2Group.csv</csv> |
====Task 2 Summary Results by Query ==== | ====Task 2 Summary Results by Query ==== | ||
− | [https://www.music-ir.org/mirex/2009 | + | [https://www.music-ir.org/mirex/results/2009/qbsh/QbshTask2Query.csv] |
===Runtime Results=== | ===Runtime Results=== | ||
− | <csv>qbsh | + | <csv>2009/qbsh/qbshRunTime.csv</csv> |
[[Category: Results]] | [[Category: Results]] |
Latest revision as of 22:42, 13 May 2010
Contents
- 1 Introduction
- 1.1 Task Descriptions
- 1.2 General Legend
- 1.3 Task 1 Results
- 1.4 Task 2 Results
- 1.5 Runtime Results
Introduction
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2009:Query by Singing/Humming page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
- Jang's Dataset Results Roger Jang's MIR-QBSH corpus with 48 songs as ground truth + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The queries consists of 4431 humming. All queries are from the beginning of references
- ThinkIT's Dataset Results IOACAS corpus 1 data set with 106 songs as ground truth + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The queries consists of 355 humming. There are no "singing from beginning" gurantee.
- IOACAS's 2nd Dataset Results IOACAS corpus 2 data set with 192 songs as ground truth + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The queries consists of 404 humming. There are no "singing from beginning" gurantee.
Task 2 Goto Task 2 Results: The second subtask is the query against other humming. In the second subtask, Roger Jang's MIR-QBSH corpus has been divided into two groups (2040 as queries and 2391 as database). The query is performed against the other humming database and the top 10 closed are returned. The score is simple count of how many returns belong to the same ground truth song.
General Legend
Team ID
CSJ1 = Jyh-Shing Roger Jang matched by beginning of the sond
CSJ2 = Jyh-Shing Roger Jang matched by anywhere of the song
HAFR = Pierre Hanna, Julien Allali, Pascal Ferraro,Matthias Robine
Task 1 Results
Task 1a, Jang's dataset Results
Task 1a Overall Results
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
Simple Count | 0.94 | 0.9 | 0.77 |
MRR | 0.91 | 0.86 | 0.66 |
Total count | 4431 | 4431 | 4409 |
Task 1a Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ1 | CSJ2 | 0.3232 | 0.7812 | 1.2393 | TRUE |
CSJ1 | HAFR | 1.1670 | 1.6250 | 2.0830 | TRUE |
CSJ2 | HAFR | 0.3857 | 0.8438 | 1.3018 | TRUE |
File:2009 sqbsh1atask2simple friedman mean ranks.pngγçä
MRR Method:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ1 | CSJ2 | 0.2009 | 0.6667 | 1.1324 | TRUE |
CSJ1 | HAFR | 1.1801 | 1.6458 | 2.1116 | TRUE |
CSJ2 | HAFR | 0.5134 | 0.9792 | 1.4449 | TRUE |
File:2009 sqbsh1atask2mrr friedman mean ranks.pngγçä
Task 1a Summary Results by Query Group
Simple Hit/Miss Count
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 0.89 | 0.84 | 0.89 |
2 | 0.71 | 0.5 | 0.36 |
3 | 0.64 | 0.64 | 0.64 |
4 | 0.8 | 0.8 | 0.67 |
5 | 0.93 | 0.73 | 0.87 |
6 | 0.92 | 0.75 | 0.67 |
7 | 0.93 | 0.87 | 0.47 |
8 | 0.93 | 0.93 | 0.6 |
9 | 0.92 | 0.92 | 0.73 |
10 | 0.74 | 0.84 | 0.84 |
11 | 0.94 | 0.89 | 0.8 |
12 | 0.96 | 0.95 | 0.92 |
13 | 0.98 | 0.97 | 0.88 |
14 | 0.98 | 0.97 | 0.92 |
15 | 0.96 | 0.96 | 0.78 |
16 | 0.93 | 0.9 | 0.93 |
17 | 0.97 | 0.85 | 0.73 |
18 | 0.99 | 0.96 | 0.75 |
19 | 0.97 | 0.94 | 0.89 |
20 | 0.99 | 0.96 | 0.46 |
21 | 0.88 | 0.79 | 0.32 |
22 | 0.97 | 0.97 | 0.95 |
23 | 0.98 | 0.98 | 0.85 |
24 | 0.83 | 0.82 | 0.58 |
25 | 0.72 | 0.45 | 0.57 |
26 | 0.98 | 0.92 | 0.65 |
27 | 0.95 | 0.98 | 0.76 |
28 | 0.87 | 0.69 | 0.49 |
29 | 0.98 | 0.97 | 0.57 |
30 | 0.97 | 0.94 | 0.94 |
31 | 0.97 | 0.95 | 0.94 |
32 | 0.96 | 0.93 | 0.8 |
33 | 0.92 | 0.86 | 0.5 |
34 | 0.93 | 0.91 | 0.64 |
35 | 0.95 | 0.87 | 0.76 |
36 | 0.93 | 0.91 | 0.91 |
37 | 0.92 | 0.91 | 0.88 |
38 | 0.95 | 0.95 | 0.78 |
39 | 0.89 | 0.83 | 0.82 |
40 | 0.94 | 0.9 | 0.44 |
41 | 0.94 | 0.93 | 0.9 |
42 | 0.96 | 0.93 | 0.86 |
43 | 0.95 | 0.94 | 0.9 |
44 | 0.93 | 0.94 | 0.88 |
45 | 0.95 | 0.91 | 0.83 |
46 | 0.95 | 0.92 | 0.8 |
47 | 1 | 0.97 | 0.93 |
48 | 1 | 0.96 | 0.81 |
MRR Method
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 0.82 | 0.77 | 0.82 |
2 | 0.61 | 0.5 | 0.36 |
3 | 0.64 | 0.64 | 0.58 |
4 | 0.8 | 0.8 | 0.62 |
5 | 0.82 | 0.7 | 0.72 |
6 | 0.79 | 0.62 | 0.59 |
7 | 0.89 | 0.87 | 0.47 |
8 | 0.88 | 0.93 | 0.52 |
9 | 0.88 | 0.83 | 0.64 |
10 | 0.7 | 0.84 | 0.72 |
11 | 0.92 | 0.85 | 0.72 |
12 | 0.94 | 0.94 | 0.79 |
13 | 0.96 | 0.94 | 0.74 |
14 | 0.95 | 0.9 | 0.81 |
15 | 0.94 | 0.96 | 0.7 |
16 | 0.9 | 0.89 | 0.9 |
17 | 0.91 | 0.7 | 0.56 |
18 | 0.97 | 0.93 | 0.64 |
19 | 0.96 | 0.91 | 0.8 |
20 | 0.98 | 0.95 | 0.3 |
21 | 0.75 | 0.69 | 0.24 |
22 | 0.95 | 0.95 | 0.91 |
23 | 0.96 | 0.96 | 0.71 |
24 | 0.78 | 0.75 | 0.49 |
25 | 0.58 | 0.35 | 0.4 |
26 | 0.97 | 0.9 | 0.57 |
27 | 0.95 | 0.95 | 0.58 |
28 | 0.74 | 0.58 | 0.38 |
29 | 0.96 | 0.95 | 0.43 |
30 | 0.96 | 0.9 | 0.88 |
31 | 0.97 | 0.94 | 0.84 |
32 | 0.93 | 0.89 | 0.64 |
33 | 0.88 | 0.81 | 0.36 |
34 | 0.91 | 0.87 | 0.53 |
35 | 0.93 | 0.86 | 0.67 |
36 | 0.93 | 0.91 | 0.84 |
37 | 0.87 | 0.86 | 0.8 |
38 | 0.9 | 0.91 | 0.63 |
39 | 0.86 | 0.79 | 0.69 |
40 | 0.89 | 0.85 | 0.31 |
41 | 0.91 | 0.91 | 0.82 |
42 | 0.94 | 0.91 | 0.76 |
43 | 0.91 | 0.85 | 0.8 |
44 | 0.92 | 0.93 | 0.84 |
45 | 0.93 | 0.9 | 0.77 |
46 | 0.92 | 0.9 | 0.74 |
47 | 0.98 | 0.94 | 0.83 |
48 | 0.95 | 0.95 | 0.7 |
Task 1a Summary Results by Query
Simple Hit/Miss Counting [1]
MRR Method [2]
Task 1b, ThinkIT's dataset Results
Task 1b Overall Results
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
Simple Count | 0.43 | 0.86 | 0.78 |
MRR | 0.41 | 0.8 | 0.68 |
Total count | 355 | 355 | 355 |
Task 1b Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
Simple Hit/Miss Count:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ2 | HAFR | -0.0394 | 0.2217 | 0.4828 | FALSE |
CSJ2 | CSJ1 | 0.7058 | 0.9670 | 1.2281 | TRUE |
HAFR | CSJ1 | 0.4841 | 0.7453 | 1.0064 | TRUE |
File:2009 sqbsh1btask2simple friedman mean ranks.pngγçä
MRR Method:
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ2 | HAFR | 0.0442 | 0.3208 | 0.5974 | TRUE |
CSJ2 | CSJ1 | 0.6904 | 0.9670 | 1.2436 | TRUE |
HAFR | CSJ1 | 0.3696 | 0.6462 | 0.9228 | TRUE |
File:2009 sqbsh1btask2mrr friedman mean ranks.pngγçä
Task 1b Summary Results by Query Group
Simple Hit/Miss Count
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 0.55 | 1 | 0.82 |
2 | 0.6 | 0.8 | 0.8 |
3 | 0 | 0 | 1 |
4 | 0 | 1 | 1 |
5 | 0 | 0 | 0 |
6 | 0 | 0.83 | 0.83 |
7 | 1 | 1 | 0.83 |
8 | 0.33 | 0.33 | 0 |
9 | 0 | 0.8 | 1 |
10 | 0 | 1 | 1 |
11 | 0 | 1 | 0.5 |
12 | 1 | 1 | 1 |
13 | 0 | 0.6 | 0.6 |
14 | 0 | 0.5 | 0.5 |
15 | 0 | 0.5 | 0 |
16 | 1 | 0.5 | 1 |
17 | 0 | 0 | 0 |
18 | 0.5 | 1 | 0.5 |
19 | 0 | 1 | 1 |
20 | 0 | 0 | 1 |
21 | 0 | 0 | 0.5 |
22 | 0.44 | 0.89 | 0.89 |
23 | 1 | 1 | 1 |
24 | 0 | 1 | 1 |
25 | 0.5 | 1 | 1 |
26 | 0.64 | 0.91 | 0.91 |
27 | 0.5 | 1 | 0 |
28 | 0.5 | 1 | 0.5 |
29 | 1 | 1 | 1 |
30 | 0.75 | 1 | 0.75 |
31 | 1 | 1 | 1 |
32 | 0 | 1 | 0.86 |
33 | 0 | 1 | 1 |
34 | 0.5 | 0.5 | 0 |
35 | 0 | 1 | 1 |
36 | 0 | 0.8 | 0.6 |
37 | 0 | 0 | 0 |
38 | 0.5 | 1 | 1 |
39 | 0.43 | 1 | 1 |
40 | 1 | 1 | 0.5 |
41 | 0.13 | 0.88 | 0.63 |
42 | 0.67 | 0.92 | 0.5 |
43 | 0.67 | 0.67 | 0.56 |
44 | 0.56 | 1 | 1 |
45 | 1 | 1 | 0.67 |
46 | 0.5 | 0.75 | 0.75 |
47 | 0.75 | 1 | 1 |
48 | 1 | 1 | 1 |
49 | 1 | 1 | 1 |
50 | 1 | 1 | 1 |
51 | 0.5 | 0.75 | 1 |
52 | 0.25 | 1 | 0.75 |
53 | 0.33 | 1 | 0.83 |
54 | 0.67 | 1 | 1 |
55 | 0 | 1 | 0.75 |
56 | 0.5 | 0.5 | 0.5 |
57 | 1 | 1 | 0.4 |
58 | 0 | 1 | 1 |
59 | 0 | 1 | 1 |
60 | 1 | 1 | 0.5 |
61 | 0 | 1 | 1 |
62 | 0.33 | 1 | 1 |
63 | 1 | 1 | 1 |
64 | 0 | 1 | 0 |
65 | 0 | 0.67 | 1 |
66 | 0.5 | 0.75 | 1 |
67 | 0 | 1 | 0.67 |
68 | 1 | 1 | 1 |
69 | 0 | 1 | 1 |
70 | 1 | 1 | 1 |
71 | 0 | 0.67 | 1 |
72 | 1 | 1 | 1 |
73 | 1 | 1 | 0.5 |
74 | 1 | 1 | 1 |
75 | 0.33 | 0.67 | 0.67 |
76 | 0 | 1 | 0.5 |
77 | 1 | 1 | 1 |
78 | 1 | 1 | 1 |
79 | 0.5 | 1 | 1 |
80 | 0.33 | 1 | 1 |
81 | 0.33 | 0.5 | 1 |
82 | 0 | 1 | 1 |
83 | 0 | 1 | 1 |
84 | 0 | 0.25 | 0.5 |
85 | 0.5 | 1 | 0.75 |
86 | 1 | 1 | 1 |
87 | 0 | 1 | 0.33 |
88 | 0 | 1 | 1 |
89 | 0.43 | 0.86 | 1 |
90 | 0 | 1 | 1 |
91 | 0 | 1 | 0.5 |
92 | 0.5 | 1 | 1 |
93 | 0.5 | 1 | 1 |
94 | 0 | 1 | 1 |
95 | 0 | 1 | 0 |
96 | 1 | 0 | 1 |
97 | 0 | 1 | 1 |
98 | 0 | 1 | 1 |
99 | 0.75 | 1 | 1 |
100 | 0.33 | 1 | 1 |
101 | 0 | 0 | 0 |
102 | 0 | 1 | 1 |
103 | 0.38 | 0.75 | 0.63 |
104 | 0 | 1 | 1 |
105 | 1 | 0 | 1 |
106 | 0 | 0 | 0 |
MRR Method
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 0.42 | 0.95 | 0.82 |
2 | 0.6 | 0.8 | 0.55 |
3 | 0 | 0 | 1 |
4 | 0 | 0.67 | 0.5 |
5 | 0 | 0 | 0 |
6 | 0 | 0.83 | 0.83 |
7 | 1 | 0.92 | 0.83 |
8 | 0.04 | 0.33 | 0 |
9 | 0 | 0.8 | 0.85 |
10 | 0 | 1 | 1 |
11 | 0 | 0.75 | 0.17 |
12 | 1 | 1 | 1 |
13 | 0 | 0.6 | 0.3 |
14 | 0 | 0.5 | 0.5 |
15 | 0 | 0.5 | 0 |
16 | 1 | 0.5 | 1 |
17 | 0 | 0 | 0 |
18 | 0.5 | 1 | 0.5 |
19 | 0 | 1 | 1 |
20 | 0 | 0 | 0.57 |
21 | 0 | 0 | 0.5 |
22 | 0.44 | 0.81 | 0.65 |
23 | 1 | 1 | 1 |
24 | 0 | 1 | 1 |
25 | 0.5 | 1 | 0.58 |
26 | 0.64 | 0.66 | 0.82 |
27 | 0.38 | 0.88 | 0 |
28 | 0.5 | 0.78 | 0.5 |
29 | 1 | 1 | 1 |
30 | 0.75 | 1 | 0.75 |
31 | 1 | 1 | 1 |
32 | 0 | 1 | 0.79 |
33 | 0 | 1 | 1 |
34 | 0.17 | 0.25 | 0 |
35 | 0 | 1 | 1 |
36 | 0 | 0.8 | 0.6 |
37 | 0 | 0 | 0 |
38 | 0.5 | 1 | 1 |
39 | 0.43 | 0.93 | 0.7 |
40 | 1 | 1 | 0.5 |
41 | 0.13 | 0.76 | 0.56 |
42 | 0.67 | 0.92 | 0.42 |
43 | 0.57 | 0.67 | 0.46 |
44 | 0.56 | 1 | 0.85 |
45 | 1 | 1 | 0.67 |
46 | 0.5 | 0.5 | 0.5 |
47 | 0.75 | 1 | 1 |
48 | 1 | 1 | 1 |
49 | 1 | 1 | 0.5 |
50 | 1 | 1 | 1 |
51 | 0.5 | 0.54 | 1 |
52 | 0.25 | 1 | 0.75 |
53 | 0.33 | 0.85 | 0.83 |
54 | 0.58 | 1 | 1 |
55 | 0 | 1 | 0.36 |
56 | 0.5 | 0.5 | 0.5 |
57 | 1 | 1 | 0.23 |
58 | 0 | 1 | 1 |
59 | 0 | 1 | 0.46 |
60 | 1 | 1 | 0.5 |
61 | 0 | 1 | 1 |
62 | 0.33 | 1 | 0.83 |
63 | 1 | 0.75 | 1 |
64 | 0 | 1 | 0 |
65 | 0 | 0.67 | 1 |
66 | 0.5 | 0.75 | 1 |
67 | 0 | 1 | 0.37 |
68 | 1 | 1 | 0.63 |
69 | 0 | 1 | 1 |
70 | 1 | 1 | 1 |
71 | 0 | 0.42 | 1 |
72 | 1 | 1 | 1 |
73 | 1 | 1 | 0.5 |
74 | 1 | 1 | 1 |
75 | 0.33 | 0.4 | 0.67 |
76 | 0 | 1 | 0.5 |
77 | 1 | 1 | 0.78 |
78 | 1 | 1 | 0.2 |
79 | 0.5 | 1 | 1 |
80 | 0.33 | 1 | 1 |
81 | 0.33 | 0.35 | 1 |
82 | 0 | 1 | 1 |
83 | 0 | 0.71 | 1 |
84 | 0 | 0.08 | 0.09 |
85 | 0.5 | 1 | 0.63 |
86 | 1 | 1 | 1 |
87 | 0 | 1 | 0.33 |
88 | 0 | 1 | 1 |
89 | 0.43 | 0.59 | 0.87 |
90 | 0 | 1 | 0.29 |
91 | 0 | 0.58 | 0.5 |
92 | 0.5 | 1 | 1 |
93 | 0.5 | 1 | 1 |
94 | 0 | 1 | 1 |
95 | 0 | 0.25 | 0 |
96 | 0.5 | 0 | 1 |
97 | 0 | 1 | 1 |
98 | 0 | 1 | 0.33 |
99 | 0.75 | 0.68 | 0.78 |
100 | 0.33 | 1 | 1 |
101 | 0 | 0 | 0 |
102 | 0 | 0.33 | 1 |
103 | 0.38 | 0.75 | 0.46 |
104 | 0 | 1 | 1 |
105 | 0.1 | 0 | 0.14 |
106 | 0 | 0 | 0 |
Task 1b Summary Results by Query
Simple Hit/Miss Counting [3]
MRR Method [4]
Task 1c, IOACAS2's dataset Results
As argued by Jang, there are some abnormal in the IOACAS2 data set. Therefore, we just provide the raw result without any further analysis.
Task 1c Summary Results by Query Group
Simple Hit/Miss Count
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 0 | 1 | 0 |
2 | 0 | 0 | 1 |
3 | 0 | 1 | 1 |
4 | 0 | 0 | 0.33 |
5 | 0.27 | 0.27 | 0.36 |
6 | 0 | 0 | 0 |
7 | 0.5 | 0 | 0.5 |
8 | 0 | 0 | 0 |
9 | 0 | 0 | 1 |
10 | 1 | 1 | 1 |
11 | 0 | 0.5 | 0.75 |
12 | 0 | 1 | 1 |
13 | 0.14 | 0.14 | 0.43 |
14 | 0 | 0 | 0.4 |
15 | 0 | 0 | 0 |
16 | 1 | 1 | 1 |
17 | 0 | 0 | 1 |
18 | 0 | 1 | 0 |
19 | 0.5 | 0 | 0 |
20 | 0.25 | 0 | 0.5 |
21 | 0 | 0 | 1 |
22 | 0 | 0 | 0 |
23 | 0 | 0 | 0 |
24 | 0 | 0.14 | 0.14 |
25 | 0 | 0 | 0 |
26 | 0 | 0.25 | 1 |
27 | 0 | 1 | 0 |
28 | 0 | 0 | 0 |
29 | 0.5 | 0.38 | 0.63 |
30 | 0 | 0 | 1 |
31 | 0 | 0 | 0 |
32 | 0 | 0 | 1 |
33 | 0.33 | 0 | 0.33 |
34 | 0 | 1 | 1 |
35 | 0 | 1 | 0 |
36 | 0 | 0 | 1 |
37 | 0.11 | 0.33 | 0.89 |
38 | 0.5 | 0.5 | 0.5 |
39 | 0 | 1 | 1 |
40 | 0.5 | 0.5 | 1 |
41 | 0 | 1 | 1 |
42 | 0 | 0 | 1 |
43 | 0 | 0 | 1 |
44 | 0.5 | 1 | 1 |
45 | 0 | 0 | 0 |
46 | 0 | 0 | 1 |
47 | 0 | 1 | 1 |
48 | 0 | 1 | 0 |
49 | 0 | 0 | 0 |
50 | 0.13 | 0 | 0.88 |
51 | 0.25 | 0.5 | 0.75 |
52 | 0.33 | 0.33 | 0.67 |
53 | 0 | 0 | 0.5 |
54 | 0 | 0 | 0.5 |
55 | 0 | 0 | 1 |
56 | 0.33 | 0.33 | 0 |
57 | 0 | 0.33 | 1 |
58 | 0 | 0 | 0.29 |
59 | 1 | 1 | 0 |
60 | 0 | 1 | 1 |
61 | 0 | 0.5 | 0.5 |
62 | 0 | 0 | 0 |
63 | 0 | 0 | 0 |
64 | 0.44 | 0.33 | 0.33 |
65 | 0 | 0 | 0 |
66 | 0 | 0 | 0 |
67 | 0 | 0 | 0.5 |
68 | 0.6 | 0.7 | 0.5 |
69 | 0 | 0 | 0.33 |
70 | 0 | 0.6 | 0.3 |
71 | 0 | 0 | 1 |
72 | 0.5 | 0.33 | 0.33 |
73 | 0 | 0 | 0 |
74 | 0 | 0 | 0 |
75 | 0 | 0 | 1 |
76 | 0 | 0 | 0 |
77 | 1 | 1 | 1 |
78 | 0.83 | 0 | 0.17 |
79 | 0 | 0 | 1 |
80 | 0.54 | 0.62 | 0.85 |
81 | 0 | 0 | 1 |
82 | 0 | 0 | 0 |
83 | 0.43 | 0.14 | 1 |
84 | 0 | 0 | 0 |
85 | 0 | 0 | 1 |
86 | 0 | 1 | 0.25 |
87 | 1 | 1 | 1 |
88 | 0.25 | 0.75 | 0.75 |
89 | 0 | 1 | 1 |
90 | 0 | 0 | 0 |
91 | 0.5 | 0.5 | 1 |
92 | 0.5 | 1 | 1 |
93 | 0.4 | 0.6 | 0.8 |
94 | 0 | 0 | 0 |
95 | 0 | 0 | 1 |
96 | 0.33 | 0.33 | 1 |
97 | 0 | 0.5 | 1 |
98 | 1 | 0.5 | 0.5 |
99 | 0 | 0 | 0 |
100 | 0 | 0 | 1 |
101 | 0 | 1 | 1 |
102 | 0.5 | 0.5 | 0.5 |
103 | 0 | 1 | 0 |
104 | 1 | 0 | 1 |
105 | 0 | 1 | 0 |
106 | 0 | 0 | 0 |
107 | 0 | 1 | 1 |
108 | 0 | 1 | 1 |
109 | 1 | 0.5 | 1 |
110 | 0 | 0 | 1 |
111 | 0 | 0 | 0 |
112 | 0 | 0 | 0.5 |
113 | 0 | 0 | 0 |
114 | 1 | 1 | 1 |
115 | 0 | 0 | 0.5 |
116 | 0 | 0 | 0.5 |
117 | 0 | 0 | 0 |
118 | 1 | 0 | 1 |
119 | 0 | 0.2 | 0.4 |
120 | 0 | 0 | 1 |
121 | 0 | 0 | 0 |
122 | 1 | 0 | 1 |
123 | 0 | 1 | 1 |
124 | 0 | 0 | 0 |
125 | 0 | 1 | 1 |
126 | 0 | 0 | 0 |
127 | 0 | 0 | 0 |
128 | 0 | 0 | 0 |
129 | 0 | 1 | 1 |
130 | 0 | 0 | 0 |
131 | 1 | 1 | 1 |
132 | 0 | 0 | 0.5 |
133 | 1 | 1 | 0 |
134 | 1 | 1 | 1 |
135 | 0 | 0 | 0 |
136 | 0 | 1 | 1 |
137 | 0 | 1 | 1 |
138 | 0 | 1 | 1 |
139 | 1 | 1 | 1 |
140 | 1 | 0 | 1 |
141 | 1 | 1 | 1 |
142 | 0 | 0.5 | 1 |
143 | 0 | 0 | 1 |
144 | 0 | 0 | 1 |
145 | 0.33 | 0.33 | 1 |
146 | 0 | 0 | 0 |
147 | 0 | 1 | 0 |
148 | 0.5 | 1 | 0.5 |
149 | 1 | 1 | 1 |
150 | 0.5 | 0.5 | 1 |
151 | 0 | 1 | 1 |
152 | 0.33 | 0.33 | 1 |
153 | 0 | 0 | 1 |
154 | 0 | 1 | 0 |
155 | 0 | 0 | 1 |
156 | 1 | 0 | 1 |
157 | 1 | 1 | 1 |
158 | 0 | 0 | 0 |
159 | 0 | 0 | 0 |
160 | 0 | 0 | 0 |
161 | 0 | 0 | 0 |
162 | 1 | 1 | 0 |
163 | 0 | 0 | 1 |
164 | 0 | 1 | 0 |
165 | 0 | 1 | 1 |
166 | 0 | 0 | 0 |
167 | 1 | 1 | 1 |
168 | 1 | 1 | 0 |
169 | 1 | 1 | 1 |
170 | 1 | 1 | 1 |
171 | 0 | 0 | 1 |
172 | 1 | 1 | 1 |
173 | 1 | 0 | 1 |
174 | 1 | 0 | 1 |
175 | 0 | 1 | 1 |
176 | 0 | 1 | 1 |
177 | 0 | 0 | 1 |
178 | 1 | 0.5 | 1 |
179 | 1 | 0 | 1 |
180 | 0 | 0 | 1 |
181 | 1 | 1 | 1 |
182 | 0 | 0 | 1 |
183 | 0 | 1 | 1 |
184 | 0 | 1 | 0 |
185 | 0 | 1 | 1 |
186 | 0 | 0.33 | 0.33 |
187 | 0 | 1 | 1 |
188 | 0 | 0 | 1 |
189 | 0 | 1 | 1 |
190 | 0 | 0.5 | 1 |
191 | 1 | 1 | 1 |
192 | 0 | 1 | 0 |
MRR Method
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 0 | 0.17 | 0 |
2 | 0 | 0 | 0.58 |
3 | 0 | 1 | 1 |
4 | 0 | 0 | 0.33 |
5 | 0.15 | 0.2 | 0.19 |
6 | 0 | 0 | 0 |
7 | 0.17 | 0 | 0.5 |
8 | 0 | 0 | 0 |
9 | 0 | 0 | 1 |
10 | 1 | 1 | 1 |
11 | 0 | 0.28 | 0.63 |
12 | 0 | 1 | 1 |
13 | 0.02 | 0.04 | 0.43 |
14 | 0 | 0 | 0.4 |
15 | 0 | 0 | 0 |
16 | 1 | 0.25 | 1 |
17 | 0 | 0 | 0.63 |
18 | 0 | 0.17 | 0 |
19 | 0.06 | 0 | 0 |
20 | 0.03 | 0 | 0.5 |
21 | 0 | 0 | 1 |
22 | 0 | 0 | 0 |
23 | 0 | 0 | 0 |
24 | 0 | 0.02 | 0.01 |
25 | 0 | 0 | 0 |
26 | 0 | 0.13 | 0.57 |
27 | 0 | 0.25 | 0 |
28 | 0 | 0 | 0 |
29 | 0.39 | 0.26 | 0.41 |
30 | 0 | 0 | 1 |
31 | 0 | 0 | 0 |
32 | 0 | 0 | 0.75 |
33 | 0.33 | 0 | 0.17 |
34 | 0 | 0.2 | 1 |
35 | 0 | 0.14 | 0 |
36 | 0 | 0 | 0.75 |
37 | 0.01 | 0.08 | 0.75 |
38 | 0.25 | 0.25 | 0.5 |
39 | 0 | 0.13 | 1 |
40 | 0.5 | 0.5 | 1 |
41 | 0 | 1 | 1 |
42 | 0 | 0 | 1 |
43 | 0 | 0 | 0.78 |
44 | 0.25 | 0.6 | 1 |
45 | 0 | 0 | 0 |
46 | 0 | 0 | 1 |
47 | 0 | 0.25 | 0.33 |
48 | 0 | 1 | 0 |
49 | 0 | 0 | 0 |
50 | 0.03 | 0 | 0.66 |
51 | 0.03 | 0.29 | 0.53 |
52 | 0.33 | 0.17 | 0.67 |
53 | 0 | 0 | 0.13 |
54 | 0 | 0 | 0.5 |
55 | 0 | 0 | 0.14 |
56 | 0.06 | 0.08 | 0 |
57 | 0 | 0.08 | 0.16 |
58 | 0 | 0 | 0.29 |
59 | 1 | 0.14 | 0 |
60 | 0 | 1 | 0.33 |
61 | 0 | 0.25 | 0.25 |
62 | 0 | 0 | 0 |
63 | 0 | 0 | 0 |
64 | 0.36 | 0.22 | 0.2 |
65 | 0 | 0 | 0 |
66 | 0 | 0 | 0 |
67 | 0 | 0 | 0.07 |
68 | 0.55 | 0.36 | 0.43 |
69 | 0 | 0 | 0.33 |
70 | 0 | 0.6 | 0.3 |
71 | 0 | 0 | 1 |
72 | 0.5 | 0.13 | 0.13 |
73 | 0 | 0 | 0 |
74 | 0 | 0 | 0 |
75 | 0 | 0 | 0.6 |
76 | 0 | 0 | 0 |
77 | 1 | 0.5 | 1 |
78 | 0.42 | 0 | 0.02 |
79 | 0 | 0 | 1 |
80 | 0.4 | 0.43 | 0.55 |
81 | 0 | 0 | 0.8 |
82 | 0 | 0 | 0 |
83 | 0.17 | 0.02 | 0.81 |
84 | 0 | 0 | 0 |
85 | 0 | 0 | 0.5 |
86 | 0 | 0.79 | 0.13 |
87 | 0.5 | 1 | 1 |
88 | 0.06 | 0.63 | 0.75 |
89 | 0 | 1 | 1 |
90 | 0 | 0 | 0 |
91 | 0.07 | 0.5 | 1 |
92 | 0.5 | 1 | 1 |
93 | 0.25 | 0.43 | 0.7 |
94 | 0 | 0 | 0 |
95 | 0 | 0 | 1 |
96 | 0.33 | 0.17 | 0.78 |
97 | 0 | 0.5 | 1 |
98 | 0.57 | 0.5 | 0.5 |
99 | 0 | 0 | 0 |
100 | 0 | 0 | 0.67 |
101 | 0 | 1 | 0.25 |
102 | 0.5 | 0.5 | 0.44 |
103 | 0 | 1 | 0 |
104 | 0.2 | 0 | 0.14 |
105 | 0 | 1 | 0 |
106 | 0 | 0 | 0 |
107 | 0 | 0.67 | 0.67 |
108 | 0 | 1 | 1 |
109 | 0.63 | 0.5 | 0.56 |
110 | 0 | 0 | 1 |
111 | 0 | 0 | 0 |
112 | 0 | 0 | 0.5 |
113 | 0 | 0 | 0 |
114 | 1 | 1 | 1 |
115 | 0 | 0 | 0.5 |
116 | 0 | 0 | 0.05 |
117 | 0 | 0 | 0 |
118 | 0.5 | 0 | 1 |
119 | 0 | 0.2 | 0.13 |
120 | 0 | 0 | 1 |
121 | 0 | 0 | 0 |
122 | 0.5 | 0 | 0.14 |
123 | 0 | 0.2 | 1 |
124 | 0 | 0 | 0 |
125 | 0 | 1 | 1 |
126 | 0 | 0 | 0 |
127 | 0 | 0 | 0 |
128 | 0 | 0 | 0 |
129 | 0 | 0.1 | 1 |
130 | 0 | 0 | 0 |
131 | 1 | 1 | 1 |
132 | 0 | 0 | 0.5 |
133 | 1 | 1 | 0 |
134 | 1 | 1 | 0.33 |
135 | 0 | 0 | 0 |
136 | 0 | 1 | 1 |
137 | 0 | 0.14 | 0.33 |
138 | 0 | 0.61 | 1 |
139 | 1 | 1 | 1 |
140 | 0.14 | 0 | 1 |
141 | 1 | 0.33 | 1 |
142 | 0 | 0.12 | 0.58 |
143 | 0 | 0 | 0.11 |
144 | 0 | 0 | 0.5 |
145 | 0.04 | 0.17 | 1 |
146 | 0 | 0 | 0 |
147 | 0 | 0.11 | 0 |
148 | 0.5 | 1 | 0.5 |
149 | 1 | 1 | 1 |
150 | 0.5 | 0.25 | 0.63 |
151 | 0 | 0.33 | 1 |
152 | 0.03 | 0.33 | 0.83 |
153 | 0 | 0 | 1 |
154 | 0 | 0.5 | 0 |
155 | 0 | 0 | 1 |
156 | 0.2 | 0 | 0.33 |
157 | 1 | 1 | 1 |
158 | 0 | 0 | 0 |
159 | 0 | 0 | 0 |
160 | 0 | 0 | 0 |
161 | 0 | 0 | 0 |
162 | 1 | 0.5 | 0 |
163 | 0 | 0 | 0.25 |
164 | 0 | 0.25 | 0 |
165 | 0 | 0.2 | 1 |
166 | 0 | 0 | 0 |
167 | 1 | 1 | 1 |
168 | 1 | 0.17 | 0 |
169 | 1 | 1 | 1 |
170 | 0.2 | 0.5 | 1 |
171 | 0 | 0 | 0.11 |
172 | 1 | 1 | 0.5 |
173 | 0.17 | 0 | 1 |
174 | 0.2 | 0 | 0.25 |
175 | 0 | 0.5 | 1 |
176 | 0 | 0.13 | 1 |
177 | 0 | 0 | 1 |
178 | 1 | 0.1 | 0.63 |
179 | 1 | 0 | 0.33 |
180 | 0 | 0 | 1 |
181 | 1 | 0.5 | 1 |
182 | 0 | 0 | 1 |
183 | 0 | 1 | 0.25 |
184 | 0 | 1 | 0 |
185 | 0 | 1 | 1 |
186 | 0 | 0.33 | 0.03 |
187 | 0 | 1 | 0.25 |
188 | 0 | 0 | 1 |
189 | 0 | 1 | 1 |
190 | 0 | 0.06 | 1 |
191 | 1 | 0.17 | 1 |
192 | 0 | 1 | 0 |
Task 1c Summary Results by Query
Simple Hit/Miss Counting [5]
MRR Method [6]
Task 2 Results
Task 2 Overall Results
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
Simple Count | 8.48 | 6.97 | 6.98 |
Total count | 2040 | 2040 | 2020 |
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
---|---|---|---|---|---|
CSJ1 | CSJ2 | 0.8116 | 1.2812 | 1.7509 | TRUE |
CSJ1 | HAFR | 0.8741 | 1.3438 | 1.8134 | TRUE |
CSJ2 | HAFR | -0.4071 | 0.0625 | 0.5321 | FALSE |
File:2009 sqbshtask2 friedman mean ranks.png γçä
Task 2 Summary Results by Query Group
CSJ1 | CSJ2 | HAFR | |
---|---|---|---|
1 | 4.56 | 3.67 | 4.11 |
2 | 2 | 1.75 | 0.25 |
3 | 0 | 0 | 0 |
4 | 3.4 | 3.4 | 2.6 |
5 | 6.6 | 6.6 | 4.8 |
6 | 4 | 2.5 | 1.5 |
7 | 5.8 | 4.4 | 2.2 |
8 | 5.4 | 6.2 | 4 |
9 | 2 | 1.5 | 2.5 |
10 | 4.33 | 2.56 | 3 |
11 | 8.3 | 7.56 | 6.33 |
12 | 9.25 | 7.54 | 6.93 |
13 | 9.22 | 7.57 | 7.35 |
14 | 8.63 | 7.79 | 8.69 |
15 | 4.36 | 3.93 | 3.93 |
16 | 8.43 | 6.35 | 7.38 |
17 | 8.67 | 7.15 | 5.41 |
18 | 8.39 | 6.03 | 8.14 |
19 | 9.22 | 7.94 | 7.69 |
20 | 9.1 | 8.12 | 8.27 |
21 | 8.91 | 6 | 3.26 |
22 | 9.17 | 7.82 | 8.1 |
23 | 8.29 | 6.87 | 4.93 |
24 | 8.9 | 7.61 | 6.86 |
25 | 8.1 | 7.39 | 7.45 |
26 | 7.52 | 4.86 | 6 |
27 | 7.93 | 6.42 | 4.76 |
28 | 8.98 | 7.93 | 7.07 |
29 | 9.04 | 8.03 | 7.31 |
30 | 9.42 | 8.03 | 9.05 |
31 | 9.57 | 8.7 | 7.83 |
32 | 9.59 | 8.08 | 7.84 |
33 | 7.59 | 6.1 | 6.52 |
34 | 9.29 | 7.51 | 8.38 |
35 | 8.74 | 6.43 | 7.5 |
36 | 7.71 | 5.79 | 4.33 |
37 | 7.43 | 5.21 | 6.25 |
38 | 8.28 | 7.11 | 7.7 |
39 | 6.97 | 5.91 | 7.19 |
40 | 8.81 | 6.96 | 6.44 |
41 | 9.15 | 7.14 | 7.59 |
42 | 6.86 | 6.12 | 5.71 |
43 | 8.2 | 5.88 | 6.68 |
44 | 9.19 | 8.11 | 6.56 |
45 | 7.11 | 5.13 | 5.09 |
46 | 7.66 | 5.42 | 6.7 |
47 | 7.92 | 5.69 | 3.38 |
48 | 8.18 | 6 | 7.59 |
Task 2 Summary Results by Query
Runtime Results
Participant | Task | Runtime (min) | Machine |
---|---|---|---|
CSJ1 | 1a | ~55 | BIGWIN |
CSJ1 | 1b | 7 | BIGWIN |
CSJ1 | 2 | ~740 | BIGWIN |
CSJ2 | 1a | ~1800 | BIGWIN |
CSJ2 | 1b | ~210 | BIGWIN |
CSJ2 | 2 | ~730 | BIGWIN |
HAFR | 1a | 1689 | BEER 2 |
HAFR | 1b | 247 | BEER 2 |
HAFR | 2 | 242 | BEER 2 |