Difference between revisions of "2011:Query-by-Tapping Results"
(→Task 1b Summary Results by Query Group) |
|||
Line 58: | Line 58: | ||
MRR Method | MRR Method | ||
− | <csv p=2>2011/qbt/qbtTask1bMRR</csv> | + | <csv p=2>2011/qbt/qbtTask1bMRR.csv</csv> |
====Task 1b Summary Results by Query ==== | ====Task 1b Summary Results by Query ==== |
Latest revision as of 13:32, 31 October 2011
Contents
Introduction
These are the results for the 2011 running of the Query-by-tappingn task. For background information about this task set please refer to the 2011:Query_by_Tapping page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a symbolic sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Two data sets are used:
- Jang's Dataset Roger Jang's MIR-QBT: This dataset contains both wav files (recorded via microphone) and onset files (human-labeled onset time). 136 ground truth songs with 890 queries.
- Hsiao's Dataset Show Hsiao's QBT_symbolic: This dataset contains only onset files (obtained from the user's tapping on keyboard). 143 ground truth songs with 410 queries.
Task 2 Goto Task 2 Results:The second subtask is the same as last year too. In this subtask, submitted systems take a wave-file sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth, as well as the simple hit(1)/miss(0) counting, is calculated over the top 10 returns. Only Roger Jang's data http://neural.cs.nthu.edu.tw/jang2/dataSet/qbt4public/MIR-QBT.rar MIR-QBT] is used as the other dataset has no wave files.
General Legend
Team ID
Sub code | Submission name | Abstract | Contributors |
---|---|---|---|
CJ1 | QBT | ChunTa Chen, Roger Jang |
Task 1 Results
Task 1a, Jang's Dataset Results
Task 1a Overall Results
CJ1 | |
---|---|
Simple Count | 0.747 |
MRR | 0.599 |
Total Count | 890 |
Task 1a Summary Results by Query Group
Simple Hit/Miss Counting
CJ1 | |
---|---|
1 | 0.85 |
2 | 1.00 |
3 | 1.00 |
4 | 1.00 |
5 | 1.00 |
6 | 0.75 |
7 | 1.00 |
8 | 1.00 |
9 | 0.00 |
10 | 1.00 |
11 | 1.00 |
12 | 1.00 |
13 | 1.00 |
14 | 0.88 |
15 | 1.00 |
16 | 0.80 |
17 | 0.73 |
18 | 0.77 |
19 | 0.25 |
20 | 1.00 |
21 | 0.50 |
22 | 1.00 |
23 | 0.00 |
24 | 1.00 |
25 | 1.00 |
26 | 0.00 |
27 | 0.79 |
28 | 1.00 |
29 | 1.00 |
30 | 0.67 |
31 | 1.00 |
32 | 1.00 |
33 | 0.47 |
34 | 0.92 |
35 | 1.00 |
36 | 0.50 |
37 | 0.00 |
38 | 1.00 |
39 | 1.00 |
40 | 0.93 |
41 | 0.75 |
42 | 1.00 |
43 | 0.70 |
44 | 0.75 |
45 | 0.50 |
46 | 1.00 |
47 | 1.00 |
48 | 0.33 |
49 | 1.00 |
50 | 0.67 |
51 | 0.40 |
52 | 1.00 |
53 | 0.60 |
54 | 1.00 |
55 | 0.93 |
56 | 1.00 |
57 | 1.00 |
58 | 1.00 |
59 | 0.20 |
60 | 0.43 |
61 | 1.00 |
62 | 1.00 |
63 | 1.00 |
64 | 0.50 |
65 | 1.00 |
66 | 1.00 |
67 | 0.00 |
68 | 0.50 |
69 | 0.79 |
70 | 0.50 |
71 | 0.00 |
72 | 0.67 |
73 | 0.86 |
74 | 1.00 |
75 | 0.83 |
76 | 1.00 |
77 | 0.00 |
78 | 0.00 |
79 | 0.50 |
80 | 0.67 |
81 | 1.00 |
82 | 1.00 |
83 | 1.00 |
84 | 1.00 |
85 | 0.71 |
86 | 1.00 |
87 | 0.50 |
88 | 1.00 |
89 | 1.00 |
90 | 0.00 |
91 | 1.00 |
92 | 0.25 |
93 | 1.00 |
94 | 0.33 |
95 | 1.00 |
96 | 1.00 |
97 | 1.00 |
98 | 0.50 |
99 | 0.88 |
100 | 0.75 |
101 | 0.33 |
102 | 0.88 |
103 | 1.00 |
104 | 1.00 |
105 | 0.93 |
106 | 0.50 |
107 | 0.61 |
108 | 0.44 |
109 | 0.94 |
110 | 1.00 |
111 | 1.00 |
112 | 0.62 |
113 | 1.00 |
114 | 1.00 |
115 | 0.40 |
116 | 0.50 |
117 | 0.50 |
118 | 0.00 |
119 | 0.33 |
120 | 0.67 |
121 | 0.33 |
122 | 0.00 |
123 | 0.33 |
124 | 0.33 |
125 | 0.67 |
126 | 0.00 |
127 | 1.00 |
128 | 0.67 |
129 | 0.50 |
130 | 0.50 |
131 | 0.50 |
132 | 0.50 |
133 | 1.00 |
134 | 1.00 |
135 | 1.00 |
136 | 1.00 |
MRR Method
CJ1 | |
---|---|
1 | 0.85 |
2 | 1.00 |
3 | 1.00 |
4 | 1.00 |
5 | 1.00 |
6 | 0.27 |
7 | 1.00 |
8 | 0.75 |
9 | 0.00 |
10 | 0.20 |
11 | 0.62 |
12 | 0.75 |
13 | 0.25 |
14 | 0.88 |
15 | 1.00 |
16 | 0.35 |
17 | 0.67 |
18 | 0.70 |
19 | 0.25 |
20 | 1.00 |
21 | 0.25 |
22 | 0.75 |
23 | 0.00 |
24 | 1.00 |
25 | 0.87 |
26 | 0.00 |
27 | 0.38 |
28 | 0.83 |
29 | 1.00 |
30 | 0.60 |
31 | 0.14 |
32 | 0.78 |
33 | 0.24 |
34 | 0.85 |
35 | 1.00 |
36 | 0.33 |
37 | 0.00 |
38 | 1.00 |
39 | 0.57 |
40 | 0.93 |
41 | 0.58 |
42 | 1.00 |
43 | 0.70 |
44 | 0.56 |
45 | 0.05 |
46 | 1.00 |
47 | 0.38 |
48 | 0.33 |
49 | 0.62 |
50 | 0.50 |
51 | 0.22 |
52 | 1.00 |
53 | 0.37 |
54 | 1.00 |
55 | 0.54 |
56 | 1.00 |
57 | 0.85 |
58 | 1.00 |
59 | 0.02 |
60 | 0.11 |
61 | 0.95 |
62 | 1.00 |
63 | 0.10 |
64 | 0.29 |
65 | 1.00 |
66 | 1.00 |
67 | 0.00 |
68 | 0.47 |
69 | 0.64 |
70 | 0.50 |
71 | 0.00 |
72 | 0.67 |
73 | 0.81 |
74 | 1.00 |
75 | 0.83 |
76 | 0.88 |
77 | 0.00 |
78 | 0.00 |
79 | 0.38 |
80 | 0.67 |
81 | 0.17 |
82 | 1.00 |
83 | 1.00 |
84 | 1.00 |
85 | 0.39 |
86 | 1.00 |
87 | 0.31 |
88 | 1.00 |
89 | 0.73 |
90 | 0.00 |
91 | 1.00 |
92 | 0.12 |
93 | 0.58 |
94 | 0.17 |
95 | 0.89 |
96 | 1.00 |
97 | 1.00 |
98 | 0.50 |
99 | 0.80 |
100 | 0.62 |
101 | 0.05 |
102 | 0.88 |
103 | 1.00 |
104 | 0.91 |
105 | 0.91 |
106 | 0.47 |
107 | 0.42 |
108 | 0.44 |
109 | 0.82 |
110 | 1.00 |
111 | 1.00 |
112 | 0.46 |
113 | 0.58 |
114 | 0.51 |
115 | 0.25 |
116 | 0.50 |
117 | 0.38 |
118 | 0.00 |
119 | 0.33 |
120 | 0.67 |
121 | 0.33 |
122 | 0.00 |
123 | 0.33 |
124 | 0.33 |
125 | 0.12 |
126 | 0.00 |
127 | 1.00 |
128 | 0.67 |
129 | 0.50 |
130 | 0.17 |
131 | 0.50 |
132 | 0.50 |
133 | 0.10 |
134 | 1.00 |
135 | 0.33 |
136 | 0.33 |
Task 1a Summary Results by Query
Simple Hit/Miss Counting [1]
MRR Method [2]
Task 1b, Hsiao's Dataset Results
Task 1b Overall Results
CJ1 | |
---|---|
Simple Count | 0.68 |
MRR | 0.51 |
Total Count | 410.00 |
Task 1b Summary Results by Query Group
Simple Hit/Miss Counting
CJ1 | |
---|---|
1 | 0.50 |
2 | NaN |
3 | 0.75 |
4 | 1.00 |
5 | 1.00 |
6 | 1.00 |
7 | 1.00 |
8 | 1.00 |
9 | 1.00 |
10 | 0.33 |
11 | 1.00 |
12 | 0.50 |
13 | 1.00 |
14 | 0.00 |
15 | NaN |
16 | 1.00 |
17 | 1.00 |
18 | 1.00 |
19 | 1.00 |
20 | 0.33 |
21 | 0.67 |
22 | 1.00 |
23 | 0.83 |
24 | 0.33 |
25 | 0.00 |
26 | 0.00 |
27 | 1.00 |
28 | NaN |
29 | 1.00 |
30 | 1.00 |
31 | NaN |
32 | 1.00 |
33 | 0.50 |
34 | 1.00 |
35 | NaN |
36 | 0.00 |
37 | 0.00 |
38 | 0.29 |
39 | 1.00 |
40 | 0.00 |
41 | 1.00 |
42 | 0.67 |
43 | 1.00 |
44 | 0.00 |
45 | 0.25 |
46 | 0.67 |
47 | 1.00 |
48 | 1.00 |
49 | 0.67 |
50 | 1.00 |
51 | 0.00 |
52 | 1.00 |
53 | 0.89 |
54 | 0.33 |
55 | 0.00 |
56 | 0.50 |
57 | 1.00 |
58 | NaN |
59 | 1.00 |
60 | 1.00 |
61 | 0.75 |
62 | 0.00 |
63 | 1.00 |
64 | NaN |
65 | 0.75 |
66 | NaN |
67 | 0.00 |
68 | NaN |
69 | 0.40 |
70 | 0.00 |
71 | NaN |
72 | 1.00 |
73 | 0.50 |
74 | NaN |
75 | 1.00 |
76 | 0.20 |
77 | NaN |
78 | 1.00 |
79 | 0.25 |
80 | 0.00 |
81 | 0.00 |
82 | 0.00 |
83 | 1.00 |
84 | 1.00 |
85 | 0.00 |
86 | 1.00 |
87 | 1.00 |
88 | 0.40 |
89 | 0.80 |
90 | 0.33 |
91 | 1.00 |
92 | 1.00 |
93 | 0.50 |
94 | 0.00 |
95 | 1.00 |
96 | 1.00 |
97 | 0.00 |
98 | 1.00 |
99 | 1.00 |
100 | 1.00 |
101 | 1.00 |
102 | 1.00 |
103 | NaN |
104 | 1.00 |
105 | 1.00 |
106 | 0.00 |
107 | NaN |
108 | 1.00 |
109 | 1.00 |
110 | 0.00 |
111 | 1.00 |
112 | 0.33 |
113 | 1.00 |
114 | 0.50 |
115 | NaN |
116 | 0.00 |
117 | 1.00 |
118 | 1.00 |
119 | 0.00 |
120 | 0.33 |
121 | 0.67 |
122 | 0.75 |
123 | 0.50 |
124 | 1.00 |
125 | 0.00 |
126 | 1.00 |
127 | 1.00 |
128 | 1.00 |
129 | 1.00 |
130 | 0.00 |
131 | 0.00 |
132 | 1.00 |
133 | 0.00 |
134 | 0.00 |
135 | 1.00 |
136 | 0.50 |
137 | 1.00 |
138 | 0.50 |
139 | 0.40 |
140 | 0.00 |
141 | 0.50 |
142 | 1.00 |
143 | 1.00 |
MRR Method
CJ1 | |
---|---|
1 | 0.50 |
2 | NaN |
3 | 0.50 |
4 | 0.42 |
5 | 1.00 |
6 | 0.50 |
7 | 1.00 |
8 | 1.00 |
9 | 1.00 |
10 | 0.33 |
11 | 1.00 |
12 | 0.12 |
13 | 0.75 |
14 | 0.00 |
15 | NaN |
16 | 1.00 |
17 | 1.00 |
18 | 0.33 |
19 | 0.84 |
20 | 0.33 |
21 | 0.58 |
22 | 1.00 |
23 | 0.71 |
24 | 0.33 |
25 | 0.00 |
26 | 0.00 |
27 | 0.27 |
28 | NaN |
29 | 1.00 |
30 | 1.00 |
31 | NaN |
32 | 1.00 |
33 | 0.25 |
34 | 0.21 |
35 | NaN |
36 | 0.00 |
37 | 0.00 |
38 | 0.03 |
39 | 0.50 |
40 | 0.00 |
41 | 0.33 |
42 | 0.50 |
43 | 0.48 |
44 | 0.00 |
45 | 0.04 |
46 | 0.14 |
47 | 0.33 |
48 | 1.00 |
49 | 0.20 |
50 | 1.00 |
51 | 0.00 |
52 | 0.67 |
53 | 0.69 |
54 | 0.08 |
55 | 0.00 |
56 | 0.25 |
57 | 0.42 |
58 | NaN |
59 | 1.00 |
60 | 0.32 |
61 | 0.27 |
62 | 0.00 |
63 | 0.67 |
64 | NaN |
65 | 0.42 |
66 | NaN |
67 | 0.00 |
68 | NaN |
69 | 0.04 |
70 | 0.00 |
71 | NaN |
72 | 1.00 |
73 | 0.50 |
74 | NaN |
75 | 1.00 |
76 | 0.05 |
77 | NaN |
78 | 0.67 |
79 | 0.03 |
80 | 0.00 |
81 | 0.00 |
82 | 0.00 |
83 | 1.00 |
84 | 0.56 |
85 | 0.00 |
86 | 1.00 |
87 | 1.00 |
88 | 0.40 |
89 | 0.21 |
90 | 0.17 |
91 | 0.75 |
92 | 1.00 |
93 | 0.10 |
94 | 0.00 |
95 | 0.75 |
96 | 1.00 |
97 | 0.00 |
98 | 0.67 |
99 | 0.65 |
100 | 0.62 |
101 | 0.90 |
102 | 1.00 |
103 | NaN |
104 | 1.00 |
105 | 0.71 |
106 | 0.00 |
107 | NaN |
108 | 1.00 |
109 | 0.25 |
110 | 0.00 |
111 | 0.20 |
112 | 0.17 |
113 | 1.00 |
114 | 0.25 |
115 | NaN |
116 | 0.00 |
117 | 1.00 |
118 | 0.83 |
119 | 0.00 |
120 | 0.17 |
121 | 0.19 |
122 | 0.38 |
123 | 0.33 |
124 | 0.90 |
125 | 0.00 |
126 | 1.00 |
127 | 0.50 |
128 | 0.43 |
129 | 1.00 |
130 | 0.00 |
131 | 0.00 |
132 | 1.00 |
133 | 0.00 |
134 | 0.00 |
135 | 0.71 |
136 | 0.25 |
137 | 1.00 |
138 | 0.10 |
139 | 0.23 |
140 | 0.00 |
141 | 0.21 |
142 | 0.44 |
143 | 1.00 |
Task 1b Summary Results by Query
Simple Hit/Miss Counting [3]
MRR Method [4]
Task 2 Results
Task 2 Overall Results
CJ1 | |
---|---|
Simple Count | 0.727 |
MRR | 0.588 |
Total Count | 890 |
Task 2 Summary Results by Query Group
Simple Hit/Miss Counting
CJ1 | |
---|---|
1 | 0.85 |
2 | 1.00 |
3 | 1.00 |
4 | 1.00 |
5 | 1.00 |
6 | 0.75 |
7 | 1.00 |
8 | 1.00 |
9 | 1.00 |
10 | 1.00 |
11 | 1.00 |
12 | 1.00 |
13 | 1.00 |
14 | 0.88 |
15 | 1.00 |
16 | 0.75 |
17 | 0.73 |
18 | 0.69 |
19 | 0.25 |
20 | 1.00 |
21 | 0.50 |
22 | 1.00 |
23 | 0.00 |
24 | 1.00 |
25 | 1.00 |
26 | 0.00 |
27 | 0.71 |
28 | 1.00 |
29 | 1.00 |
30 | 0.67 |
31 | 1.00 |
32 | 1.00 |
33 | 0.37 |
34 | 0.83 |
35 | 1.00 |
36 | 0.75 |
37 | 0.00 |
38 | 1.00 |
39 | 1.00 |
40 | 0.93 |
41 | 0.75 |
42 | 1.00 |
43 | 0.70 |
44 | 0.75 |
45 | 0.50 |
46 | 1.00 |
47 | 1.00 |
48 | 0.33 |
49 | 1.00 |
50 | 0.67 |
51 | 0.40 |
52 | 1.00 |
53 | 0.60 |
54 | 1.00 |
55 | 0.93 |
56 | 1.00 |
57 | 1.00 |
58 | 1.00 |
59 | 0.00 |
60 | 0.43 |
61 | 1.00 |
62 | 0.92 |
63 | 1.00 |
64 | 0.50 |
65 | 1.00 |
66 | 1.00 |
67 | 0.00 |
68 | 0.50 |
69 | 0.76 |
70 | 0.50 |
71 | 0.00 |
72 | 0.67 |
73 | 0.86 |
74 | 1.00 |
75 | 0.83 |
76 | 1.00 |
77 | 0.00 |
78 | 0.00 |
79 | 0.50 |
80 | 0.67 |
81 | 1.00 |
82 | 1.00 |
83 | 1.00 |
84 | 1.00 |
85 | 0.71 |
86 | 1.00 |
87 | 0.50 |
88 | 1.00 |
89 | 1.00 |
90 | 0.00 |
91 | 1.00 |
92 | 0.25 |
93 | 1.00 |
94 | 0.33 |
95 | 1.00 |
96 | 1.00 |
97 | 1.00 |
98 | 0.50 |
99 | 0.81 |
100 | 0.75 |
101 | 0.50 |
102 | 0.88 |
103 | 1.00 |
104 | 0.89 |
105 | 0.93 |
106 | 0.50 |
107 | 0.67 |
108 | 0.44 |
109 | 0.88 |
110 | 0.87 |
111 | 1.00 |
112 | 0.75 |
113 | 0.86 |
114 | 0.80 |
115 | 0.40 |
116 | 0.50 |
117 | 0.50 |
118 | 0.00 |
119 | 0.33 |
120 | 0.67 |
121 | 0.67 |
122 | 0.00 |
123 | 0.33 |
124 | 0.33 |
125 | 0.67 |
126 | 0.00 |
127 | 1.00 |
128 | 0.67 |
129 | 0.50 |
130 | 0.50 |
131 | 1.00 |
132 | 0.50 |
133 | 1.00 |
134 | 1.00 |
135 | 1.00 |
136 | 0.00 |
MRR Method
CJ1 | |
---|---|
1 | 0.85 |
2 | 1.00 |
3 | 1.00 |
4 | 1.00 |
5 | 1.00 |
6 | 0.27 |
7 | 1.00 |
8 | 0.75 |
9 | 0.25 |
10 | 0.20 |
11 | 0.62 |
12 | 0.75 |
13 | 0.25 |
14 | 0.88 |
15 | 1.00 |
16 | 0.33 |
17 | 0.69 |
18 | 0.64 |
19 | 0.25 |
20 | 1.00 |
21 | 0.25 |
22 | 0.75 |
23 | 0.00 |
24 | 1.00 |
25 | 0.87 |
26 | 0.00 |
27 | 0.39 |
28 | 0.83 |
29 | 1.00 |
30 | 0.60 |
31 | 0.14 |
32 | 0.78 |
33 | 0.20 |
34 | 0.77 |
35 | 1.00 |
36 | 0.40 |
37 | 0.00 |
38 | 1.00 |
39 | 0.57 |
40 | 0.87 |
41 | 0.58 |
42 | 1.00 |
43 | 0.70 |
44 | 0.56 |
45 | 0.05 |
46 | 1.00 |
47 | 0.50 |
48 | 0.33 |
49 | 0.62 |
50 | 0.50 |
51 | 0.22 |
52 | 1.00 |
53 | 0.37 |
54 | 1.00 |
55 | 0.51 |
56 | 1.00 |
57 | 0.87 |
58 | 1.00 |
59 | 0.00 |
60 | 0.12 |
61 | 0.95 |
62 | 0.92 |
63 | 0.10 |
64 | 0.29 |
65 | 1.00 |
66 | 1.00 |
67 | 0.00 |
68 | 0.43 |
69 | 0.61 |
70 | 0.50 |
71 | 0.00 |
72 | 0.67 |
73 | 0.81 |
74 | 1.00 |
75 | 0.83 |
76 | 0.88 |
77 | 0.00 |
78 | 0.00 |
79 | 0.38 |
80 | 0.67 |
81 | 0.17 |
82 | 1.00 |
83 | 1.00 |
84 | 1.00 |
85 | 0.38 |
86 | 1.00 |
87 | 0.31 |
88 | 1.00 |
89 | 0.73 |
90 | 0.00 |
91 | 1.00 |
92 | 0.12 |
93 | 0.58 |
94 | 0.25 |
95 | 0.89 |
96 | 1.00 |
97 | 1.00 |
98 | 0.50 |
99 | 0.75 |
100 | 0.62 |
101 | 0.06 |
102 | 0.88 |
103 | 1.00 |
104 | 0.87 |
105 | 0.87 |
106 | 0.44 |
107 | 0.44 |
108 | 0.44 |
109 | 0.73 |
110 | 0.87 |
111 | 1.00 |
112 | 0.59 |
113 | 0.49 |
114 | 0.37 |
115 | 0.40 |
116 | 0.50 |
117 | 0.38 |
118 | 0.00 |
119 | 0.33 |
120 | 0.67 |
121 | 0.67 |
122 | 0.00 |
123 | 0.33 |
124 | 0.33 |
125 | 0.12 |
126 | 0.00 |
127 | 1.00 |
128 | 0.67 |
129 | 0.50 |
130 | 0.50 |
131 | 1.00 |
132 | 0.50 |
133 | 0.10 |
134 | 1.00 |
135 | 1.00 |
136 | 0.00 |
Task 2 Summary Results by Query
Simple Hit/Miss Counting [5]
MRR Method [6]
Runtime Results
file /nema-raid/www/mirex/results/2011/qbt/qbtRunTime.csv not found