University of Illinois Graduate School of Library and Information Science ISRL
Goal: To retrieve the most similar incipits from the
RISM A/II collection, given one of the incipits as a query.
Dataset: 558 MusicXML files of the incipits from the RISM collection. 11 incipits of this collection are chosen as queries. These MusicXML files were converted to MIDI files.
The scores reported here are averaged over the 11 queries.
| Rank | Participant |
Average Dynamic Recall |
Normalized Recall at Group
Boundaries |
Average precision (non-interpolated) |
Precision at N documents (N is number of relevant document) | Input Data Format | Runtime (s) | Machine | Detailed Evaluation Result |
| 1 | Grachten,
Arcos & Mántaras |
65.98% | 55.24% | 51.72% | 44.33% |
MIDI |
80.174 * | B0 | GAM_eval.txt |
| 2 | Orio, N. | 64.96% | 53.35% |
42.96% |
39.86% | XML | 24.610 | B4 | O_eval.txt |
| 3 | Suyoto
& Uitdenbogerd |
64.18% | 51.79% |
40.42% |
41.72% | MIDI | 48.133 |
B3 |
SU_eval.txt |
| 4 | Typke,
Wiering & Veltkamp |
57.09% | 48.17% |
35.64% |
33.46% | MIDI | 51240 | B4 |
TWV_eval.txt |
| 5 | Lemström, Mikkilä, Mäkinen & Ukkonen (P3) | 55.82% | 46.56% | 41.40% |
39.18% | MIDI | 10.007 * | B0 | LMMU_P3_eval.txt |
| 6 | Lemström,
Mikkilä, Mäkinen & Ukkonen (DP) |
54.27% | 47.26% |
39.91% |
36.20% | MIDI |
10.106 * | B0 | LMMU_DP_eval.txt |
| 7 | Frieler
& Müllensiefen |
51.81% | 45.10% |
33.93% |
33.71% | MIDI |
54.593 |
B4 | FM_eval.txt |
Note * : These runs were executed in M2K environment, and thus the runtime includes evaluation time.