Difference between revisions of "2010:Symbolic Music Similarity and Retrieval"
(→Human Evaluation) |
|||
| Line 33: | Line 33: | ||
==Evaluation == | ==Evaluation == | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | The same method for building the ground truth as last year will be used. This method has the advantage that no ground truth needs to be built in advance. After the algorithms have been submitted, their results are pooled for every query, and human evaluators are asked to judge the relevance of the matches for some queries. To make this evaluation feasible, it is important that the algorithms do not only return the names of the matching MIDI files for task 2 and 3, but also where the matching fragment starts and ends in the matching MIDI file. | ||
== Submission Format == | == Submission Format == | ||
Revision as of 17:43, 26 May 2010
Contents
Description
Retrieve the most similar items from a collection of symbolic documents, given a query, and rank them by melodic similarity. The following tasks could be defined this year:
Task 1: Monophonic to monophonic. Both the query and the documents in the collection will be monophonic.
Task 2: Monophonic to polyphonic. The documents will be polyphonic (i.e. can have simultaneous notes), but the query will still be monophonic.
Task 3: Polyphonic to polyphonic. Both the query and documents will be polyphonic.
Task Specific Mailing List
You can subscribe to this list to participate in the discussion.
Data
Task 1
- 5,274 tunes belonging to the Essen folksong collection. The tunes are in standard MIDI file format. Download (< 1 MB)
Tasks 2 and 3
Here are versions of the above collections consisting of equivalent MIDI files without tempo changes. These will be used in the evaluation.
You can check the program that was used to "untempify" the MIDI files.
Evaluation
The same method for building the ground truth as last year will be used. This method has the advantage that no ground truth needs to be built in advance. After the algorithms have been submitted, their results are pooled for every query, and human evaluators are asked to judge the relevance of the matches for some queries. To make this evaluation feasible, it is important that the algorithms do not only return the names of the matching MIDI files for task 2 and 3, but also where the matching fragment starts and ends in the matching MIDI file.
Submission Format
Inputs/Outputs
Task 1:
Input: Parameters:
- the name of a directory containing the MIDI files
- the name of one MIDI file containing a monophonic query.
The program will be called 6 times. Three of the queries are going to be quantized (produced from symbolic notation) and three produced by humming or whistling, thus with slight rhythmic and pitch deviations.
Output: - a list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity. Write the file name in separate lines, without empty lines in between.
Task 2
Task 2: Input: same interface as for task 1, thus the name of the directory with files to be searched and the name of the query. However, the directory will contain either about 10,000 mostly polyphonic MIDI files or 1000 Karaoke files.
Output: a list of the names of 10 different MIDI file names that contain melodically similar musical material, ordered by similarity, plus for each file the time (offset from the beginning in seconds) where the query matches and where the matching bit ends. If the query matches in more than one position, return the position of the most similar match (or any one of them if there is more than one most similar match). If the algorithm does not align the query with the MIDI file at any particular position, just return 0 as start time and the duration of the MIDI file as end time.
Sample output line:
somefile.mid,0,2.3
(means that somefile.mid matches the query, and the matching bit starts at the very beginning of the file and ends 2.3 seconds later). The most similar match should be returned first. In evaluation, the human graders will only here that part of the song +-10 sec buffer in both ends.
Measures
Use the same measures as 2006:Symbolic_Melodic_Similarity_Results to compare the search results of the various algorithms.
Packaging submissions
All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).
All submissions should include a README file including the following the information:
- Command line calling format for all executables and an example formatted set of commands
- Number of threads/cores used or whether this should be specified on the command line
- Expected memory footprint
- Expected runtime
- Any required environments (and versions), e.g. python, java, bash, matlab.
