Difference between revisions of "2010:Symbolic Music Similarity and Retrieval"

From MIREX Wiki
(Data)
(Description)
Line 3: Line 3:
 
=== Description ===
 
=== Description ===
  
Given a query, each system is supposed to return 10 most melodically similar songs from a given collection.  
+
Given a query, each system is supposed to return 10 most melodically similar songs from a given collection.
 +
 
 +
1. Retrieve the most similar incipits from the UK subset of the RISM A/II collection (about 15,000 incipits), given one of the incipits as a query, and rank them by melodic similarity. Both the query and the collection are monophonic. Half the queries are hummed or whistled queries that have been converted to MIDI, thus with slight rhythmic and pitch imperfections, and half the queries are quantized in pitch and rhythm.
 +
 
 +
2. Like task 1, but with two collections of mostly polyphonic MIDI files to be searched for matches.The songs are are long in duration. The queries would still be monophonic. The first collection would be 10,000 randomly picked MIDI files from a collection of about 60,000 MIDI files that were harvested from the Web. They include different genres (Western and Asian popular music, classical music, ringtones, just to name a few). The second collection would be more focused: about 1000 .kar files (Karaoke MIDI files) with mostly Western popular music which stem from the same web harvest.
  
 
== Data ==
 
== Data ==

Revision as of 17:27, 26 May 2010

Task suggestion: Symbolic Melodic Similarity

Description

Given a query, each system is supposed to return 10 most melodically similar songs from a given collection.

1. Retrieve the most similar incipits from the UK subset of the RISM A/II collection (about 15,000 incipits), given one of the incipits as a query, and rank them by melodic similarity. Both the query and the collection are monophonic. Half the queries are hummed or whistled queries that have been converted to MIDI, thus with slight rhythmic and pitch imperfections, and half the queries are quantized in pitch and rhythm.

2. Like task 1, but with two collections of mostly polyphonic MIDI files to be searched for matches.The songs are are long in duration. The queries would still be monophonic. The first collection would be 10,000 randomly picked MIDI files from a collection of about 60,000 MIDI files that were harvested from the Web. They include different genres (Western and Asian popular music, classical music, ringtones, just to name a few). The second collection would be more focused: about 1000 .kar files (Karaoke MIDI files) with mostly Western popular music which stem from the same web harvest.

Data

3 different datasets are used for 3 subtasks.

  • RISM: monophonic; 10,000 short files
  • Karoke polyphonic; 1,000 long files.
  • Mixed polyphonic; 15,741 long files.

All in MIDI format.

Evaluation

Human Evaluation

The primary evaluation will involve subjective judgments by human evaluators of the retrieved sets using IMIRSEL's Evalutron 6000 system.

  • Evaluator question: Given a search based on track A, the following set of results was returned by all systems. Please place each returned track into one of three classes (not similar, somewhat similar, very similar) and provide an inidcation on a continuous scale of 0 - 10 of high similar the track is to the query.
  • 6/5/6 queries for RISM/Karaoke/Mixed datasets, 10 results per query, 1 set of eyes, ~10 participating labs
  • Higher number of queries preferred as IR research indicates variance is in queries
  • It will be possible for researchers to use this data for other types of system comparisons after MIREX 2007 results have been finalized.
  • Human evaluation to be designed and led by IMIRSEL following a similar format to that used at MIREX 2006
  • Human evaluators will be drawn from the participating labs (and any volunteers from IMIRSEL or on the MIREX lists)


Submission Format

Inputs/Outputs

Input: Parameters:

  • the name of a directory containing the MIDI files
  • the name of one MIDI file containing a monophonic query.

The program will be called 6 times. Three of the queries are going to be quantized (produced from symbolic notation) and three produced by humming or whistling, thus with slight rhythmic and pitch deviations.

Output: - a list of the names of the 10 most similar matching MIDI files, ordered by melodic similarity. Write the file name in separate lines, without empty lines in between.

Measures

Use the same measures as 2006:Symbolic_Melodic_Similarity_Results to compare the search results of the various algorithms.

Packaging submissions

All submissions should be statically linked to all libraries (the presence of dynamically linked libraries cannot be guarenteed).

All submissions should include a README file including the following the information:

  • Command line calling format for all executables and an example formatted set of commands
  • Number of threads/cores used or whether this should be specified on the command line
  • Expected memory footprint
  • Expected runtime
  • Any required environments (and versions), e.g. python, java, bash, matlab.