Establishing Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation Frameworks:
Preliminary Foundations and Infrastructures

Greetings Colleagues:

This background document is presented to you with the hope that it will contextualize for you the strategies and long-term goals of the ongoing effort to establish robust and meaningful evaluation frameworks for MIR and MDL research. Once you have familiarized yourself with its contents, please jump to

http://www.music-ir.org/evaluation/JCDL_Workshop_Info.html

where you find more specific information about "White paper" submissions and/or participation in the Workshop on the Creation of Standardized Test Collections, Tasks, and Metrics for Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation. Joint ACM/IEEE Conference on Digital Libraries, Portland, OR, July 14-18, 2002.

If you have any comments, suggestions or questions please contact me, J. Stephen Downie, at jdownie@uiuc.edu.

Problem Statement:

There is a very strong feeling among the MIR and MDL research communities that we must acquire standardized collections of music (in multiple representations and formats), standardized retrieval tasks (i.e., query sets, etc.), and standardized retrieval and evaluation metrics (please see http://www.music-ir.org/mirbib2/resolution). At the heart of all of this is the very real concern that current MIR/MDL research falls short of being objective and scientific insofar as its work is represented in the literature. One model being put forward is the formation of a TREC-like (see http://trec.nist.gov) set of collections and evaluation criteria. These collections and criteria will have to be designed specifically to address the wide variety of music-specific problems facing the MIR/MDL research communities. Establishing the ideal framework for developing these collections and criteria is the open question to be addressed over the period of this project.

Project Goals:

To establish the infrastructural foundation for the formation of meaningful and comprehensive MIR/MDL evaluation through the identification and/or creation of standardized test collections, retrieval tasks and performance metrics.

Principal Project Components:

There are three principal components to this project

Workshop on the Creation of Standardized Test Collections, Tasks, and Metrics for Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation. Joint ACM/IEEE Conference on Digital Libraries, Portland, OR, July 14-18, 2002. [Hereafter, the "Workshop"] Specifics found at: http://www.ohsu.edu/jcdl/ws.html#W4.
Music Information Retrieval Evaluation Frameworks. (ISMIR 2002, 3rd International Conference on Music Information Retrieval, IRCAM – Centre Pompidou, Paris, France, October 13-17, 2002.) [Hereafter, the "Panel"] Specifics found at: http://ismir2002.ircam.fr/panels.html#panel1.
Information Dissemination and Intelligence Gathering (Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign, June 2002 to January 2003 [tentative])

Project Plans:

JCDL Workshop: Communication will be grounded in the creation of "White papers" or "Position papers". These documents will be scholarly works outlining the thoughts, opinions, and arguments of their authors with regard to their visions of what needs to be included/considered in the creation of standardized MIR/MDL testbed collections, evaluation tasks, and evaluation metrics. I expect (and want) these documents to be highly idiosyncratic: a musicologist will write with respect to the needs and wants of musicologists, a lawyer with respect to legal issues, an audio engineer with regard to audio issues, and so on. It is my goal to attract such a wide range of submissions that the multi-disciplinary approach to MIR /MDL research is truly represented.

Dr. Ellen Vorhees of the National Institute of Standards and Technology has kindly consented to be our keynote speaker on the subject of TREC-like evaluation scenarios.

Goals of the JCDL Workshop include:

The creation of a reference document. That is, the collection of our submitted papers would form the starting-point literature from which future arguments and work in the area of MIR/MDL evaluation could proceed.
A consensus document or "manifesto" outlining the basic framework for further development of collections and evaluation standards would be the "formal" outcome of the workshop itself.
Both documents mentioned above would be made available for all to use as foundational material for the solicitation of research funding, for the establishment of collection creation agreements, and for the creation of ongoing evaluation meetings and "contests."

Panel: The ISMIR 2002 Panel will build upon and extend the findings of the JCDL Workshop. I hope to have this panel session lead to the development of prototype collections, tasks, and metrics that could be unveiled at ISMIR 2003. To this end, participants will include a) those who will be reporting on, and extending, the findings of the JCDL Workshop; and, b) those who were unable to contribute to the JCDL Workshop so new viewpoints and issues may be considered. Submissions, notes and revisions of earlier "White papers" will be incorporated into the preceding JCDL collection, perhaps as Volume II or as a "second edition."

Information Dissemination and Intelligence Gathering: This overarching component will be the "glue" that cements the efforts and results of the other two components together to form the solid foundation for MIR/MDL evaluation.

The dissemination sub-component includes the solicitation, formatting and editing of the "White papers" (from both the JCDL Workshop and the ISMIR Panel) for both print and WWW-based distribution (to be centrally housed at http://www.music-ir.org). It also includes incorporating the results of the discussions undertaken at the meetings and the results of whatever intelligence has been gathered into the main body of documentation.

The intelligence sub-component includes the preliminary identification and analyses of possible extant collections and evaluation issues that might meet or hinder the needs of MIR/MDL evaluation. Issues to be addressed include:

Format availability (i.e., audio, notation, etc.)
Coverage (i.e., genre and/or cultural coverage of the possible collection(s))
Identification of stakeholders (i.e., owners of possible collections, etc.)
Legal issues (i.e., copyright concerns, ownership and dissemination issues, possible licensing scenarios, etc.)
Research issues (i.e., human-subjects concerns, ethics, etc.)
Infrastructural issues (i.e., where to house collections, rights management issues, access issues, etc.)

music-ir.org is hosted by the ISRL (Information Science Research Laboratories) which is part of GSLIS (the Graduate School of Library and Information Science) at UIUC (the University of Illinois at Urbana-Champaign).