Workshop on the Creation of Standardized Test Collections, Tasks, and Metrics for Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation

Greetings Colleagues:

If you have not already done so, please familiarize yourself with the background information found at

http://www.music-ir.org/evaluation/MIR_MDL_evaluation.html

This document intends to provide more specific information about submitting to, and/or participating in, the upcoming Workshop on the Creation of Standardized Test Collections, Tasks, and Metrics for Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation. Joint ACM/IEEE Conference on Digital Libraries, Portland, OR, July 14-18, 2002. The original Workshop outline can be found at: http://www.ohsu.edu/jcdl/ws.html#W4. The Workshop itself will be held Thursday, 18 July 2002 at the JCDL conference venue.

If you have any comments, suggestions or questions please contact me, J. Stephen Downie, at jdownie@uiuc.edu.

General Information:

We are looking for the submission of scholarly "White Papers" or "Position Papers". These papers will be scholarly in the sense that they must conform to the standards of attribution and referencing generally accepted in the academic world. The "White Paper" notion, however, does afford us latitude as to content. Your personal and professional expertise, experiences and judgements are explicitly be sought to help the MIR/MDL communities design and manage an ongoing framework for meaningful and rigourous MIR/MDL evaluation. As such, if you have an opinion or argument, either for, or against, a particular aspect of MIR/MDL evaluation and you can provide sufficient background information (i.e., supporting references) that will help clarify your opinion or argument, then I am almost certain that your submission will be accepted. Remember, the MIR/MDL research communities are multi-disciplinary and this fact should be taken into account as you present your opinions (i.e., provide pointers to background information so that non-subject experts can better appreciate your arguments and world-view).

Open Questions and Topics:

The following, non-exclusive (nor all-encompassing) list of open questions should help you understand just a few of the many possible paper topics:

As a music librarian, are there issues that evaluation standards must address for their results to be credible? Do you know of possible collections that might form the basis of a test collection? What prior research should we be considering?
As a musicologist, what things need examination that are possibly being overlooked?
As digital library (DL) developer, what standards for evaluation should we borrow from the traditional DL community? Any perils or pitfalls that we should consider?
As an audio engineer, what do you need to test your approaches? What methods have worked in other contexts that might or might not work in the MIR/MDL context?
As an information retrieval specialist, what lessons have you learned about other traditional IR evaluation frameworks? Any suggestions about what to avoid or consider as we build our MIR/MDL framework from "scratch"?
As an intellectual property expert, what rights and responsibilities will we have as we strive to build and distribute our test collections?
As an interface/human computer interaction (HCI) expert, what tests should we consider to validate our many different types of interfaces?
As a business person, what format of results will help you make selection decisions? Are there business research models and methods that should be considered?
As a computer scientist, what are the strengths and weaknesses of the CS approach to validation in the MIR/MDL context?
etc.

These are just a few of the possible questions/topics that can be addressed. The underlying questions are:

How do we determine, and then appropriately classify, the tasks that should make up the legitimate purviews of the MIR/MDL domains?
What do we mean by "success"? What do we mean by "failure"?
How will we decide that one MIR/MDL approach works better than another?
How do we best decide which MIR/MDL approach is best suited for a particular task?

Submission Instructions:

Unlike many other Workshops, you need not necessarily attend the Workshop for your submission to be considered for inclusion in the "White Paper" collection. It is much more important that your expertise be heard and considered by those interested in MIR/MDL evaluation issue.

If you plan on attending and/or submitting a "White Paper" please drop me an email at jdownie@uiuc.edu that briefly outlines your intentions at your earliest convenience. I will be structuring Workshop presentation schedules with an eye toward being as inclusive as possible.

Due to email constraints I would appreciate that submissions, if possible, be mounted on the Web and the URL of the submission emailed to me. I or my assistant will then download your submission.

Papers should range between 2000-5000 words.

Deadline for submission: 22 JUNE 2000

Formatting Information:

We will be using the basic ACM-style formatting structure. This is the structure that was used for the 2nd International Symposium on Music Information Retrieval, Bloomington, IN (ISMIR 2001). The formatting instructions and templates can be found at the ISMIR 2001 site http://ismir2001.indiana.edu/template.html. You may use any standard bibliographic reference style including those in the (Author-Date) genre.

Turn around time from submission to printing/mounting on the Web will be VERY TIGHT so I ask that submissions be made in the PDF format if at all possible. Ready-to-go Postscript files are also acceptable *if necessary*. In cases of undue hardship MS-Word or RTF files *may be* considered. Please be considerate of this formatting request: we simply do not have the resources nor time to reformat documents into shape for printing.

music-ir.org is hosted by the ISRL (Information Science Research Laboratories) which is part of GSLIS (the Graduate School of Library and Information Science at UIUC (the University of Illinois at Urbana-Champaign).

J. Stephen Downie
15 May 2002