University of Illinois · Graduate School of Library and Information Science · ISRL

music-ir.org  >  mirbib  >  Evaluation  >>  White Paper Collection

The MIR/MDL Evaluation Project White Paper Collection

Edition #3

published in 2003


Downloadable pdf files - (pdf files require Adobe Acrobat Reader. Download Adobe Acrobat Reader)

Large Sections which were published incrementally
5.0 M Entire document -- all 132 pages pages i - 129
3.7 M Part 1 -- from the JCDL workshop in Oregon pages 1 - 42
3.8 M Part 2 -- for the panel in Paris pages 43 - 84
5.1 M Part 3 -- for the workshop in Toronto pages 85 - 129

Table of Contents

42k Cover Page cover
12k Table of Contents i - ii
Part I. Papers Presented at the Workshop on the Creation of Standardized Test Collections, Tasks, and Metrics for Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation, 18 July, 2002.
Size Title and Author Pages
15k Who, What, When, Where and Why: Introduction and Acknowledgments (First Edition) 1 - 2
J. Stephen Downie, University of Illinois at Urbana-Champaign
 
26k Establishing Music Information Retrieval (MIR) and Music Digital Library (MDL) Evaluation Frameworks: Preliminary Foundations and Infrastructures 3-6
J. Stephen Downie, University of Illinois at Urbana-Champaign
 
298k Whither Music IR Evaluation Infrastructure: Lessons to be Learned from TREC [Keynote Address] 7 - 13
Ellen M. Voorhees, National Institute of Standards and Technology
 
177k Towards a Workbench for Symbolic Music Information Retrieval 15 - 17
David Bainbridge, University of Waikato
 
167 k User Studies: A First Step in Designing an MIR Testbed 19 - 21
Sally Jo Cunningham, University of Waikato
 
139k Three Criteria for the Evaluation of Music Information Retrieval Techniques Against Collections of Musical Material 23 - 25
Joe Futrelle, University of Illinois at Urbana-Champaign
 
169k Music IR for Music Theory 27 - 30
Eric J. Isaacson, Indiana University
 
116k Common Music Notation as a Source for Music Information Retrieval 31 - 32
Karl MacMillan, Johns Hopkins University
 
179k A Task-Oriented Approach for the Development of a Test Collection for Music Information Retrieval 33 - 35
Massimo Melucci and Nicola Orio, University of Padova
 
133k A MIDI Track for Music IR Evaluation 36 - 37
José A. Montalvo, Indiana University
 
163k Comparing Aural Music-Information Retrieval Systems 39 - 41
Bryan Pardo, Colin Meek and William Birmingham, University of Michigan
 
234k Benchmarking Music Information Retrieval Systems 43 - 48
Josh Reiss and Mark Sandler, Queen Mary, University of London
 
Part II. Panel on Music Information Retrieval Evaluation Frameworks at ISMIR 2002, 17 October, 2002.

129k Interim Report on Establishing MIR/MDL Evaluation Frameworks: Commentary on Consensus Building 43 - 44
J. Stephen Downie, University of Illinois
 
170k Evaluation in Information Retrieval 45 - 49
Edie Rasmussen, University of Pittsburgh
 
161k Music GRID - A Collaborative Virtual Organization for Music Information Retrieval Collaboration and Evaluation 50 - 52
Matthew J. Dovey, Oxford University
 
193k Setting Up an Audio Database for Music Information Retrieval Benchmarking 53 - 55
Perfecto Herrera-Boyer, Universitat Pompeu Fabra
 
234k What is a Sung Query? 56 - 57
Colin Meek, William P. Birmingham, and Bryan Pardo, University of Michigan
 
318k Beyond Recall and Precision: A Full Framework for MIR System Evaluation 58 - 63
Josh Reiss and Mark Sandler, Queen Mary, University of London
 
189k Towards Large Databases for Music Information Retrieval Systems Development and Evaluation 64 - 67
Gaël Richard, Ecole Nationale Supérieure des Télécommunications
 
na A Framework for the Evaluation of Content-Based Music Information Retrieval Using the TREC Paradigm (Extended Abstract) - this item is not avaible online 68 - 70
Stefan Rüger, Imperial College London
 
533k Evaluating a Music Information Retrieval System - TREC Style 71 - 78
Thomas Sødring and Alan F. Smeaton, Dublin City University
 
Part III. Workshop on the Evaluation of Music Information Retrieval (MIR) Systems at SIGIR 2003, 1 August, 2003.
37k MIR/MDL Evaluation: Making Progress 79 - 80
J. Stephen Downie, University of Illinois
 
1.3M Toward Evaluation Techniques for Music Similarity [Keynote Address] 81 - 85
Beth Logan, HP Labs; Daniel P. W. Ellis and Adam Berenzweig, Columbia University
 
325k Open Position: Multilingual Orchestra Conductor. Lifetime Opportunity. 86 - 89
Eloi Batlle, Enric Guaus, and Jaume Masip, Universitat Pompeu Fabra
 
96k Emphasizing the Need for TREC-like Collaboration Towards MIR Evaluation 90 - 96
Shyamala Doraisamy and Stefan M. Rüger, Imperial College London
 
120k If It Sounds As Good As It Looks: Lessons Learned From Video Retrieval Evaluation 97 - 102
Abby A. Goodrum, Syracuse University
 
103k Comparison of User Ratings of Music in Copyright-free Databases and On-the-market CDs 103 - 106
Keiichiro Hoashi, Kazunori Matsumoto, and Naomi Inoue, KDDI R&D Laboratories, Inc.
 
206k Query by Humming: How Good Can It Get? 107 - 109
Bryan Pardo and William P. Birmingham, University of Michigan
 
85k Tracks and Topics: Ideas for Structuring Music Retrieval Test Collections and Avoiding Balkanization 110 - 113
Jeremy Pickens, University of Massachusetts, Amherst
 
133k MIR Benchmarking: Lessons Learned from the Multimedia Community 114 - 120
Josh Reiss and Mark Sandler, University of London
 
50k Appendix A: Chart of Candidate Music IR Test Collections 121 - 123
Don Byrd, Indiana University
 
53k Appendix B: ISMIR 2001 Resolution on MIR Evaluation 124 - 127
(taken from https://www-s.isrl.uiuc.edu/~music-ir/gsdl/cgi-bin/resolution.cgi)
 
337k Appendix C: The TREC-Like Evaluation of Music IR Systems 128 - 129
J. Stephen Downie, University of Illinois at Urbana-Champaign
(published in Proceedings of SIGIR 2003)
 
For more leading-edge research like MIR, see the ISRL webpage.
music-ir.org is hosted by the ISRL (Information Science Research Laboratories) which is part of GSLIS (the Graduate School of Library and Information Science at UIUC (the University of Illinois at Urbana-Champaign).
Maintained by :J Stephen Downie -
Comments to : jdownie at uiuc dot edu
Last modified: 26 July, 2003
music-ir.org

"This material is based upon work supported by the National Science Foundation under Grant No. 0340597."
"Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation."