2006:2006 Plenary Notes

Oct. 12th @ Empress Crystal Hall, Victoria

Openning

Professor Stephen Downie gave the openning remarks:

• We will present certificates for participants. Feel free to grab yours if you are leaving.
• Appreciation to IMIRSEL team members.

Overview

• This year MIREX is highly successful. We got everything done on time!
• Matlab is widely used (universal retrieval language!)
• All the evaluation result data files are available on the wiki.

Tasks

• We had sub-tasks as tasks are getting matured.
• New tasks:
  • Audio cover song: 13 different songs, each of which has 11 different versions
  • Score following: have ground work done for future years
  • QBSH: 48 ground truth melodies. Different versions of queries on the 48 melodies. About 2000 noise songs were selected from Essen dataset. Both audio input and MIDI input are supported.
• Please think about new tasks next year.
• New evaluations:
  • Evalutron 6000 got real-world human judgment.
  • Audio onset detection supported multiple parameters.
  • Friedman test: It is valuable experience from TREC conferences, the annual contests in Text Retrieval area.

Onset Detection

By tuning the parameters, we can get an optimal setting which is a tradeoff between precision and recall. We need new dataset to see if the tuned parameters are good for onseen data. Question: comparison to last year results? Answer:

Evalutron 6000

Two judgments:

• category judgment: Not similar; Similar; Very similar
• continurous score: from 0 to 10, allowing one decimal after the decimal point.
• the system: using CMS open source software
• still have data that we haven't fully processed (other user/evaluator behaviors)
• new evaluation on other facets? e.g. mood
• suggestions?
• appreciate evaluators' volunteer work. Your work makes life beautiful!

Questions: consistency across users? Answer: the data appear to be quite consistency. More analysis can be done on the data which are publicly assessable.

• automatic evaluation using available metadata (vs human judgment)

Friedman tests

a variation of chi-square test Matlab script code is on the wiki Compare different algorithms this test is conservative