Difference between revisions of "2006:2006 Plenary Notes"
From MIREX Wiki
| Line 8: | Line 8: | ||
*Appreciation to IMIRSEL team members. | *Appreciation to IMIRSEL team members. | ||
| − | =Overview= | + | ==Overview== |
*This year MIREX is highly successful. We got everything done on time! | *This year MIREX is highly successful. We got everything done on time! | ||
| Line 29: | Line 29: | ||
By tuning the parameters, we can get an optimal setting which is a tradeoff between precision and recall. | By tuning the parameters, we can get an optimal setting which is a tradeoff between precision and recall. | ||
We need new dataset to see if the tuned parameters are good for onseen data. | We need new dataset to see if the tuned parameters are good for onseen data. | ||
| + | Question: comparison to last year results? | ||
| + | Answer: | ||
| + | |||
| + | ==Evalutron 6000== | ||
| + | Two judgments: | ||
| + | *category judgment: Not similar; Similar; Very similar | ||
| + | *continurous score: from 0 to 10, allowing one decimal after the decimal point. | ||
| + | *the system: using CMS open source software | ||
| + | *still have data that we haven't fully processed (other user/evaluator behaviors) | ||
| + | *new evaluation on other facets? e.g. mood | ||
| + | *suggestions? | ||
| + | *appreciate evaluators' volunteer work. Your work makes life beautiful! | ||
Revision as of 15:41, 12 October 2006
Oct. 12th @ Empress Crystal Hall, Victoria
Openning
Professor Stephen Downie gave the openning remarks:
- We will present certificates for participants. Feel free to grab yours if you are leaving.
- Appreciation to IMIRSEL team members.
Overview
- This year MIREX is highly successful. We got everything done on time!
- Matlab is widely used (universal retrieval language!)
- All the evaluation result data files are available on the wiki.
Tasks
- We had sub-tasks as tasks are getting matured.
- New tasks:
- Audio cover song: 13 different songs, each of which has 11 different versions
- Score following: have ground work done for future years
- QBSH: 48 ground truth melodies. Different versions of queries on the 48 melodies. About 2000 noise songs were selected from Essen dataset. Both audio input and MIDI input are supported.
- Please think about new tasks next year.
- New evaluations:
- Evalutron 6000 got real-world human judgment.
- Audio onset detection supported multiple parameters.
- Friedman test: It is valuable experience from TREC conferences, the annual contests in Text Retrieval area.
Onset Detection
By tuning the parameters, we can get an optimal setting which is a tradeoff between precision and recall. We need new dataset to see if the tuned parameters are good for onseen data. Question: comparison to last year results? Answer:
Evalutron 6000
Two judgments:
- category judgment: Not similar; Similar; Very similar
- continurous score: from 0 to 10, allowing one decimal after the decimal point.
- the system: using CMS open source software
- still have data that we haven't fully processed (other user/evaluator behaviors)
- new evaluation on other facets? e.g. mood
- suggestions?
- appreciate evaluators' volunteer work. Your work makes life beautiful!
