2005:Audio Onset Detect
Paul Brossier (Queen Mary) email@example.com
Onset Detection Contest
The aim of this contest is to compare state-of-the-art onset detection algorithms on music recordings. The methods will be evaluated on a large, various and reliably-annotated dataset, composed of sub-datasets grouping files of the same type.
1) Input data Audio format: The data will be monophonic sound files, with the associated onset times and data about the annotation robustness.
- CD-quality (PCM, 16-bit, 44100 Hz)
- single channel (mono)
- the file length is not critical for that task, but 30 seconds max. excerpts would be convenient if we want to have a correct diversity in the dataset. It must be reminded that real-world sounds must be manually annotated (painful and time-consuming task, as pointed by J. Bello at MIREX 2004).
Audio content: The dataset will be subdivided into classes. This idea has been evoked by D. Ellis at last MIREX. The reasons why:
- onset detection are performed in various applications, some of them are dedicated for a single type of signal (ex: segmentation of a single track in a mix, drum transcription, complex mixes databases segmentation...)
- the composition of the entire database can determine the relative rank of the onset detection algorithms. For example, an evaluation of a dataset principally composed of complex mixes will not emphasize an onset detection performing well on solo phrases of bowed strings, but a little less than the others on complex mixes.
- it can show the weak points of the compared methods. I think it is more useful than an evaluation based on an overall success percentage or curve.
Suggestions for such classes: We can define 2 types of subdivisions:
- monophonic instruments solo phrases
- polyphonic instruments solo phrases
- complex mixes
Or, as suggested by Bello and al.:
- pitched percussive instruments phrases
- pitched non-percussive instruments phrases
- non-pitched percussive instruments phrases
- complex mixes
Meta data: Two types of annotation can be provided:
- Manual annotation for the real word sounds. For this type of annotation, our article mentions these potential difficulties:
- Midi score for synthesized sounds or MIDI commanded instruments. They are considered as robust ground-truth.
Notes on annotation: As mentioned above, the sound files will be provided with their onset time annotation. The ground-truth we will define can be critical for the evaluation. For the MIDI commanded instruments, care should be taken to synchronize the MIDI clock and the audio recording clock. For real world sounds, annotation volunteers are needed. The annotations should be cross-validated (errare humanum est). Precise instructions on which events to annotate must be given to the annotators. Some sounds are easy to annotate: isolated notes, percussive instruments, quantized music (techno). It also means that the annotations by several annotators are very close, because the visualizations (signal plot, spectrogram) are clear enough. Other sounds are quite impossible to annotate precisely: legato bowed strings phrases, even more difficult if you add reverb. Slightly broken chords also introduce ambiguities on the number of onsets to mark. In these cases the annotations can be spread, and the annotation precision must be taken into account in the evaluation.How the annotation is taken into account must be precisely defined... my opinion is to discard sound events that are not music notes, for example breathing, key strokes etc..., that are quite frequent in the solo recordings, even if they're detected by most of the onset detection algorithms...
Article and matlab tool for annotation by Pierre Leveau et al.
2) Output data The onset detection algoritms will return onset times in a text file: <Results of evaluated Algo path>/<AudioFileName>_onsets.txt.
- Tampere University of Technnology, Audio Research Group
Ansii Klapuri <firstname.lastname@example.org>
- MIT, MediaLab
Tristan Jehan <email@example.com>
- LAM, France
Pierre Leveau <firstname.lastname@example.org> Laurent Daudet <email@example.com>
- IRCAM, France
Xavier Rodet <firstname.lastname@example.org>
- University of Pompeo Fabra, Multimedia Technology Group
Julien Ricard <email@example.com> Fabien Gouyon <firstname.lastname@example.org>
- Queen Mary College, Centre for Digital Music
Juan Pablo Bello <email@example.com> Paul Brossier <firstname.lastname@example.org>
The detected onset times will be compared with the ground-truth ones. For one onset time detected, if it belongs to a tolerance time-window around it, it is considered as a correct detection. If not, it is a false positive. Evaluation measures:
- percentage of correct detections / false positives (can also be expressed as precision/recall)
- time precision (tolerance from 50 ms to less). For certain file, we can't be much more accurate than 50 ms because of the weak annotation precision. This must be taken into account.
- separate scoring for different instrument types (percussive, strings, winds)
More detailed data:
- percentage of doubled detections
- speed measurements of the algorithms
- scalability to large files
- robustness to noise, loudness
Relevant Test Collections
Possible sources: excerpts of RWC Database, recordings in the labs (MIDI generated or human), upcoming FreeSound database, etc... Some of them have already been cross-annotated. It would be fine that each people owning an already annotated sound onset database details its contents (source of the annotation (MIDI, how many human subjects, etc.). It could give an overview of the amount of onsets we already have, and of from where they come...
Onset detection is a first step towards a number of very important DSP-oriented tasks that are relevant to the MIR community. However I wonder if it is too-low level to be of interest to the wider ISMIR bunch. I think the authors need to justify in clear terms the gains to the MIR community of carrying such an evaluation exercise.
The problem is well defined, however the author needs to take care when defining the task of onset detection for non-percussive events (e.g. bowed onset from a cello) or for non-musical events (e.g. breathing, key strokes that produce transient noise in the signal). Evaluations need to consider these cases.
The list of participants is good. I would add to the list Nick Collins and Stephen Hainsworth from Cambridge U., Chris Duxbury and Samer Abdallah from Queen Mary, and perhaps Chris Raphael from Indiana University.
The evaluation procedures are not clear to me. The current proposal is quite verbose, I will suggest that the author reduces the length of the proposal and makes it more assertive. There seems to be a few different possibilities for evaluation: measuring the precision/recall of the algorithms against a database of hand-labeled onsets (from different genres/instrumentations); measuring the temporal localization of detected onsets against a database of "precisely-labeled" onsets (perhaps from MIDI-generated sounds); measuring the computational complexity of the algorithms; measuring their scalability to large sound files; and measuring their robustness to signal distortion/noise. I think the first three evaluations are a must, and that the last two evaluations will depend on the organizers and the feedback from the contestants. For the first two evaluations, there needs to be a large set of ground truth data. The ground truth could be generated using the semi-automatic tool developed by Leveau et al. Each sound file needs to be cross-annotated by a set of different annotators (5?), such that the variability between the different annotations is used to define the "tolerance window" for each onset. Onsets with too-high variance in their annotation should be discarded for the evaluation (obviously also eliminating from the evaluation the false positives that they might produce). Onsets with very little variance can be used to evaluate the temporal precision of the algorithms. You should expect, for example, percussive onsets in low polyphonies to present low variance in the annotations, while non-percussive onsets in, say, pop music are more likely to present a high variance in their annotations. These observations on the annotated database, could be already of great interest to the community. Additionally, if the evaluated systems output some measure of the reliability of their detections, you should incorporate that into your evaluation procedures. I am not entirely sure how could you do that, so it is probably a matter for discussion within the community.
Regarding the test data, I cannot see why sounds should be monophonic and not polyphonic. Most music is polyphonic and for results to be of interest to the community the test data should contain real-life cases. I will also suggest keeping the use of MIDI sounds to the minimum possible. Separating results by type of onset (e.g. percussive, pop, etc) seems a logical choice, so I agree with the author on that the dataset should comprise music that covers the relevant categories. I personally prefer the classification of onsets according to the context on which they appear: onsets on pitched percussive music (e.g. piano and guitar music), onsets on pitched non-percussive music (e.g. string and brass music, voice or orchestral music), onsets on non-pitched percussive music (e.g. drums) and a combination of the above ("complex mixes", e.g. pop, rock and jazz music, presenting leading instruments such as voice and sax, combined with drums, pianos and bass in the background). I don't think a classification regarding monophonic and polyphonic instruments is that relevant.