Difference between revisions of "2015:Multiple Fundamental Frequency Estimation & Tracking Results - Su Dataset"

From MIREX Wiki
(Chroma Results Based on Onset Only)
(Tukey-Kramer HSD Multi-Comparison for Task2)
Line 127: Line 127:
 
=====Tukey-Kramer HSD Multi-Comparison for Task2=====
 
=====Tukey-Kramer HSD Multi-Comparison for Task2=====
  
<csv>2014/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv</csv>
+
<csv>2015/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv</csv>
  
[[File:2014_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png|500px]]
+
[[File:2015_Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_Mean_Ranks.png|500px]]
  
 
===NT Piano-Only Overall Summary Results===
 
===NT Piano-Only Overall Summary Results===

Revision as of 07:12, 20 October 2015

Introduction

In this year we propose a newly annotated polyphonic dataset. This dataset contains a wider range of real-world music in comparison to the old dataset used from 2009. Specifically, the new dataset contains 3 clips of piano solo, 3 clips of string quartet, 2 clips of piano quintet, and 2 clips of violin sonata (violin with piano accompaniment), all of which are selected from real-world recordings. The length of each clip is between 20 and 30 seconds. The dataset is annotated by the method described in the following paper:

Li Su and Yi-Hsuan Yang, "Escaping from the Abyss of Manual Annotation: New Methodology of Building Polyphonic Datasets for Automatic Music Transcription," in Int. Symp. Computer Music Multidisciplinary Research (CMMR), June 2015.

As also mentioned in the paper, we tried our best to calibrate the errors (mostly the mismatch between onset and offset time stamps) in the preliminary annotation by human labor. Since there are still potential errors of annotation that we didn’t find, we decide to make the data and the annotation publicly available after the announcement of MIREX result this year. Specifically, we encourage every participant to help us check the annotation. The result of each competing algorithm will be updated based on the revised annotation. We hope that this can let the participants get more detailed information about the behaviors of the algorithm performing on the dataset. Moreover, in this way we can join our efforts to create a better dataset for the research on multiple-F0 estimation and tracking.

General Legend

Sub code Submission name Abstract Contributors
BW1 doMultiF0 PDF Emmanouil Benetos, Tillman Weyde
BW2 NoteTracking1 PDF Emmanouil Benetos, Tillman Weyde
BW3 NoteTracking2 PDF Emmanouil Benetos, Tillman Weyde
CB1 Silvet1 PDF Chris Cannam, Emmanouil Benetos, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, and Dan Stowell
CB2 Silvet2 PDF Chris Cannam, Emmanouil Benetos, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy Noland, and Dan Stowell
SY1 MPE1 PDF Li Su, Yi-Hsuan Yang
SY2 MPE2 PDF Li Su, Yi-Hsuan Yang
SY3 MPE3 PDF Li Su, Yi-Hsuan Yang
SY4 MPE4 PDF Li Su, Yi-Hsuan Yang

Task 1: Multiple Fundamental Frequency Estimation (MF0E)

MF0E Overall Summary Results

file /nema-raid/www/mirex/results/2015/mf0/est/summary/task1.overall.csv not found

Detailed Results

file /nema-raid/www/mirex/results/2015/mf0/est/summary/task1.results.csv not found

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

file /nema-raid/www/mirex/results/2015/mf0/est/summary/task1.chroma.results.csv not found

Individual Results Files for Task 1

BW1= Emmanouil Benetos, Tillman Weyde
CB1= Anders Elowsson, Anders Friberg
CB2= Karin Dressler
SY1= Daniel Recoskie, Richard Mann
SY2= Li Su, Yi-Hsuan Yang
SY3= Li Su, Yi-Hsuan Yang
SY4= Li Su, Yi-Hsuan Yang

Info about the filenames

The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar

Run Times

file /nema-raid/www/mirex/results/2014/mf0/est/runtimes_mf0_2015.csv not found

Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

Tukey-Kramer HSD Multi-Comparison

file /nema-raid/www/mirex/results/2015/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv not found

500px

Task 2:Note Tracking (NT)

NT Mixed Set Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

file /nema-raid/www/mirex/results/2015/mf0/nt/summary/task2.overall.results.csv not found

Detailed Results

file /nema-raid/www/mirex/results/2015/mf0/nt/summary/task2.results.csv not found

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

file /nema-raid/www/mirex/results/2015/mf0/nt/summary/task2.chroma.results.csv not found


Results Based on Onset Only

file /nema-raid/www/mirex/results/2015/mf0/nt/summary/task2.onsetOnly.results.csv not found

Chroma Results Based on Onset Only

file /nema-raid/www/mirex/results/2015/mf0/nt/summary/task2.onsetOnly.chroma.results.csv not found

Run Times

file /nema-raid/www/mirex/results/2015/mf0/nt/runtimes_NT_2014.csv not found

Friedman Tests for Note Tracking

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

Tukey-Kramer HSD Multi-Comparison for Task2

file /nema-raid/www/mirex/results/2015/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.onsetOnly.friedman.Friedman_TukeyKramerHSD.csv not found

500px

NT Piano-Only Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger. 6 piano recordings are evaluated separately for this subtask.

BW2 BW3 CB1 CD3 DT1 DT2 DT3 EF1 KD2 RM1 SB5 SY4
Ave. F-Measure Onset-Offset 0.1537 0.2051 0.1850 0.1948 0.1505 0.1742 0.1745 0.2942 0.1719 0.0546 0.0423 0.1337
Ave. F-Measure Onset Only 0.5588 0.6268 0.6635 0.4174 0.3201 0.3834 0.3813 0.8016 0.6778 0.2194 0.6802 0.4963
Ave. F-Measure Chroma 0.1675 0.2176 0.2029 0.2341 0.1574 0.1749 0.1759 0.2737 0.1517 0.0624 0.0443 0.1413
Ave. F-Measure Onset Only Chroma 0.5727 0.6393 0.6747 0.4940 0.3385 0.3914 0.3892 0.7347 0.6165 0.2391 0.6876 0.5233

download these results as csv

Detailed Results

Precision Recall Ave. F-measure Ave. Overlap
BW2 0.163 0.146 0.154 0.837
BW3 0.198 0.213 0.205 0.842
CB1 0.203 0.171 0.185 0.813
CD3 0.187 0.207 0.195 0.728
DT1 0.121 0.222 0.150 0.752
DT2 0.165 0.199 0.174 0.760
DT3 0.163 0.201 0.175 0.759
EF1 0.313 0.278 0.294 0.835
KD2 0.166 0.180 0.172 0.838
RM1 0.050 0.062 0.055 0.799
SB5 0.041 0.044 0.042 0.773
SY4 0.155 0.119 0.134 0.834

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Ave. F-measure Ave. Overlap
BW2 0.178 0.159 0.168 0.828
BW3 0.210 0.226 0.218 0.832
CB1 0.221 0.189 0.203 0.805
CD3 0.226 0.248 0.234 0.859
DT1 0.126 0.239 0.157 0.737
DT2 0.165 0.202 0.175 0.760
DT3 0.163 0.205 0.176 0.753
EF1 0.292 0.259 0.274 0.836
KD2 0.147 0.158 0.152 0.837
RM1 0.057 0.071 0.062 0.804
SB5 0.043 0.046 0.044 0.774
SY4 0.164 0.126 0.141 0.833

download these results as csv

Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
BW2 0.590 0.534 0.559 0.558
BW3 0.602 0.655 0.627 0.583
CB1 0.700 0.636 0.664 0.566
CD3 0.422 0.430 0.417 0.555
DT1 0.253 0.508 0.320 0.553
DT2 0.353 0.461 0.383 0.554
DT3 0.347 0.460 0.381 0.555
EF1 0.845 0.764 0.802 0.618
KD2 0.659 0.701 0.678 0.594
RM1 0.202 0.244 0.219 0.555
SB5 0.640 0.728 0.680 0.408
SY4 0.584 0.437 0.496 0.552

download these results as csv

Chroma Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
BW2 0.605 0.547 0.573 0.559
BW3 0.614 0.668 0.639 0.582
CB1 0.712 0.646 0.675 0.562
CD3 0.506 0.506 0.494 0.546
DT1 0.267 0.538 0.339 0.518
DT2 0.360 0.472 0.391 0.534
DT3 0.354 0.470 0.389 0.533
EF1 0.775 0.700 0.735 0.615
KD2 0.601 0.636 0.616 0.591
RM1 0.220 0.266 0.239 0.537
SB5 0.646 0.736 0.688 0.409
SY4 0.616 0.461 0.523 0.346

download these results as csv

Individual Results Files for Task 2

BW2= Emmanouil Benetos, Tillman Weyde
BW3= Emmanouil Benetos, Tillman Weyde
CB1= Chris Cannam, Emmanouil Benetos
CD3= Andrea Cogliati, Zhiyao Duan
DT1= Zhiyao Duan, David Temperley
DT2= Zhiyao Duan, David Temperley
DT3= Zhiyao Duan, David Temperley
EF1= Anders Elowsson, Anders Friberg
KD2= Karin Dressler
RM1= Daniel Recoskie, Richard Mann
SB5= Sebastian Böck
SY4= Li Su, Yi-Hsuan Yang