2017:Multiple Fundamental Frequency Estimation & Tracking Results - MIREX Dataset

From MIREX Wiki
Revision as of 16:26, 17 October 2017 by Yun Hao (talk | contribs) (Created page with "==Introduction== These are the results for the 2017 running of the Multiple Fundamental Frequency Estimation and Tracking task on MIREX dataset. For background information about ...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Introduction

These are the results for the 2017 running of the Multiple Fundamental Frequency Estimation and Tracking task on MIREX dataset. For background information about this task set please refer to the 2016:Multiple Fundamental Frequency Estimation & Tracking page.

General Legend

Sub code Submission name Abstract Contributors
CB1 Silvet PDF Chris Cannam, Emmanouil Benetos
CB2 Silvet Live PDF Chris Cannam, Emmanouil Benetos
KD1 multiF0_sampled PDF Karin Dressler
KD2 multiF0_midi PDF Karin Dressler
MHMTM1 End-to-End Multi-instrumental ConvNet PDF [Gaku Hatanaka, Shinjiro Mita, Alexis Meneses, Daiki Miura, Nattapong Thammasan
MHMTM2 Ensemble category ConvNet to F0 ConvNet PDF [Gaku Hatanaka, Shinjiro Mita, Alexis Meneses, Daiki Miura, Nattapong Thammasan
PR1 LPCR PDF Leonid Pogorelyuk, Clarence Rowley
PRGR1 SOT MFFE&T 901 PDF Katarzyna Rokicka, Adam Pluta, Rafal Rokicki, Marcin Gawrysz
PRGR2 SOT MFFE&T 902 PDF Katarzyna Rokicka, Adam Pluta, Rafal Rokicki, Marcin Gawrysz
THK1 Spectral Convolutions PDF John Thickstun, Sham Kakade, Zaid Harchaoui
WCS1 Piano_Transcription PDF Li Su, Derek Wu, Berlin Chen
ZCY2 Multiple pitch estimation PDF Fuliang Yin, Weiwei Zhang, Zhe Chen

Task 1: Multiple Fundamental Frequency Estimation (MF0E)

MF0E Overall Summary Results

Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.

Detailed Results

Precision Recall Accuracy Etot Esubs Emiss Efa
CB1.results.task1 0.804 0.519 0.498 0.529 0.093 0.389 0.047
CB2.results.task1 0.656 0.460 0.420 0.636 0.174 0.367 0.095
KD1.results.task1 0.724 0.811 0.669 0.419 0.104 0.085 0.231
KD2.results.task1 0.724 0.811 0.669 0.419 0.104 0.085 0.231
MHMTM1.results.task1 0.727 0.782 0.655 0.441 0.119 0.099 0.223
MHMTM2.results.task1 0.391 0.200 0.190 0.860 0.288 0.512 0.060
PR1.results.task1 0.487 0.584 0.418 0.926 0.237 0.178 0.511
PRGR1.results.task1 0.603 0.443 0.408 0.646 0.225 0.332 0.089
PRGR2.results.task1 0.514 0.681 0.476 0.781 0.216 0.103 0.462
THK1.results.task1 0.822 0.789 0.720 0.316 0.076 0.136 0.104
WCS1.results.task1 0.640 0.806 0.593 0.569 0.101 0.094 0.375
ZCY2.results.task1 0.627 0.562 0.506 0.601 0.236 0.203 0.163

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Accuracy Etot Esubs Emiss Efa
CB1.results.task1 0.851 0.551 0.527 0.497 0.062 0.389 0.047
CB2.results.task1 0.747 0.527 0.479 0.568 0.106 0.367 0.095
KD1.results.task1 0.751 0.842 0.694 0.389 0.074 0.085 0.231
KD2.results.task1 0.751 0.842 0.694 0.389 0.074 0.085 0.231
MHMTM1.results.task1 0.758 0.817 0.682 0.406 0.084 0.099 0.223
MHMTM2.results.task1 0.602 0.314 0.295 0.746 0.174 0.512 0.060
PR1.results.task1 0.569 0.689 0.489 0.822 0.132 0.178 0.511
PRGR1.results.task1 0.712 0.522 0.479 0.567 0.146 0.332 0.089
PRGR2.results.task1 0.595 0.783 0.547 0.680 0.115 0.103 0.462
THK1.results.task1 0.833 0.800 0.730 0.304 0.064 0.136 0.104
WCS1.results.task1 0.660 0.831 0.612 0.544 0.075 0.094 0.375
ZCY2.results.task1 0.740 0.670 0.600 0.493 0.128 0.203 0.163

download these results as csv

Individual Results Files for Task 1

CB1= Chris Cannam, Emmanouil Benetos
CB2= Chris Cannam, Emmanouil Benetos
KD1= Karin Dressler
KD2= Karin Dressler
MHMTM1= Gaku Hatanaka, Shinjiro Mita, Alexis Meneses, Daiki Miura, Nattapong Thammasan
MHMTM2= Gaku Hatanaka, Shinjiro Mita, Alexis Meneses, Daiki Miura, Nattapong Thammasan
PR1= Leonid Pogorelyuk, Clarence Rowley
PRGR1= Katarzyna Rokicka, Adam Pluta, Rafal Rokicki, Marcin Gawrysz
PRGR2= Katarzyna Rokicka, Adam Pluta, Rafal Rokicki, Marcin Gawrysz
THK1= John Thickstun, Sham Kakadu, Zaid Harchaoui
WCS1= Li Su, Derek Wu, Berlin Chen
ZCY2= Fuliang Yin, Weiwei Zhang, Zhe Chen


Info about the filenames

The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar

Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

Tukey-Kramer HSD Multi-Comparison

TeamID TeamID Lowerbound Mean Upperbound Significance
MM1 CB1 -0.0916 0.6500 1.3916 FALSE
MM1 DT1 0.5334 1.2750 2.0166 TRUE
MM1 CB2 1.1334 1.8750 2.6166 TRUE
CB1 DT1 -0.1166 0.6250 1.3666 FALSE
CB1 CB2 0.4834 1.2250 1.9666 TRUE
DT1 CB2 -0.1416 0.6000 1.3416 FALSE

download these results as csv

2016 Accuracy Per Song Friedman Mean Rankstask1.friedman.Friedman Mean Ranks.png

Task 2:Note Tracking (NT)

NT Mixed Set Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

CB1 CB2 DT1 MM1
Ave. F-Measure Onset-Offset 0.3045 0.2061 0.4053 0.3518
Ave. F-Measure Onset Only 0.5027 0.3734 0.7118 0.6184
Ave. F-Measure Chroma 0.3206 0.2360 0.4162 0.3701
Ave. F-Measure Onset Only Chroma 0.5340 0.4268 0.7261 0.6475

download these results as csv

Detailed Results

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.312 0.304 0.305 0.865
CB2 0.200 0.230 0.206 0.862
DT1 0.439 0.379 0.405 0.852
MM1 0.384 0.331 0.352 0.877

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.328 0.320 0.321 0.860
CB2 0.228 0.265 0.236 0.858
DT1 0.451 0.390 0.416 0.852
MM1 0.404 0.349 0.370 0.874

download these results as csv


Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.524 0.493 0.503 0.720
CB2 0.374 0.403 0.373 0.677
DT1 0.768 0.668 0.712 0.676
MM1 0.674 0.583 0.618 0.674

download these results as csv

Chroma Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.557 0.524 0.534 0.700
CB2 0.424 0.465 0.427 0.652
DT1 0.784 0.681 0.726 0.671
MM1 0.706 0.611 0.648 0.661

download these results as csv


Friedman Tests for Note Tracking

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

Tukey-Kramer HSD Multi-Comparison for Task2
TeamID TeamID Lowerbound Mean Upperbound Significance
DT1 MM1 0.1074 0.9118 1.7162 TRUE
DT1 CB1 0.9897 1.7941 2.5985 TRUE
DT1 CB2 2.0191 2.8235 3.6279 TRUE
MM1 CB1 0.0780 0.8824 1.6867 TRUE
MM1 CB2 1.1074 1.9118 2.7162 TRUE
CB1 CB2 0.2250 1.0294 1.8338 TRUE

download these results as csv

2016 Accuracy Per Song Friedman Mean Rankstask2.onsetOnly.friedman.Friedman Mean Ranks.png

NT Piano-Only Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger. 6 piano recordings are evaluated separately for this subtask.

CB1 CB2 DT1 KB1 MM1
Ave. F-Measure Onset-Offset 0.2378 0.1749 0.5518 0.0245 0.3376
Ave. F-Measure Onset Only 0.6674 0.4967 0.8199 0.4850 0.7537
Ave. F-Measure Chroma 0.2535 0.1862 0.5527 0.0287 0.3185
Ave. F-Measure Onset Only Chroma 0.6779 0.5142 0.8205 0.4965 0.6984

download these results as csv

*Submissions marked by asterisk are with bugs

Detailed Results

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.274 0.211 0.238 0.813
CB2 0.209 0.153 0.175 0.797
DT1 0.618 0.505 0.552 0.796
KB1* 0.020 0.032 0.025 0.118
MM1 0.362 0.317 0.338 0.813

download these results as csv

*Submissions marked by asterisk are with bugs

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.292 0.226 0.254 0.801
CB2 0.221 0.164 0.186 0.796
CT1 0.627 0.627 0.626 0.782
KD2 0.132 0.142 0.137 0.835
PR1 0.199 0.263 0.220 0.817
PRGR1 0.070 0.036 0.046 0.837
PRGR2 0.157 0.103 0.121 0.859
SL1 0.006 0.006 0.006 -0.195
ZCY2 0.020 0.028 0.023 0.697

download these results as csv

*Submissions marked by asterisk are with bugs

Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.744 0.613 0.668 0.585
CB2 0.561 0.454 0.497 0.576
CT1 0.814 0.817 0.814 0.694
KD2 0.690 0.756 0.720 0.559
PR1 0.488 0.687 0.557 0.606
PRGR1 0.639 0.334 0.424 -0.438
PRGR2 0.689 0.493 0.553 0.544
SL1 0.325 0.324 0.312 -0.053
ZCY2 0.142 0.161 0.148 0.453

download these results as csv

*Submissions marked by asterisk are with bugs

Chroma Results Based on Onset Only

Precision Recall Ave. F-measure Ave. Overlap
CB1 0.755 0.623 0.679 0.585
CB2 0.580 0.470 0.514 0.577
CT1 0.815 0.819 0.815 0.676
KD2 0.634 0.696 0.661 0.557
PR1 0.461 0.652 0.528 0.606
PRGR1 0.643 0.341 0.431 -0.557
PRGR2 0.665 0.477 0.533 0.541
SL1 0.368 0.361 0.348 -0.060
ZCY2 0.176 0.203 0.185 0.449

download these results as csv

*Submissions marked by asterisk are with bugs

Individual Results Files for Task 2

DT1= Daylin Troxel
KB1= Rainer Kelz, Sebastian Böck
MM1= Matija Marolt
CB1= Chris Cannam, Emmanouil Benetos
CB2= Chris Cannam, Emmanouil Benetos