Difference between revisions of "2010:Multiple Fundamental Frequency Estimation & Tracking Results"

From MIREX Wiki
(Task 1: Multiple Fundamental Frequency Estimation (MF0E))
Line 137: Line 137:
  
 
TBA
 
TBA
 +
 +
====Friedman tests for  Multiple Fundamental Frequency Estimation (MF0E)====
 +
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.
 +
 +
=====Tukey-Kramer HSD Multi-Comparison=====
 +
 +
<csv>2010/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv</csv>
 +
 +
https://music-ir.org/mirex/results/2009/mf0/est/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png

Revision as of 17:09, 3 August 2010

Introduction

These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the 2009:Multiple Fundamental Frequency Estimation & Tracking page.


General Legend

Sub code Submission name Abstract Contributors
AR1 MultiF0_Yeh_IRCAM 1 Band MF0Est PDF Chunghsin Yeh, Axel Roebel
AR2 MultiF0_Yeh_IRCAM 20 Band MF0Est PDF Chunghsin Yeh, Axel Roebel
AR3 MultiF0_Yeh_IRCAM 1 Band Tracking MF0Est PDF Chunghsin Yeh, Axel Roebel
AR4 MultiF0_Yeh_IRCAM 20 Band Tracking MF0Est PDF Chunghsin Yeh, Axel Roebel
AR5 MultiF0_Yeh_IRCAM 1 Band Note Tracking PDF Chunghsin Yeh, Axel Roebel
AR6 MultiF0_Yeh_IRCAM 20 Band Note Tracking PDF Chunghsin Yeh, Axel Roebel
BD1 Frame-BasedMultiF0 PDF Emmanouil Benetos, Simon Dixon
CRVRC1 MF0UJAV1T1 MultiF0 Est PDF Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
CRVRC2 MF0UJAV1T2 Note Tracking PDF Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
CRVRC3 MF0UJAV2T1 MultiF0 Est PDF Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
CRVRC4 MF0UJAV2T2 Note Tracking PDF Francisco Jesus Canadas Quesada, Francisco Jose Rodriguez Serrano, Pedro Vera-Candeas, Nicolas Ruiz Reyes, Julio José Carabias-Orti
DCL1 Multi-Pitch ND Beta MultiF0 Est PDF Arnaud Dessein, Arshia Cont, Guillaume Lemaitre
DCL2 Multi-Pitch ND Beta Note Tracking PDF Arnaud Dessein, Arshia Cont, Guillaume Lemaitre
DHP1 DuanHanPardo_MultiF0EstTrack MultiF0 Est PDF Zhiyao Duan, Jinyu Han, Bryan Pardo
DHP2 DuanHanPardo_MultiF0EstTrack Note Tracking PDF Zhiyao Duan, Jinyu Han, Bryan Pardo
DHP3 DuanHanPardo_MultiF0EstTrack Instrument Tracking PDF Zhiyao Duan, Jinyu Han, Bryan Pardo
JW1 flexible harmonic model for multipitch estimation MultiF0 Est PDF Jun Wu
JW2 Harmonic Temporal Model for Multiple Pitch Estimation MultiF0 Est PDF Jun Wu
JW3 flexible harmonic model for multipitch estimation Note Tracking PDF Jun Wu
JW4 Harmonic Temporal Model for Multiple Pitch Estimation Note Tracking PDF Jun Wu
LYLC1 CTL_MF0E Multi F0 Est PDF Cheng-Te Lee, Yi-Hsuan Yang, Keng-Sheng Lin, Homer Chen
LYLC2 CTL_MF0E Note Tracking PDF Cheng-Te Lee, Yi-Hsuan Yang, Keng-Sheng Lin, Homer Chen
NNTOS1 MTINS PDF Masahiro Nakano, Tomohiko Nakamura, Iwao Tanuma, Nobutaka Ono, Shigeki Sagayama


Task 1: Multiple Fundamental Frequency Estimation (MF0E)

MF0E Overall Summary Results

Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.


AR1 AR2 AR3 AR4 BD1 CRVRC1 CRVRC3 DCL1 DHP1 JW1 JW2 LYLC1 NNTOS1
Accuracy 0.654 0.655 0.692 0.692 0.468 0.49 0.457 0.457 0.553 0.354 0.361 0.373 0.06
Accuracy Chroma 0.68 0.681 0.71 0.71 0.545 0.544 0.52 0.524 0.594 0.459 0.473 0.457 0.109

download these results as csv

Detailed Results

Precision Recall Accuracy Etot Esubs Emiss Efa
AR1 0.721 0.799 0.654 0.448 0.096 0.105 0.247
AR2 0.722 0.799 0.655 0.446 0.095 0.106 0.245
AR3 0.752 0.829 0.692 0.391 0.084 0.088 0.219
AR4 0.753 0.828 0.692 0.390 0.084 0.088 0.218
BD1 0.716 0.485 0.468 0.558 0.167 0.349 0.043
CRVRC1 0.638 0.556 0.490 0.598 0.198 0.246 0.154
CRVRC3 0.515 0.637 0.457 0.837 0.227 0.136 0.474
DCL1 0.499 0.676 0.457 0.847 0.206 0.119 0.523
DHP1 0.712 0.632 0.553 0.518 0.129 0.239 0.150
JW1 0.369 0.579 0.354 1.146 0.361 0.060 0.725
JW2 0.384 0.541 0.361 1.033 0.382 0.077 0.575
LYLC1 0.584 0.399 0.373 0.681 0.228 0.373 0.080
NNTOS1 0.142 0.061 0.060 0.967 0.277 0.661 0.028

download these results as csv

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

Precision Recall Accuracy Etot Esubs Emiss Efa
AR1 0.750 0.833 0.680 0.415 0.063 0.105 0.247
AR2 0.752 0.832 0.681 0.413 0.063 0.106 0.245
AR3 0.772 0.851 0.710 0.368 0.061 0.088 0.219
AR4 0.772 0.851 0.710 0.367 0.061 0.088 0.218
BD1 0.830 0.567 0.545 0.476 0.085 0.349 0.043
CRVRC1 0.707 0.621 0.544 0.533 0.134 0.246 0.154
CRVRC3 0.587 0.732 0.520 0.742 0.132 0.136 0.474
DCL1 0.574 0.775 0.524 0.748 0.107 0.119 0.523
DHP1 0.766 0.678 0.594 0.471 0.082 0.239 0.150
JW1 0.477 0.767 0.459 0.958 0.172 0.060 0.725
JW2 0.501 0.726 0.473 0.849 0.197 0.077 0.575
LYLC1 0.715 0.490 0.457 0.590 0.137 0.373 0.080
NNTOS1 0.303 0.112 0.109 0.916 0.227 0.661 0.028

download these results as csv

Individual Results Files for Task 1

BVB = Nancy Bertin, Emmanuel Vincent, Roland Badeau
DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo
DHP2 = Zhiyao Duan, Jinyu Han,Bryan Pardo
NEOS1 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NEOS2 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NPA1 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
NPA2 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
RS1 = S. A. Raczyński, S. Sagayma
RS2 = S. A. Raczyński, S. Sagayma
RS3 = S. A. Raczyński, S. Sagayma
RS4 = S. A. Raczyński, S. Sagayma
RS5 = S. A. Raczyński, S. Sagayma
RS6 = S. A. Raczyński, S. Sagayma
YR1 = Chunghsin Yeh, Axel Roebel
YR2 = Chunghsin Yeh, Axel Roebel
ZL = Xueliang Zhang, Wenju Liu

Info about the filenames

The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar

Run Times

file /nema-raid/www/mirex/results/2009/multif0/task1_runtimes.csv not found

TBA

Friedman tests for Multiple Fundamental Frequency Estimation (MF0E)

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

Tukey-Kramer HSD Multi-Comparison
TeamID TeamID Lowerbound Mean Upperbound Significance
AR4 AR3 -2.7597 0.1250 3.0097 FALSE
AR4 AR2 -0.7472 2.1375 5.0222 FALSE
AR4 AR1 -0.3222 2.5625 5.4472 FALSE
AR4 DHP1 1.1153 4.0000 6.8847 TRUE
AR4 CRVRC1 2.3653 5.2500 8.1347 TRUE
AR4 BD1 3.1653 6.0500 8.9347 TRUE
AR4 DCL1 3.4403 6.3250 9.2097 TRUE
AR4 CRVRC3 3.4403 6.3250 9.2097 TRUE
AR4 LYLC1 5.3653 8.2500 11.1347 TRUE
AR4 JW2 5.9153 8.8000 11.6847 TRUE
AR4 JW1 6.0903 8.9750 11.8597 TRUE
AR4 NNTOS1 8.5153 11.4000 14.2847 TRUE
AR3 AR2 -0.8722 2.0125 4.8972 FALSE
AR3 AR1 -0.4472 2.4375 5.3222 FALSE
AR3 DHP1 0.9903 3.8750 6.7597 TRUE
AR3 CRVRC1 2.2403 5.1250 8.0097 TRUE
AR3 BD1 3.0403 5.9250 8.8097 TRUE
AR3 DCL1 3.3153 6.2000 9.0847 TRUE
AR3 CRVRC3 3.3153 6.2000 9.0847 TRUE
AR3 LYLC1 5.2403 8.1250 11.0097 TRUE
AR3 JW2 5.7903 8.6750 11.5597 TRUE
AR3 JW1 5.9653 8.8500 11.7347 TRUE
AR3 NNTOS1 8.3903 11.2750 14.1597 TRUE
AR2 AR1 -2.4597 0.4250 3.3097 FALSE
AR2 DHP1 -1.0222 1.8625 4.7472 FALSE
AR2 CRVRC1 0.2278 3.1125 5.9972 TRUE
AR2 BD1 1.0278 3.9125 6.7972 TRUE
AR2 DCL1 1.3028 4.1875 7.0722 TRUE
AR2 CRVRC3 1.3028 4.1875 7.0722 TRUE
AR2 LYLC1 3.2278 6.1125 8.9972 TRUE
AR2 JW2 3.7778 6.6625 9.5472 TRUE
AR2 JW1 3.9528 6.8375 9.7222 TRUE
AR2 NNTOS1 6.3778 9.2625 12.1472 TRUE
AR1 DHP1 -1.4472 1.4375 4.3222 FALSE
AR1 CRVRC1 -0.1972 2.6875 5.5722 FALSE
AR1 BD1 0.6028 3.4875 6.3722 TRUE
AR1 DCL1 0.8778 3.7625 6.6472 TRUE
AR1 CRVRC3 0.8778 3.7625 6.6472 TRUE
AR1 LYLC1 2.8028 5.6875 8.5722 TRUE
AR1 JW2 3.3528 6.2375 9.1222 TRUE
AR1 JW1 3.5278 6.4125 9.2972 TRUE
AR1 NNTOS1 5.9528 8.8375 11.7222 TRUE
DHP1 CRVRC1 -1.6347 1.2500 4.1347 FALSE
DHP1 BD1 -0.8347 2.0500 4.9347 FALSE
DHP1 DCL1 -0.5597 2.3250 5.2097 FALSE
DHP1 CRVRC3 -0.5597 2.3250 5.2097 FALSE
DHP1 LYLC1 1.3653 4.2500 7.1347 TRUE
DHP1 JW2 1.9153 4.8000 7.6847 TRUE
DHP1 JW1 2.0903 4.9750 7.8597 TRUE
DHP1 NNTOS1 4.5153 7.4000 10.2847 TRUE
CRVRC1 BD1 -2.0847 0.8000 3.6847 FALSE
CRVRC1 DCL1 -1.8097 1.0750 3.9597 FALSE
CRVRC1 CRVRC3 -1.8097 1.0750 3.9597 FALSE
CRVRC1 LYLC1 0.1153 3.0000 5.8847 TRUE
CRVRC1 JW2 0.6653 3.5500 6.4347 TRUE
CRVRC1 JW1 0.8403 3.7250 6.6097 TRUE
CRVRC1 NNTOS1 3.2653 6.1500 9.0347 TRUE
BD1 DCL1 -2.6097 0.2750 3.1597 FALSE
BD1 CRVRC3 -2.6097 0.2750 3.1597 FALSE
BD1 LYLC1 -0.6847 2.2000 5.0847 FALSE
BD1 JW2 -0.1347 2.7500 5.6347 FALSE
BD1 JW1 0.0403 2.9250 5.8097 TRUE
BD1 NNTOS1 2.4653 5.3500 8.2347 TRUE
DCL1 CRVRC3 -2.8847 0.0000 2.8847 FALSE
DCL1 LYLC1 -0.9597 1.9250 4.8097 FALSE
DCL1 JW2 -0.4097 2.4750 5.3597 FALSE
DCL1 JW1 -0.2347 2.6500 5.5347 FALSE
DCL1 NNTOS1 2.1903 5.0750 7.9597 TRUE
CRVRC3 LYLC1 -0.9597 1.9250 4.8097 FALSE
CRVRC3 JW2 -0.4097 2.4750 5.3597 FALSE
CRVRC3 JW1 -0.2347 2.6500 5.5347 FALSE
CRVRC3 NNTOS1 2.1903 5.0750 7.9597 TRUE
LYLC1 JW2 -2.3347 0.5500 3.4347 FALSE
LYLC1 JW1 -2.1597 0.7250 3.6097 FALSE
LYLC1 NNTOS1 0.2653 3.1500 6.0347 TRUE
JW2 JW1 -2.7097 0.1750 3.0597 FALSE
JW2 NNTOS1 -0.2847 2.6000 5.4847 FALSE
JW1 NNTOS1 -0.4597 2.4250 5.3097 FALSE

download these results as csv

https://music-ir.org/mirex/results/2009/mf0/est/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png