Difference between revisions of "2009:Multiple Fundamental Frequency Estimation & Tracking Results"

From MIREX Wiki
(Individual Results Files for Task 1)
(Team ID)
Line 8: Line 8:
 
====Team ID====
 
====Team ID====
  
'''BVB''' = [[Nancy Bertin, Emmanuel Vincent, Roland Badeau]]<br />
+
'''BVB''' = [https://music-ir.org/mirex/2009/results/abs/ Nancy Bertin, Emmanuel Vincent, Roland Badeau]<br />
'''DHP1''' = [[Zhiyao Duan, Jinyu Han,Bryan Pardo]]<br />
+
'''DHP1''' = [https://music-ir.org/mirex/2009/results/abs/ Zhiyao Duan, Jinyu Han,Bryan Pardo]<br />
'''DHP2''' = [[Zhiyao Duan, Jinyu Han,Bryan Pardo]]<br />
+
'''DHP2''' = [https://music-ir.org/mirex/2009/results/abs/ Zhiyao Duan, Jinyu Han,Bryan Pardo]<br />
'''NEOS1''' = [[Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]]<br />
+
'''NEOS1''' = [[https://music-ir.org/mirex/2009/results/abs/ Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]<br />
'''NEOS2''' = [[Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]]<br />
+
'''NEOS2''' = [https://music-ir.org/mirex/2009/results/abs/ Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]<br />
'''NPA1''' = [[Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]]<br />
+
'''NPA1''' = [https://music-ir.org/mirex/2009/results/abs/ Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]<br />
'''NPA2''' = [[Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]]<br />
+
'''NPA2''' = [https://music-ir.org/mirex/2009/results/abs/ Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]<br />
'''RS1''' = [[S. A. Racz├╜nski, S. Sagayma]]<br />
+
'''RS1''' = [https://music-ir.org/mirex/2009/results/abs/ S. A. Racz├╜nski, S. Sagayma]<br />
'''RS2''' = [[S. A. Racz├╜nski, S. Sagayma]]<br />
+
'''RS2''' = [https://music-ir.org/mirex/2009/results/abs/ S. A. Racz├╜nski, S. Sagayma]<br />
'''RS3''' = [[S. A. Racz├╜nski, S. Sagayma]]<br />
+
'''RS3''' = [https://music-ir.org/mirex/2009/results/abs/ S. A. Racz├╜nski, S. Sagayma]<br />
'''RS4''' = [[S. A. Racz├╜nski, S. Sagayma]]<br />
+
'''RS4''' = [https://music-ir.org/mirex/2009/results/abs/ S. A. Racz├╜nski, S. Sagayma]<br />
'''RS5''' = [[S. A. Racz├╜nski, S. Sagayma]]<br />
+
'''RS5''' = [https://music-ir.org/mirex/2009/results/abs/ S. A. Racz├╜nski, S. Sagayma]<br />
'''RS6''' = [[S. A. Racz├╜nski, S. Sagayma]]<br />
+
'''RS6''' = [https://music-ir.org/mirex/2009/results/abs/ S. A. Racz├╜nski, S. Sagayma]<br />
'''YR1''' = [[Chunghsin Yeh, Axel Roebel]]<br />
+
'''YR1''' = [https://music-ir.org/mirex/2009/results/abs/ Chunghsin Yeh, Axel Roebel]<br />
'''YR2''' = [[Chunghsin Yeh, Axel Roebel]]<br />
+
'''YR2''' = [https://music-ir.org/mirex/2009/results/abs/ Chunghsin Yeh, Axel Roebel]<br />
 +
'''ZL''' = [https://music-ir.org/mirex/2009/results/abs/zhang_MFF.pdf Xueliang Zhang, Wenju Liu]<br />
  
 
==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==
 
==Task 1: Multiple Fundamental Frequency Estimation (MF0E)==

Revision as of 23:53, 21 October 2009

Introduction

These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the Multiple Fundamental Frequency Estimation & Tracking page.


General Legend

Team ID

BVB = Nancy Bertin, Emmanuel Vincent, Roland Badeau
DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo
DHP2 = Zhiyao Duan, Jinyu Han,Bryan Pardo
NEOS1 = [Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NEOS2 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NPA1 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
NPA2 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
RS1 = S. A. Racz├╜nski, S. Sagayma
RS2 = S. A. Racz├╜nski, S. Sagayma
RS3 = S. A. Racz├╜nski, S. Sagayma
RS4 = S. A. Racz├╜nski, S. Sagayma
RS5 = S. A. Racz├╜nski, S. Sagayma
RS6 = S. A. Racz├╜nski, S. Sagayma
YR1 = Chunghsin Yeh, Axel Roebel
YR2 = Chunghsin Yeh, Axel Roebel
ZL = Xueliang Zhang, Wenju Liu

Task 1: Multiple Fundamental Frequency Estimation (MF0E)

MF0E Overall Summary Results

Below are the average scores across 40 test files. These files come from 3 different sources: woodwind quintet recording of bassoon, clarinet, horn,flute and oboe (UIUC); Rendered MIDI using RWC database donated by IRCAM and a quartet recording of bassoon, clarinet, violin and sax donated by Dr. Bryan Pardo`s Interactive Audio Lab (IAL). 20 files coming from 5 sections of the woodwind recording where each section has 4 files ranging from 2 polyphony to 5 polyphony. 12 files from IAL, coming from 4 different songs ranging from 2 polyphony to 4 polyphony and 8 files from RWC synthesized midi ranging from 2 different songs ranging from 2 polphony to 5 polyphony.


file /nema-raid/www/mirex/results/mf0/est/summary/task1.overall.csv not found

Detailed Results

file /nema-raid/www/mirex/results/mf0/est/summary/task1.results.csv not found

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

file /nema-raid/www/mirex/results/mf0/est/summary/task1.chroma.results.csv not found

Individual Results Files for Task 1

BVB = Nancy Bertin, Emmanuel Vincent, Roland Badeau
DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo
DHP2 = Zhiyao Duan, Jinyu Han,Bryan Pardo
NEOS1 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NEOS2 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NPA1 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
NPA2 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
RS1 = S. A. Racz├╜nski, S. Sagayma
RS2 = S. A. Racz├╜nski, S. Sagayma
RS3 = S. A. Racz├╜nski, S. Sagayma
RS4 = S. A. Racz├╜nski, S. Sagayma
RS5 = S. A. Racz├╜nski, S. Sagayma
RS6 = S. A. Racz├╜nski, S. Sagayma
YR1 = Chunghsin Yeh, Axel Roebel
YR2 = Chunghsin Yeh, Axel Roebel
ZL = Xueliang Zhang, Wenju Liu

Info about the filenames

The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:

bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar

Run Times

file /nema-raid/www/mirex/results/multif0/task1_runtimes.csv not found

TBA

Task 2A:Mixed Set Note Tracking (NT)

NT Mixed Set Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.

A total of 34 files were used in this subtask: 16 from woodwind recording, 8 from IAL quintet recording and 6 piano.

file /nema-raid/www/mirex/results/mf0/nt/summary/task2.overall.results.csv not found

Detailed Results

file /nema-raid/www/mirex/results/mf0/nt/summary/task2.results.csv not found

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

file /nema-raid/www/mirex/results/mf0/nt/summary/task2.chroma.results.csv not found

Results Based on Onset Only

file /nema-raid/www/mirex/results/mf0/nt/summary/task2.onsetOnly.results.csv not found

Chroma Results Based on Onset Only

file /nema-raid/www/mirex/results/mf0/nt/summary/task2.onsetOnly.chroma.results.csv not found

Task 2B:Piano-Only Note Tracking (NT)

NT Piano-Only Overall Summary Results

This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger. 6 piano recordings are evaluated separately for this subtask.

file /nema-raid/www/mirex/results/mf0/nt/summary_piano_subtask/task2_piano.overall.results.csv not found

Detailed Results

file /nema-raid/www/mirex/results/mf0/nt/summary_piano_subtask/task2_piano.results.csv not found

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

file /nema-raid/www/mirex/results/mf0/nt/summary_piano_subtask/task2_piano.chroma.results.csv not found

Results Based on Onset Only

file /nema-raid/www/mirex/results/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.results.csv not found

Chroma Results Based on Onset Only

file /nema-raid/www/mirex/results/mf0/nt/summary_piano_subtask/task2_piano.onsetOnly.chroma.results.csv not found


Individual Results Files for Task 2

BVB = Nancy Bertin, Emmanuel Vincent, Roland Badeau
DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo
DHP2 = Zhiyao Duan, Jinyu Han,Bryan Pardo
NEOS1 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NEOS2 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NPA1 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
NPA2 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
RS1 = S. A. Racz├╜nski, S. Sagayma
RS2 = S. A. Racz├╜nski, S. Sagayma
RS3 = S. A. Racz├╜nski, S. Sagayma
RS4 = S. A. Racz├╜nski, S. Sagayma
RS5 = S. A. Racz├╜nski, S. Sagayma
RS6 = S. A. Racz├╜nski, S. Sagayma
YR = Chunghsin Yeh, Axel Roebel

Info About Filenames

The filenames starting with part* come from acoustic woodwind recording, the ones starting with RWC are synthesized. The piano files are: RA_C030_align.wav, bach_847TESTp.wav, beet_pathetique_3TESTp.wav, mz_333_1TESTp.wav, scn_4TESTp.wav.note, ty_januarTESTp.wav.note. The filenames starting with 01*, 03*, 07*, 09* are coming from the quartet recording.

Run Times

file /nema-raid/www/mirex/results/multif0/task2_runtimes.csv not found TBA

Task 3 Instrument Tracking

Same dataset was used as in Task1. The evaluations were performed by first one-to-one matching the detected contours to the ground-truth contours. This is done by selecting the best scoring duo`s of detected and ground-truth contours. If there are extra detected contours that are not matched to any of the ground-truth contours, all the returned F0`s in those contours are added to false positives. If there are extra ground-truth contours that are not matched to any detected contours, all the F0`s in the ground-truth contours are added to false negatives.


MF0It Detailed Results

file /nema-raid/www/mirex/results/mf0/it/summary/task3.results.csv not found

Detailed Chroma Results

Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)

file /nema-raid/www/mirex/results/mf0/it/summary/task3.chroma.results.csv not found

Individual Results Files for Task 3

DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo

Info About Filenames

The filenames starting with part* come from acoustic woodwind recording, the ones starting with RWC are synthesized. The piano files are: RA_C030_align.wav, bach_847TESTp.wav, beet_pathetique_3TESTp.wav, mz_333_1TESTp.wav, scn_4TESTp.wav.note, ty_januarTESTp.wav.note. The filenames starting with 01*, 03*, 07*, 09* are coming from the quartet recording.

Friedman's Test for Significant Differences

Task 1: Multiple Fundamental Frequency Estimation (MF0E)

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.

Tukey-Kramer HSD Multi-Comparison

file /nema-raid/www/mirex/results/mf0/est/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_TukeyKramerHSD.csv not found

https://music-ir.org/mirex/2009/results/mf0/est/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask1.friedman.Friedman_Mean_Ranks.png

Task 2: Note Tracking

The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the F-measure on individual files.

Tukey-Kramer HSD Multi-Comparison for Task2A

file /nema-raid/www/mirex/results/mf0/nt/summary/Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_TukeyKramerHSD.csv not found

https://music-ir.org/mirex/2009/results/mf0/nt/summary/small.Accuracy_Per_Song_Friedman_Mean_Rankstask2.friedman.Friedman_Mean_Ranks.png

Tukey-Kramer HSD Multi-Comparison for Task2B (Piano)

file /nema-raid/www/mirex/results/mf0/nt/summary_piano_subtask/Accuracy_Per_Song_Friedman_Mean_Rankstask2_piano.friedman.Friedman_TukeyKramerHSD.csv not found

https://www.music-ir.org/mirex/2009/results/mf0/nt/summary_piano_subtask/small.Accuracy_Per_Song_Friedman_Mean_Rankstask2_piano.friedman.Friedman_Mean_Ranks.png