Difference between revisions of "2009:Multiple Fundamental Frequency Estimation & Tracking Results"
(→Node Tracking (NT)) |
|||
Line 1: | Line 1: | ||
+ | ==Introduction== | ||
+ | These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the [[Multiple Fundamental Frequency Estimation & Tracking]] page. | ||
+ | |||
===General Legend=== | ===General Legend=== | ||
Line 5: | Line 8: | ||
====Team ID==== | ====Team ID==== | ||
− | '''BVB''' = [[Nancy Bertin,Emmanuel Vincent,Roland Badeau]]<br /> | + | '''BVB''' = [[Nancy Bertin, Emmanuel Vincent, Roland Badeau]]<br /> |
− | '''DHP1''' = [[Zhiyao Duan,Jinyu Han,Bryan Pardo]]<br /> | + | '''DHP1''' = [[Zhiyao Duan, Jinyu Han,Bryan Pardo]]<br /> |
− | '''DHP2''' = [[Zhiyao Duan,Jinyu Han,Bryan Pardo]]<br /> | + | '''DHP2''' = [[Zhiyao Duan, Jinyu Han,Bryan Pardo]]<br /> |
'''NEOS1''' = [[Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]]<br /> | '''NEOS1''' = [[Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]]<br /> | ||
'''NEOS2''' = [[Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]]<br /> | '''NEOS2''' = [[Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]]<br /> | ||
− | '''NPA1''' = [[Paolo Nesi | + | '''NPA1''' = [[Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]]<br /> |
− | '''NPA2''' = [[Paolo Nesi | + | '''NPA2''' = [[Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]]<br /> |
'''RS1''' = [[S. A. Racz├╜nski, S. Sagayma]]<br /> | '''RS1''' = [[S. A. Racz├╜nski, S. Sagayma]]<br /> | ||
'''RS2''' = [[S. A. Racz├╜nski, S. Sagayma]]<br /> | '''RS2''' = [[S. A. Racz├╜nski, S. Sagayma]]<br /> | ||
Line 21: | Line 24: | ||
'''YR2''' = [[Chunghsin Yeh, Axel Roebel]]<br /> | '''YR2''' = [[Chunghsin Yeh, Axel Roebel]]<br /> | ||
− | ===Multiple | + | ===Task 1: Multiple Fundamental Frequency Estimation (MF0E)=== |
+ | |||
+ | |||
+ | ===MF0E Overall Summary Results=== | ||
+ | Below are the average scores across 36 test files. These files consisted of 9 groups, each group having 4 files ranging from 2 polyphony to 5 polyphony. 28 real recordings, 8 synthesized from RWC samples. | ||
+ | |||
+ | <csv>mf0/est/summary/task1.overall..csv</csv> | ||
+ | |||
+ | ====Detailed Results==== | ||
+ | |||
+ | <csv>mf0/est/summary/task1.results.csv</csv> | ||
+ | |||
+ | ====Detailed Chroma Results==== | ||
+ | Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | ||
+ | |||
+ | <csv>f0/est/task1.chroma.results.csv</csv> | ||
+ | |||
+ | ====Individual Results Files for Task 1: Scores per Query==== | ||
+ | '''BVB''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/BVB.tar.gz Nancy Bertin, Emmanuel Vincent, Roland Badeau]<br /> | ||
+ | '''DHP1''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/DHP1.tar.gz Zhiyao Duan, Jinyu Han,Bryan Pardo]<br /> | ||
+ | '''DHP2''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/DHP2.tar.gz Zhiyao Duan, Jinyu Han,Bryan Pardo]<br /> | ||
+ | '''NEOS1''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/NEOS1.tar.gz Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]<br /> | ||
+ | '''NEOS2''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/NEOS2.tar.gz Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama]<br /> | ||
+ | '''NPA1''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/NPA1 .tar.gz Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]<br /> | ||
+ | '''NPA2''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/NPA2.tar.gz Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti]<br /> | ||
+ | '''RS1''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/RS1.tar.gz S. A. Racz├╜nski, S. Sagayma]<br /> | ||
+ | '''RS2''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/RS2.tar.gz S. A. Racz├╜nski, S. Sagayma]<br /> | ||
+ | '''RS3''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/RS3.tar.gz S. A. Racz├╜nski, S. Sagayma]<br /> | ||
+ | '''RS4''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/RS4.tar.gz S. A. Racz├╜nski, S. Sagayma]<br /> | ||
+ | '''RS5''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/RS5.tar.gz S. A. Racz├╜nski, S. Sagayma]<br /> | ||
+ | '''RS6''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/RS6.tar.gz S. A. Racz├╜nski, S. Sagayma]<br /> | ||
+ | '''YR1''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/YR1.tar.gz Chunghsin Yeh, Axel Roebel]<br /> | ||
+ | '''YR2''' = [https://music-ir.org/mirex/2009/results/mf0/est/tars/YR2.tar.gz Chunghsin Yeh, Axel Roebel]<br /> | ||
+ | |||
+ | =====Info about the filenames===== | ||
+ | The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are: | ||
+ | |||
+ | '''bs''' = bassoon, | ||
+ | '''cl''' = clarinet, | ||
+ | '''fl''' = flute, | ||
+ | '''hn''' = horn, | ||
+ | '''ob''' = oboe, | ||
+ | '''vl''' = violin, | ||
+ | '''cel''' = cello, | ||
+ | '''gtr''' = guitar, | ||
+ | '''sax''' = saxophone, | ||
+ | '''bass''' = electric bass guitar | ||
+ | |||
+ | ====Run Times==== | ||
+ | <csv>multif0/task1_runtimes.csv</csv> | ||
+ | |||
+ | MG ran on MAC, all other systems ran on ALE Nodes. | ||
+ | |||
+ | ===Overall Summary Results Task 2=== | ||
+ | This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger. | ||
+ | |||
+ | A total of 30 files were used in this task: 16 real recordings, 8 synthesized from RWC samples, and 6 piano. The results below are the average of these 30 files. | ||
+ | |||
+ | <csv>multif0/task2_summary.csv</csv> | ||
+ | |||
+ | ====Detailed Results==== | ||
+ | |||
+ | <csv>multif0/task2_res.csv</csv> | ||
+ | |||
+ | ====Detailed Chroma Results==== | ||
+ | Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating) | ||
+ | |||
+ | <csv>multif0/task2_res_chroma.csv</csv> | ||
+ | |||
+ | ====Results Based on Onset Only==== | ||
+ | |||
+ | <csv>multif0/task2_res_onsetonly.csv</csv> | ||
+ | |||
+ | ====Chroma Results Based on Onset Only==== | ||
+ | |||
+ | <csv>multif0/task2_res_onsetonly_chroma.csv</csv> | ||
+ | |||
+ | ====Piano Subset Results Based on Onset Only==== | ||
+ | |||
+ | <csv>multif0/task2_res_onsetonly_piano.csv</csv> | ||
+ | |||
+ | ====Individual Results Files for Task 2==== | ||
+ | '''EBD1''' = [https://www.music-ir.org/mirex/2008/results/multif0/EBD1task2.tar.gz CV. Emiya, R. Badeau, B. David 1]<br /> | ||
+ | '''EBD2''' = [https://www.music-ir.org/mirex/2008/results/multif0/EBD2task2.tar.gz V. Emiya, R. Badeau, B. David 2]<br /> | ||
+ | '''EOS''' = [https://www.music-ir.org/mirex/2008/results/multif0/EOStask2.tar.gz K. Egashira, N. Ono, S. Sagayama]<br /> | ||
+ | '''PI1''' = [https://www.music-ir.org/mirex/2008/results/multif0/PI1task2.tar.gz A. Pertusa, J. M. I├▒esta 1]<br /> | ||
+ | '''PI2''' = [https://www.music-ir.org/mirex/2008/results/multif0/PI2task2.tar.gz A. Pertusa, J. M. I├▒esta 2]<br /> | ||
+ | '''RFF1''' = [https://www.music-ir.org/mirex/2008/results/multif0/RFF1task2.tar.gz G. Reis, F. Fernandez, A. Ferreira 1]<br /> | ||
+ | '''RFF2''' = [https://www.music-ir.org/mirex/2008/results/multif0/RFF2task2.tar.gz GG. Reis, F. Fernandez, A. Ferreira 2]<br /> | ||
+ | '''RK''' = [https://www.music-ir.org/mirex/2008/results/multif0/RKtask2.tar.gz M. Ryynänen, A. Klapuri]<br /> | ||
+ | '''VBB''' = [https://www.music-ir.org/mirex/2008/results/multif0/VBBtask2.tar.gz E. Vincent, N. Bertin, R. Badeau]<br /> | ||
+ | '''YRC1''' = [https://www.music-ir.org/mirex/2008/results/multif0/YRC1task2.tar.gz C. Yeh, A. Roebel, W-C. Chang 1]<br /> | ||
+ | '''ZR1''' = [https://www.music-ir.org/mirex/2008/results/multif0/ZR1task2.tar.gz R. Zhou, J. D. Reiss 1]<br /> | ||
+ | '''ZR2''' = [https://www.music-ir.org/mirex/2008/results/multif0/ZR2task2.tar.gz R. Zhou, J. D. Reiss 2]<br /> | ||
+ | '''ZR3''' = [https://www.music-ir.org/mirex/2008/results/multif0/ZR3task2.tar.gz R. Zhou, J. D. Reiss 3]<br /> | ||
+ | |||
+ | ======Info About Filenames====== | ||
+ | The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The piano files are: RA_C030_align.wav,bach_847TESTp.wav,beet_pathetique_3TESTp.wav,mz_333_1TESTp.wav,scn_4TESTp.wav.note, ty_januarTESTp.wav.note | ||
+ | |||
+ | ====Run Times==== | ||
+ | <csv>multif0/task2_runtimes.csv</csv> | ||
+ | |||
+ | ZR1,ZR2,ZR3 ran on BLACK. All other systems ran on ALE Nodes. | ||
+ | |||
+ | ===Friedman's Test for Significant Differences=== | ||
+ | |||
+ | ====Task 1==== | ||
+ | The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files. | ||
+ | |||
+ | =====Friedman's Anova Table===== | ||
+ | |||
+ | <csv>multif0/task1.friedman.csv</csv> | ||
+ | |||
+ | =====Tukey-Kramer HSD Multi-Comparison===== | ||
+ | |||
+ | <csv>multif0/task1.friedman.detailed.csv</csv> | ||
+ | |||
+ | [[Image:Multif0.task1.friedman.png]] | ||
+ | |||
+ | ====Task 2==== | ||
+ | The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files. | ||
+ | |||
+ | =====Friedman's Anova Table===== | ||
+ | |||
+ | <csv>multif0/task2.friedman.csv</csv> | ||
+ | |||
+ | =====Tukey-Kramer HSD Multi-Comparison===== | ||
+ | |||
+ | <csv>multif0/task2.friedman.detailed.csv</csv> | ||
+ | |||
+ | [[Image:Multif0.task2.friedman.png]] | ||
+ | |||
+ | [[Category: Results]] | ||
+ | |||
+ | |||
===Note Tracking (NT)=== | ===Note Tracking (NT)=== | ||
===Instrument Tracking (IT)=== | ===Instrument Tracking (IT)=== |
Revision as of 16:24, 16 October 2009
Contents
Introduction
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the Multiple Fundamental Frequency Estimation & Tracking page.
General Legend
Team ID
BVB = Nancy Bertin, Emmanuel Vincent, Roland Badeau
DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo
DHP2 = Zhiyao Duan, Jinyu Han,Bryan Pardo
NEOS1 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NEOS2 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NPA1 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
NPA2 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
RS1 = S. A. Racz├╜nski, S. Sagayma
RS2 = S. A. Racz├╜nski, S. Sagayma
RS3 = S. A. Racz├╜nski, S. Sagayma
RS4 = S. A. Racz├╜nski, S. Sagayma
RS5 = S. A. Racz├╜nski, S. Sagayma
RS6 = S. A. Racz├╜nski, S. Sagayma
YR1 = Chunghsin Yeh, Axel Roebel
YR2 = Chunghsin Yeh, Axel Roebel
Task 1: Multiple Fundamental Frequency Estimation (MF0E)
MF0E Overall Summary Results
Below are the average scores across 36 test files. These files consisted of 9 groups, each group having 4 files ranging from 2 polyphony to 5 polyphony. 28 real recordings, 8 synthesized from RWC samples.
file /nema-raid/www/mirex/results/mf0/est/summary/task1.overall..csv not found
Detailed Results
file /nema-raid/www/mirex/results/mf0/est/summary/task1.results.csv not found
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
file /nema-raid/www/mirex/results/f0/est/task1.chroma.results.csv not found
Individual Results Files for Task 1: Scores per Query
BVB = Nancy Bertin, Emmanuel Vincent, Roland Badeau
DHP1 = Zhiyao Duan, Jinyu Han,Bryan Pardo
DHP2 = Zhiyao Duan, Jinyu Han,Bryan Pardo
NEOS1 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NEOS2 = Masahiro Nakano, Koji Egashira, Nobutaka Ono, Shigeki Sagayama
NPA1 = .tar.gz Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
NPA2 = Paolo Nesi, Gianni Pantaleo, Fabrizio Argenti
RS1 = S. A. Racz├╜nski, S. Sagayma
RS2 = S. A. Racz├╜nski, S. Sagayma
RS3 = S. A. Racz├╜nski, S. Sagayma
RS4 = S. A. Racz├╜nski, S. Sagayma
RS5 = S. A. Racz├╜nski, S. Sagayma
RS6 = S. A. Racz├╜nski, S. Sagayma
YR1 = Chunghsin Yeh, Axel Roebel
YR2 = Chunghsin Yeh, Axel Roebel
Info about the filenames
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:
bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar
Run Times
file /nema-raid/www/mirex/results/multif0/task1_runtimes.csv not found
MG ran on MAC, all other systems ran on ALE Nodes.
Overall Summary Results Task 2
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
A total of 30 files were used in this task: 16 real recordings, 8 synthesized from RWC samples, and 6 piano. The results below are the average of these 30 files.
file /nema-raid/www/mirex/results/multif0/task2_summary.csv not found
Detailed Results
file /nema-raid/www/mirex/results/multif0/task2_res.csv not found
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
file /nema-raid/www/mirex/results/multif0/task2_res_chroma.csv not found
Results Based on Onset Only
file /nema-raid/www/mirex/results/multif0/task2_res_onsetonly.csv not found
Chroma Results Based on Onset Only
file /nema-raid/www/mirex/results/multif0/task2_res_onsetonly_chroma.csv not found
Piano Subset Results Based on Onset Only
file /nema-raid/www/mirex/results/multif0/task2_res_onsetonly_piano.csv not found
Individual Results Files for Task 2
EBD1 = CV. Emiya, R. Badeau, B. David 1
EBD2 = V. Emiya, R. Badeau, B. David 2
EOS = K. Egashira, N. Ono, S. Sagayama
PI1 = A. Pertusa, J. M. I├▒esta 1
PI2 = A. Pertusa, J. M. I├▒esta 2
RFF1 = G. Reis, F. Fernandez, A. Ferreira 1
RFF2 = GG. Reis, F. Fernandez, A. Ferreira 2
RK = M. Ryynänen, A. Klapuri
VBB = E. Vincent, N. Bertin, R. Badeau
YRC1 = C. Yeh, A. Roebel, W-C. Chang 1
ZR1 = R. Zhou, J. D. Reiss 1
ZR2 = R. Zhou, J. D. Reiss 2
ZR3 = R. Zhou, J. D. Reiss 3
Info About Filenames
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The piano files are: RA_C030_align.wav,bach_847TESTp.wav,beet_pathetique_3TESTp.wav,mz_333_1TESTp.wav,scn_4TESTp.wav.note, ty_januarTESTp.wav.note
Run Times
file /nema-raid/www/mirex/results/multif0/task2_runtimes.csv not found
ZR1,ZR2,ZR3 ran on BLACK. All other systems ran on ALE Nodes.
Friedman's Test for Significant Differences
Task 1
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.
Friedman's Anova Table
file /nema-raid/www/mirex/results/multif0/task1.friedman.csv not found
Tukey-Kramer HSD Multi-Comparison
file /nema-raid/www/mirex/results/multif0/task1.friedman.detailed.csv not found
File:Multif0.task1.friedman.png
Task 2
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.
Friedman's Anova Table
file /nema-raid/www/mirex/results/multif0/task2.friedman.csv not found
Tukey-Kramer HSD Multi-Comparison
file /nema-raid/www/mirex/results/multif0/task2.friedman.detailed.csv not found
File:Multif0.task2.friedman.png