2008:Multiple Fundamental Frequency Estimation & Tracking Results
From MIREX Wiki
Contents |
Introduction
These are the results for the 2008 running of the Multiple Fundamental Frequency Estimation and Tracking task. For background information about this task set please refer to the 2008:Multiple Fundamental Frequency Estimation & Tracking page.
General Legend
Team ID
CL1 = C. Cao, M. Li 1
CL2 = C. Cao, M. Li 2
DRD = J-L. Durrieu, G. Richard, B. David
EOS = K. Egashira, N. Ono, S. Sagayama
EBD1 = V. Emiya, R. Badeau, B. David 1
EBD2 = V. Emiya, R. Badeau, B. David 2
MG = M. Groble
PI1 = A. Pertusa, J. M. Iñesta 1
PI2 = A. Pertusa, J. M. Iñesta 2
RFF1 = G. Reis, F. Fernandez, A. Ferreira 1
RFF2 = G. Reis, F. Fernandez, A. Ferreira 2
RK = M. Ryynänen, A. Klapuri
VBB = E. Vincent, N. Bertin, R. Badeau
YRC1 = C. Yeh, A. Roebel, W-C. Chang 1
YRC2 = C. Yeh, A. Roebel, W-C. Chang 2
ZR1 = R. Zhou, J. D. Reiss 1
ZR2 = R. Zhou, J. D. Reiss 2
ZR3 = R. Zhou, J. D. Reiss 3
Overall Summary Results Task 1
Below are the average scores across 36 test files. These files consisted of 9 groups, each group having 4 files ranging from 2 polyphony to 5 polyphony. 28 real recordings, 8 synthesized from RWC samples.
| CL1 | CL2 | DRD | EBD1 | EBD2 | EOS | MG | PI1 | PI2 | RFF1 | RFF2 | RK | VBB | YRC1 | YRC2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy | 0.358 | 0.487 | 0.495 | 0.447 | 0.452 | 0.467 | 0.427 | 0.596 | 0.618 | 0.211 | 0.183 | 0.613 | 0.54 | 0.619 | 0.665 |
| Accuracy Chroma | 0.395 | 0.519 | 0.557 | 0.504 | 0.501 | 0.55 | 0.501 | 0.639 | 0.657 | 0.269 | 0.228 | 0.658 | 0.569 | 0.655 | 0.687 |
Detailed Results
| Precision | Recall | Accuracy | Etot | Esubs | Emiss | Efa | |
|---|---|---|---|---|---|---|---|
| CL1 | 0.358 | 0.763 | 0.358 | 1.68 | 0.236 | 0.001 | 1.443 |
| CL2 | 0.671 | 0.56 | 0.487 | 0.598 | 0.148 | 0.292 | 0.158 |
| DRD | 0.541 | 0.66 | 0.495 | 0.731 | 0.245 | 0.096 | 0.391 |
| EBD1 | 0.674 | 0.498 | 0.447 | 0.629 | 0.161 | 0.341 | 0.127 |
| EBD2 | 0.713 | 0.493 | 0.452 | 0.599 | 0.146 | 0.362 | 0.092 |
| EOS | 0.591 | 0.546 | 0.467 | 0.649 | 0.21 | 0.244 | 0.194 |
| MG | 0.481 | 0.57 | 0.427 | 0.816 | 0.298 | 0.133 | 0.385 |
| PI1 | 0.824 | 0.625 | 0.596 | 0.429 | 0.101 | 0.275 | 0.053 |
| PI2 | 0.832 | 0.647 | 0.618 | 0.406 | 0.096 | 0.257 | 0.053 |
| RFF1 | 0.506 | 0.226 | 0.211 | 0.854 | 0.183 | 0.601 | 0.071 |
| RFF2 | 0.509 | 0.191 | 0.183 | 0.857 | 0.155 | 0.656 | 0.047 |
| RK | 0.698 | 0.719 | 0.613 | 0.464 | 0.151 | 0.13 | 0.183 |
| VBB | 0.714 | 0.615 | 0.54 | 0.544 | 0.118 | 0.267 | 0.159 |
| YRC1 | 0.698 | 0.741 | 0.619 | 0.477 | 0.129 | 0.129 | 0.218 |
| YRC2 | 0.741 | 0.78 | 0.665 | 0.426 | 0.108 | 0.127 | 0.19 |
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
| Precision | Recall | Accuracy | Etot | Esubs | Emiss | Efa | |
|---|---|---|---|---|---|---|---|
| CL1 | 0.395 | 0.837 | 0.395 | 1.606 | 0.162 | 0.001 | 1.443 |
| CL2 | 0.716 | 0.596 | 0.519 | 0.562 | 0.112 | 0.292 | 0.158 |
| DRD | 0.608 | 0.744 | 0.557 | 0.647 | 0.16 | 0.096 | 0.391 |
| EBD1 | 0.76 | 0.564 | 0.504 | 0.564 | 0.096 | 0.341 | 0.127 |
| EBD2 | 0.788 | 0.548 | 0.501 | 0.544 | 0.09 | 0.362 | 0.092 |
| EOS | 0.697 | 0.644 | 0.55 | 0.55 | 0.112 | 0.244 | 0.194 |
| MG | 0.563 | 0.673 | 0.501 | 0.713 | 0.195 | 0.133 | 0.385 |
| PI1 | 0.881 | 0.671 | 0.639 | 0.383 | 0.055 | 0.275 | 0.053 |
| PI2 | 0.884 | 0.689 | 0.657 | 0.364 | 0.054 | 0.257 | 0.053 |
| RFF1 | 0.639 | 0.289 | 0.269 | 0.79 | 0.119 | 0.601 | 0.071 |
| RFF2 | 0.634 | 0.239 | 0.228 | 0.808 | 0.106 | 0.656 | 0.047 |
| RK | 0.75 | 0.771 | 0.658 | 0.412 | 0.098 | 0.13 | 0.183 |
| VBB | 0.754 | 0.648 | 0.569 | 0.511 | 0.085 | 0.267 | 0.159 |
| YRC1 | 0.741 | 0.786 | 0.655 | 0.432 | 0.085 | 0.129 | 0.218 |
| YRC2 | 0.768 | 0.808 | 0.687 | 0.398 | 0.08 | 0.127 | 0.19 |
Individual Results Files for Task 1: Scores per Query
CL1 = C. Cao, M. Li 1
CL2 = C. Cao, M. Li 2
DRD = J-L. Durrieu, G. Richard, B. David
EBD1 = CV. Emiya, R. Badeau, B. David 1
EBD2 = V. Emiya, R. Badeau, B. David 2
EOS = K. Egashira, N. Ono, S. Sagayama
MG = M. Groble
PI1 = A. Pertusa, J. M. Iñesta 1
PI2 = A. Pertusa, J. M. Iñesta 2
RFF1 = G. Reis, F. Fernandez, A. Ferreira 1
RFF2 = GG. Reis, F. Fernandez, A. Ferreira 2
RK = M. Ryynänen, A. Klapuri
VBB = E. Vincent, N. Bertin, R. Badeau
YRC1 = C. Yeh, A. Roebel, W-C. Chang 1
YRC2 = C. Yeh, A. Roebel, W-C. Chang 2
Info about the filenames
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The legend about the instruments are:
bs = bassoon, cl = clarinet, fl = flute, hn = horn, ob = oboe, vl = violin, cel = cello, gtr = guitar, sax = saxophone, bass = electric bass guitar
Run Times
| CL1 | CL2 | DRD | EBD1 | EBD2 | EOS | MG | PI1 | PI2 | RFF1 | RFF2 | RK | VBB | YRC1 | YRC2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Run Time (sec) | 2430 | 2475 | 14502 | 18180 | 22270 | 9328 | 99 | 955 | 792 | 73784 | 70041 | 5058 | 2081 | 57483 | 57483 |
MG ran on MAC, all other systems ran on ALE Nodes.
Overall Summary Results Task 2
This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note`s offset, or within 50ms whichever is larger.
A total of 30 files were used in this task: 16 real recordings, 8 synthesized from RWC samples, and 6 piano. The results below are the average of these 30 files.
| EBD1 | EBD2 | EOS | PI1 | PI2 | RFF1 | RFF2 | RK | VBB | YRC | ZR1 | ZR2 | ZR3 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ave. F-measure (Onset-Offset) | 0.176 | 0.158 | 0.236 | 0.247 | 0.192 | 0.028 | 0.032 | 0.337 | 0.197 | 0.355 | 0.261 | 0.263 | 0.278 |
| Ave. F-measure (Onset-Offset Chroma) | 0.189 | 0.169 | 0.268 | 0.251 | 0.195 | 0.038 | 0.042 | 0.352 | 0.208 | 0.362 | 0.297 | 0.3 | 0.313 |
| Ave. F-measure (Onset Only) | 0.417 | 0.384 | 0.503 | 0.47 | 0.396 | 0.14 | 0.132 | 0.614 | 0.521 | 0.552 | 0.518 | 0.52 | 0.53 |
| Ave. F-measure (Onset Only Chroma) | 0.47 | 0.429 | 0.561 | 0.52 | 0.446 | 0.177 | 0.168 | 0.655 | 0.547 | 0.576 | 0.575 | 0.577 | 0.586 |
Detailed Results
| Precision | Recall | Ave. F-measure | Ave. Overlap | |
|---|---|---|---|---|
| EBD1 | 0.165 | 0.200 | 0.176 | 0.865 |
| EBD2 | 0.153 | 0.178 | 0.158 | 0.845 |
| EOS | 0.228 | 0.255 | 0.236 | 0.856 |
| PI1 | 0.201 | 0.333 | 0.247 | 0.862 |
| PI2 | 0.145 | 0.301 | 0.192 | 0.854 |
| RFF1 | 0.034 | 0.025 | 0.028 | 0.683 |
| RFF2 | 0.037 | 0.030 | 0.032 | 0.645 |
| RK | 0.312 | 0.382 | 0.337 | 0.884 |
| VBB | 0.162 | 0.268 | 0.197 | 0.829 |
| YRC | 0.307 | 0.442 | 0.355 | 0.890 |
| ZR1 | 0.233 | 0.303 | 0.261 | 0.875 |
| ZR2 | 0.236 | 0.306 | 0.263 | 0.874 |
| ZR3 | 0.256 | 0.314 | 0.278 | 0.874 |
Detailed Chroma Results
Here, accuracy is assessed on chroma results (i.e. all F0's are mapped to a single octave before evaluating)
| Precision | Recall | Ave. F-measure | Ave. Overlap | |
|---|---|---|---|---|
| EBD1 | 0.178 | 0.215 | 0.189 | 0.862 |
| EBD2 | 0.163 | 0.192 | 0.169 | 0.848 |
| EOS | 0.259 | 0.292 | 0.268 | 0.855 |
| PI1 | 0.204 | 0.338 | 0.251 | 0.856 |
| PI2 | 0.147 | 0.308 | 0.195 | 0.848 |
| RFF1 | 0.046 | 0.033 | 0.038 | 0.794 |
| RFF2 | 0.050 | 0.039 | 0.042 | 0.757 |
| RK | 0.324 | 0.398 | 0.352 | 0.884 |
| VBB | 0.171 | 0.283 | 0.208 | 0.850 |
| YRC | 0.314 | 0.451 | 0.362 | 0.889 |
| ZR1 | 0.266 | 0.344 | 0.297 | 0.874 |
| ZR2 | 0.269 | 0.349 | 0.300 | 0.874 |
| ZR3 | 0.287 | 0.352 | 0.313 | 0.874 |
Results Based on Onset Only
| Precision | Recall | Ave. F-measure | Ave. Overlap | |
|---|---|---|---|---|
| EBD1 | 0.379 | 0.494 | 0.417 | 0.559 |
| EBD2 | 0.356 | 0.455 | 0.384 | 0.561 |
| EOS | 0.482 | 0.553 | 0.503 | 0.688 |
| PI1 | 0.385 | 0.626 | 0.470 | 0.661 |
| PI2 | 0.297 | 0.624 | 0.396 | 0.612 |
| RFF1 | 0.170 | 0.123 | 0.140 | 0.428 |
| RFF2 | 0.158 | 0.121 | 0.132 | 0.474 |
| RK | 0.578 | 0.678 | 0.614 | 0.699 |
| VBB | 0.439 | 0.679 | 0.521 | 0.622 |
| YRC | 0.471 | 0.698 | 0.552 | 0.734 |
| ZR1 | 0.466 | 0.602 | 0.518 | 0.696 |
| ZR2 | 0.467 | 0.604 | 0.520 | 0.697 |
| ZR3 | 0.486 | 0.600 | 0.530 | 0.701 |
Chroma Results Based on Onset Only
| Precision | Recall | Ave. F-measure | Ave. Overlap | |
|---|---|---|---|---|
| EBD1 | 0.426 | 0.559 | 0.470 | 0.539 |
| EBD2 | 0.396 | 0.513 | 0.429 | 0.530 |
| EOS | 0.536 | 0.617 | 0.561 | 0.676 |
| PI1 | 0.426 | 0.694 | 0.520 | 0.605 |
| PI2 | 0.334 | 0.705 | 0.446 | 0.519 |
| RFF1 | 0.217 | 0.155 | 0.177 | 0.435 |
| RFF2 | 0.203 | 0.155 | 0.168 | 0.478 |
| RK | 0.616 | 0.724 | 0.655 | 0.683 |
| VBB | 0.462 | 0.710 | 0.547 | 0.607 |
| YRC | 0.493 | 0.729 | 0.576 | 0.707 |
| ZR1 | 0.516 | 0.667 | 0.575 | 0.694 |
| ZR2 | 0.519 | 0.671 | 0.577 | 0.696 |
| ZR3 | 0.537 | 0.664 | 0.586 | 0.702 |
Piano Subset Results Based on Onset Only
| Precision | Recall | Ave. F-measure | Ave. Overlap | |
|---|---|---|---|---|
| EBD1 | 0.649 | 0.639 | 0.631 | 0.606 |
| EBD2 | 0.622 | 0.541 | 0.569 | 0.610 |
| EOS | 0.541 | 0.539 | 0.527 | 0.550 |
| PI1 | 0.390 | 0.654 | 0.487 | 0.544 |
| PI2 | 0.279 | 0.704 | 0.398 | 0.486 |
| RFF1 | 0.305 | 0.239 | 0.268 | 0.401 |
| RFF2 | 0.256 | 0.235 | 0.245 | 0.452 |
| RK | 0.720 | 0.669 | 0.692 | 0.606 |
| VBB | 0.535 | 0.733 | 0.609 | 0.531 |
| YRC | 0.389 | 0.709 | 0.499 | 0.543 |
| ZR1 | 0.738 | 0.777 | 0.757 | 0.577 |
| ZR2 | 0.734 | 0.775 | 0.754 | 0.583 |
| ZR3 | 0.743 | 0.744 | 0.743 | 0.605 |
Individual Results Files for Task 2
EBD1 = CV. Emiya, R. Badeau, B. David 1
EBD2 = V. Emiya, R. Badeau, B. David 2
EOS = K. Egashira, N. Ono, S. Sagayama
PI1 = A. Pertusa, J. M. Iñesta 1
PI2 = A. Pertusa, J. M. Iñesta 2
RFF1 = G. Reis, F. Fernandez, A. Ferreira 1
RFF2 = GG. Reis, F. Fernandez, A. Ferreira 2
RK = M. Ryynänen, A. Klapuri
VBB = E. Vincent, N. Bertin, R. Badeau
YRC1 = C. Yeh, A. Roebel, W-C. Chang 1
ZR1 = R. Zhou, J. D. Reiss 1
ZR2 = R. Zhou, J. D. Reiss 2
ZR3 = R. Zhou, J. D. Reiss 3
Info About Filenames
The filenames starting with part* comes from acoustic woodwind recording, the ones starting with RWC are synthesized. The piano files are: RA_C030_align.wav,bach_847TESTp.wav,beet_pathetique_3TESTp.wav,mz_333_1TESTp.wav,scn_4TESTp.wav.note, ty_januarTESTp.wav.note
Run Times
| EBD1 | EBD2 | EOS | PI1 | PI2 | RFF1 | RFF2 | RK | VBB | YRC | ZR1 | ZR2 | ZR3 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Run Time (sec) | 18180 | 22270 | 9328 | 950 | 790 | 73718 | 71360 | 5044 | 2058 | 57483 | 1415 | 1415 | 871 |
ZR1,ZR2,ZR3 ran on BLACK. All other systems ran on ALE Nodes.
Friedman's Test for Significant Differences
Task 1
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 7516.72 | 14 | 536.909 | 375.84 | 0 |
| Error | 2563.28 | 490 | 5.231 | ||
| Total | 10080 | 539 | |||
Tukey-Kramer HSD Multi-Comparison
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| CL1 | CL2 | -6.9913 | -3.4167 | 0.1580 | FALSE |
| CL1 | DRD | -7.3802 | -3.8056 | -0.2309 | TRUE |
| CL1 | EBD1 | -5.3802 | -1.8056 | 1.7691 | FALSE |
| CL1 | EBD2 | -5.7691 | -2.1944 | 1.3802 | FALSE |
| CL1 | EOS | -6.0191 | -2.4444 | 1.1302 | FALSE |
| CL1 | MG | -5.1302 | -1.5556 | 2.0191 | FALSE |
| CL1 | PI1 | -10.2136 | -6.6389 | -3.0642 | TRUE |
| CL1 | PI2 | -11.6580 | -8.0833 | -4.5087 | TRUE |
| CL1 | RFF1 | -1.1580 | 2.4167 | 5.9913 | FALSE |
| CL1 | RFF2 | -0.5747 | 3.0000 | 6.5747 | FALSE |
| CL1 | RK | -11.8524 | -8.2778 | -4.7031 | TRUE |
| CL1 | VBB | -9.0747 | -5.5000 | -1.9253 | TRUE |
| CL1 | YRC1 | -11.2691 | -7.6944 | -4.1198 | TRUE |
| CL1 | YRC2 | -12.9913 | -9.4167 | -5.8420 | TRUE |
| CL2 | DRD | -3.9636 | -0.3889 | 3.1858 | FALSE |
| CL2 | EBD1 | -1.9636 | 1.6111 | 5.1858 | FALSE |
| CL2 | EBD2 | -2.3524 | 1.2222 | 4.7969 | FALSE |
| CL2 | EOS | -2.6024 | 0.9722 | 4.5469 | FALSE |
| CL2 | MG | -1.7136 | 1.8611 | 5.4358 | FALSE |
| CL2 | PI1 | -6.7969 | -3.2222 | 0.3524 | FALSE |
| CL2 | PI2 | -8.2413 | -4.6667 | -1.0920 | TRUE |
| CL2 | RFF1 | 2.2587 | 5.8333 | 9.4080 | TRUE |
| CL2 | RFF2 | 2.8420 | 6.4167 | 9.9913 | TRUE |
| CL2 | RK | -8.4358 | -4.8611 | -1.2864 | TRUE |
| CL2 | VBB | -5.6580 | -2.0833 | 1.4913 | FALSE |
| CL2 | YRC1 | -7.8524 | -4.2778 | -0.7031 | TRUE |
| CL2 | YRC2 | -9.5747 | -6.0000 | -2.4253 | TRUE |
| DRD | EBD1 | -1.5747 | 2.0000 | 5.5747 | FALSE |
| DRD | EBD2 | -1.9636 | 1.6111 | 5.1858 | FALSE |
| DRD | EOS | -2.2136 | 1.3611 | 4.9358 | FALSE |
| DRD | MG | -1.3247 | 2.2500 | 5.8247 | FALSE |
| DRD | PI1 | -6.4080 | -2.8333 | 0.7413 | FALSE |
| DRD | PI2 | -7.8524 | -4.2778 | -0.7031 | TRUE |
| DRD | RFF1 | 2.6476 | 6.2222 | 9.7969 | TRUE |
| DRD | RFF2 | 3.2309 | 6.8056 | 10.3802 | TRUE |
| DRD | RK | -8.0469 | -4.4722 | -0.8976 | TRUE |
| DRD | VBB | -5.2691 | -1.6944 | 1.8802 | FALSE |
| DRD | YRC1 | -7.4636 | -3.8889 | -0.3142 | TRUE |
| DRD | YRC2 | -9.1858 | -5.6111 | -2.0364 | TRUE |
| EBD1 | EBD2 | -3.9636 | -0.3889 | 3.1858 | FALSE |
| EBD1 | EOS | -4.2136 | -0.6389 | 2.9358 | FALSE |
| EBD1 | MG | -3.3247 | 0.2500 | 3.8247 | FALSE |
| EBD1 | PI1 | -8.4080 | -4.8333 | -1.2587 | TRUE |
| EBD1 | PI2 | -9.8524 | -6.2778 | -2.7031 | TRUE |
| EBD1 | RFF1 | 0.6476 | 4.2222 | 7.7969 | TRUE |
| EBD1 | RFF2 | 1.2309 | 4.8056 | 8.3802 | TRUE |
| EBD1 | RK | -10.0469 | -6.4722 | -2.8976 | TRUE |
| EBD1 | VBB | -7.2691 | -3.6944 | -0.1198 | TRUE |
| EBD1 | YRC1 | -9.4636 | -5.8889 | -2.3142 | TRUE |
| EBD1 | YRC2 | -11.1858 | -7.6111 | -4.0364 | TRUE |
| EBD2 | EOS | -3.8247 | -0.2500 | 3.3247 | FALSE |
| EBD2 | MG | -2.9358 | 0.6389 | 4.2136 | FALSE |
| EBD2 | PI1 | -8.0191 | -4.4444 | -0.8698 | TRUE |
| EBD2 | PI2 | -9.4636 | -5.8889 | -2.3142 | TRUE |
| EBD2 | RFF1 | 1.0364 | 4.6111 | 8.1858 | TRUE |
| EBD2 | RFF2 | 1.6198 | 5.1944 | 8.7691 | TRUE |
| EBD2 | RK | -9.6580 | -6.0833 | -2.5087 | TRUE |
| EBD2 | VBB | -6.8802 | -3.3056 | 0.2691 | FALSE |
| EBD2 | YRC1 | -9.0747 | -5.5000 | -1.9253 | TRUE |
| EBD2 | YRC2 | -10.7969 | -7.2222 | -3.6476 | TRUE |
| EOS | MG | -2.6858 | 0.8889 | 4.4636 | FALSE |
| EOS | PI1 | -7.7691 | -4.1944 | -0.6198 | TRUE |
| EOS | PI2 | -9.2136 | -5.6389 | -2.0642 | TRUE |
| EOS | RFF1 | 1.2864 | 4.8611 | 8.4358 | TRUE |
| EOS | RFF2 | 1.8698 | 5.4444 | 9.0191 | TRUE |
| EOS | RK | -9.4080 | -5.8333 | -2.2587 | TRUE |
| EOS | VBB | -6.6302 | -3.0556 | 0.5191 | FALSE |
| EOS | YRC1 | -8.8247 | -5.2500 | -1.6753 | TRUE |
| EOS | YRC2 | -10.5469 | -6.9722 | -3.3976 | TRUE |
| MG | PI1 | -8.6580 | -5.0833 | -1.5087 | TRUE |
| MG | PI2 | -10.1024 | -6.5278 | -2.9531 | TRUE |
| MG | RFF1 | 0.3976 | 3.9722 | 7.5469 | TRUE |
| MG | RFF2 | 0.9809 | 4.5556 | 8.1302 | TRUE |
| MG | RK | -10.2969 | -6.7222 | -3.1476 | TRUE |
| MG | VBB | -7.5191 | -3.9444 | -0.3698 | TRUE |
| MG | YRC1 | -9.7136 | -6.1389 | -2.5642 | TRUE |
| MG | YRC2 | -11.4358 | -7.8611 | -4.2864 | TRUE |
| PI1 | PI2 | -5.0191 | -1.4444 | 2.1302 | FALSE |
| PI1 | RFF1 | 5.4809 | 9.0556 | 12.6302 | TRUE |
| PI1 | RFF2 | 6.0642 | 9.6389 | 13.2136 | TRUE |
| PI1 | RK | -5.2136 | -1.6389 | 1.9358 | FALSE |
| PI1 | VBB | -2.4358 | 1.1389 | 4.7136 | FALSE |
| PI1 | YRC1 | -4.6302 | -1.0556 | 2.5191 | FALSE |
| PI1 | YRC2 | -6.3524 | -2.7778 | 0.7969 | FALSE |
| PI2 | RFF1 | 6.9253 | 10.5000 | 14.0747 | TRUE |
| PI2 | RFF2 | 7.5087 | 11.0833 | 14.6580 | TRUE |
| PI2 | RK | -3.7691 | -0.1944 | 3.3802 | FALSE |
| PI2 | VBB | -0.9913 | 2.5833 | 6.1580 | FALSE |
| PI2 | YRC1 | -3.1858 | 0.3889 | 3.9636 | FALSE |
| PI2 | YRC2 | -4.9080 | -1.3333 | 2.2413 | FALSE |
| RFF1 | RFF2 | -2.9913 | 0.5833 | 4.1580 | FALSE |
| RFF1 | RK | -14.2691 | -10.6944 | -7.1198 | TRUE |
| RFF1 | VBB | -11.4913 | -7.9167 | -4.3420 | TRUE |
| RFF1 | YRC1 | -13.6858 | -10.1111 | -6.5364 | TRUE |
| RFF1 | YRC2 | -15.4080 | -11.8333 | -8.2587 | TRUE |
| RFF2 | RK | -14.8524 | -11.2778 | -7.7031 | TRUE |
| RFF2 | VBB | -12.0747 | -8.5000 | -4.9253 | TRUE |
| RFF2 | YRC1 | -14.2691 | -10.6944 | -7.1198 | TRUE |
| RFF2 | YRC2 | -15.9913 | -12.4167 | -8.8420 | TRUE |
| RK | VBB | -0.7969 | 2.7778 | 6.3524 | FALSE |
| RK | YRC1 | -2.9913 | 0.5833 | 4.1580 | FALSE |
| RK | YRC2 | -4.7136 | -1.1389 | 2.4358 | FALSE |
| VBB | YRC1 | -5.7691 | -2.1944 | 1.3802 | FALSE |
| VBB | YRC2 | -7.4913 | -3.9167 | -0.3420 | TRUE |
| YRC1 | YRC2 | -5.2969 | -1.7222 | 1.8524 | FALSE |
Task 2
The Friedman test was run in MATLAB to test significant differences amongst systems with regard to the performance (accuracy) on individual files.
Friedman's Anova Table
| Source | SS | df | MS | Chi-sq | Prob>Chi-sq |
|---|---|---|---|---|---|
| Columns | 3106.67 | 12 | 258.889 | 204.91 | 0 |
| Error | 2351.33 | 348 | 6.757 | ||
| Total | 5458 | 389 | |||
Tukey-Kramer HSD Multi-Comparison
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| EBD1 | EBD2 | -2.6305 | 0.7000 | 4.0305 | FALSE |
| EBD1 | EOS | -5.8305 | -2.5000 | 0.8305 | FALSE |
| EBD1 | PI1 | -5.5305 | -2.2000 | 1.1305 | FALSE |
| EBD1 | PI2 | -3.2305 | 0.1000 | 3.4305 | FALSE |
| EBD1 | RFF1 | 0.7695 | 4.1000 | 7.4305 | TRUE |
| EBD1 | RFF2 | 0.7529 | 4.0833 | 7.4138 | TRUE |
| EBD1 | RK | -8.6471 | -5.3167 | -1.9862 | TRUE |
| EBD1 | VBB | -3.7138 | -0.3833 | 2.9471 | FALSE |
| EBD1 | YRC | -7.8805 | -4.5500 | -1.2195 | TRUE |
| EBD1 | ZR1 | -5.9971 | -2.6667 | 0.6638 | FALSE |
| EBD1 | ZR2 | -5.9638 | -2.6333 | 0.6971 | FALSE |
| EBD1 | ZR3 | -6.7971 | -3.4667 | -0.1362 | TRUE |
| EBD2 | EOS | -6.5305 | -3.2000 | 0.1305 | FALSE |
| EBD2 | PI1 | -6.2305 | -2.9000 | 0.4305 | FALSE |
| EBD2 | PI2 | -3.9305 | -0.6000 | 2.7305 | FALSE |
| EBD2 | RFF1 | 0.0695 | 3.4000 | 6.7305 | TRUE |
| EBD2 | RFF2 | 0.0529 | 3.3833 | 6.7138 | TRUE |
| EBD2 | RK | -9.3471 | -6.0167 | -2.6862 | TRUE |
| EBD2 | VBB | -4.4138 | -1.0833 | 2.2471 | FALSE |
| EBD2 | YRC | -8.5805 | -5.2500 | -1.9195 | TRUE |
| EBD2 | ZR1 | -6.6971 | -3.3667 | -0.0362 | TRUE |
| EBD2 | ZR2 | -6.6638 | -3.3333 | -0.0029 | TRUE |
| EBD2 | ZR3 | -7.4971 | -4.1667 | -0.8362 | TRUE |
| EOS | PI1 | -3.0305 | 0.3000 | 3.6305 | FALSE |
| EOS | PI2 | -0.7305 | 2.6000 | 5.9305 | FALSE |
| EOS | RFF1 | 3.2695 | 6.6000 | 9.9305 | TRUE |
| EOS | RFF2 | 3.2529 | 6.5833 | 9.9138 | TRUE |
| EOS | RK | -6.1471 | -2.8167 | 0.5138 | FALSE |
| EOS | VBB | -1.2138 | 2.1167 | 5.4471 | FALSE |
| EOS | YRC | -5.3805 | -2.0500 | 1.2805 | FALSE |
| EOS | ZR1 | -3.4971 | -0.1667 | 3.1638 | FALSE |
| EOS | ZR2 | -3.4638 | -0.1333 | 3.1971 | FALSE |
| EOS | ZR3 | -4.2971 | -0.9667 | 2.3638 | FALSE |
| PI1 | PI2 | -1.0305 | 2.3000 | 5.6305 | FALSE |
| PI1 | RFF1 | 2.9695 | 6.3000 | 9.6305 | TRUE |
| PI1 | RFF2 | 2.9529 | 6.2833 | 9.6138 | TRUE |
| PI1 | RK | -6.4471 | -3.1167 | 0.2138 | FALSE |
| PI1 | VBB | -1.5138 | 1.8167 | 5.1471 | FALSE |
| PI1 | YRC | -5.6805 | -2.3500 | 0.9805 | FALSE |
| PI1 | ZR1 | -3.7971 | -0.4667 | 2.8638 | FALSE |
| PI1 | ZR2 | -3.7638 | -0.4333 | 2.8971 | FALSE |
| PI1 | ZR3 | -4.5971 | -1.2667 | 2.0638 | FALSE |
| PI2 | RFF1 | 0.6695 | 4.0000 | 7.3305 | TRUE |
| PI2 | RFF2 | 0.6529 | 3.9833 | 7.3138 | TRUE |
| PI2 | RK | -8.7471 | -5.4167 | -2.0862 | TRUE |
| PI2 | VBB | -3.8138 | -0.4833 | 2.8471 | FALSE |
| PI2 | YRC | -7.9805 | -4.6500 | -1.3195 | TRUE |
| PI2 | ZR1 | -6.0971 | -2.7667 | 0.5638 | FALSE |
| PI2 | ZR2 | -6.0638 | -2.7333 | 0.5971 | FALSE |
| PI2 | ZR3 | -6.8971 | -3.5667 | -0.2362 | TRUE |
| RFF1 | RFF2 | -3.3471 | -0.0167 | 3.3138 | FALSE |
| RFF1 | RK | -12.7471 | -9.4167 | -6.0862 | TRUE |
| RFF1 | VBB | -7.8138 | -4.4833 | -1.1529 | TRUE |
| RFF1 | YRC | -11.9805 | -8.6500 | -5.3195 | TRUE |
| RFF1 | ZR1 | -10.0971 | -6.7667 | -3.4362 | TRUE |
| RFF1 | ZR2 | -10.0638 | -6.7333 | -3.4029 | TRUE |
| RFF1 | ZR3 | -10.8971 | -7.5667 | -4.2362 | TRUE |
| RFF2 | RK | -12.7305 | -9.4000 | -6.0695 | TRUE |
| RFF2 | VBB | -7.7971 | -4.4667 | -1.1362 | TRUE |
| RFF2 | YRC | -11.9638 | -8.6333 | -5.3029 | TRUE |
| RFF2 | ZR1 | -10.0805 | -6.7500 | -3.4195 | TRUE |
| RFF2 | ZR2 | -10.0471 | -6.7167 | -3.3862 | TRUE |
| RFF2 | ZR3 | -10.8805 | -7.5500 | -4.2195 | TRUE |
| RK | VBB | 1.6029 | 4.9333 | 8.2638 | TRUE |
| RK | YRC | -2.5638 | 0.7667 | 4.0971 | FALSE |
| RK | ZR1 | -0.6805 | 2.6500 | 5.9805 | FALSE |
| RK | ZR2 | -0.6471 | 2.6833 | 6.0138 | FALSE |
| RK | ZR3 | -1.4805 | 1.8500 | 5.1805 | FALSE |
| VBB | YRC | -7.4971 | -4.1667 | -0.8362 | TRUE |
| VBB | ZR1 | -5.6138 | -2.2833 | 1.0471 | FALSE |
| VBB | ZR2 | -5.5805 | -2.2500 | 1.0805 | FALSE |
| VBB | ZR3 | -6.4138 | -3.0833 | 0.2471 | FALSE |
| YRC | ZR1 | -1.4471 | 1.8833 | 5.2138 | FALSE |
| YRC | ZR2 | -1.4138 | 1.9167 | 5.2471 | FALSE |
| YRC | ZR3 | -2.2471 | 1.0833 | 4.4138 | FALSE |
| ZR1 | ZR2 | -3.2971 | 0.0333 | 3.3638 | FALSE |
| ZR1 | ZR3 | -4.1305 | -0.8000 | 2.5305 | FALSE |
| ZR2 | ZR3 | -4.1638 | -0.8333 | 2.4971 | FALSE |

