Difference between revisions of "2018:Drum Transcription Results"
Richard Vogl (talk | contribs) (→Introduction) |
Richard Vogl (talk | contribs) (→Introduction) |
||
Line 11: | Line 11: | ||
To this end, a new evaluation and training dataset (MIDI) and new annotations for two datasets already used in the three-class-task (MEDLEY, RBMA) were additionally used this year. | To this end, a new evaluation and training dataset (MIDI) and new annotations for two datasets already used in the three-class-task (MEDLEY, RBMA) were additionally used this year. | ||
− | For a more detailed discussion of the subtasks an datasets consult the [[2018:Drum Transcription] task description page | + | For a more detailed discussion of the subtasks an datasets consult the [[2018:Drum Transcription]] task description page. |
== Submissions == | == Submissions == |
Revision as of 14:09, 20 September 2018
Contents
Introduction
The drum transcription task was reintroduced last year after it's first edition in 2005. Two out of the three original datasets used in 2005 were available and have been used for evaluation also this year. For those datasets the results from 2005 may be compared to this years results. Additionally to the two datasets from 2005, three new datasets were used in the evaluation. For training the algorithms, the public training set from 2005 plus additional training data taken from the new datasets was provided to the participants. In the context of this task, only the three most common drum instruments—kick/bass drum (KD,BD), snare drum (SD), and hi-hat (HH)—are considered.
As an addition to the three-instrument-class-task, an eight-instrument-class-task, was introduced this year. To this end, a new evaluation and training dataset (MIDI) and new annotations for two datasets already used in the three-class-task (MEDLEY, RBMA) were additionally used this year.
For a more detailed discussion of the subtasks an datasets consult the 2018:Drum Transcription task description page.
Submissions
Abstract | Contributors | |
---|---|---|
AR5, JAR1-3 | Celine Jacques, Achille Aknin, Axel Roebel | |
CS2,4 | Carl Southall | |
JS1 | Julien Schroeter | |
RV1-3 | Richard Vogl |
3 Class Results
The overall results represent the mean values over all datasets.
Overall
Algorithm | mean fm | sum fm | KD mean fm | KD sum fm | SD mean fm | SD sum fm | HH mean fm | HH sum fm |
---|---|---|---|---|---|---|---|---|
JS2 | 0.62 | 0.64 | 0.78 | 0.78 | 0.63 | 0.63 | 0.40 | 0.40 |
JAR2 | 0.68 | 0.73 | 0.78 | 0.78 | 0.65 | 0.65 | 0.53 | 0.53 |
JS3 | 0.61 | 0.62 | 0.76 | 0.76 | 0.63 | 0.63 | 0.36 | 0.36 |
JS4 | 0.57 | 0.58 | 0.73 | 0.73 | 0.61 | 0.61 | 0.32 | 0.32 |
RV2 | 0.65 | 0.72 | 0.75 | 0.75 | 0.65 | 0.65 | 0.50 | 0.50 |
RV1 | 0.69 | 0.74 | 0.78 | 0.78 | 0.69 | 0.69 | 0.53 | 0.53 |
JAR1 | 0.69 | 0.72 | 0.78 | 0.78 | 0.65 | 0.65 | 0.53 | 0.53 |
JAR3 | 0.69 | 0.73 | 0.79 | 0.79 | 0.65 | 0.65 | 0.54 | 0.54 |
JS1 | 0.61 | 0.62 | 0.75 | 0.75 | 0.59 | 0.59 | 0.43 | 0.43 |
JAR5 | 0.67 | 0.71 | 0.77 | 0.77 | 0.63 | 0.63 | 0.53 | 0.53 |
CS2 | 0.63 | 0.70 | 0.78 | 0.78 | 0.59 | 0.59 | 0.50 | 0.50 |
CS4 | 0.62 | 0.68 | 0.77 | 0.77 | 0.58 | 0.58 | 0.47 | 0.47 |
2005 baseline: 0.670 (YGO)
The best overall result from 2005 is only provided to put the current results into perspective. Since the overall result form 2005 was calculated on different datasets it is problematic to compare them directly.
IDMT subset
Algorithm | mean fm | sum fm | KD mean fm | KD sum fm | SD mean fm | SD sum fm | HH mean fm | HH sum fm |
---|---|---|---|---|---|---|---|---|
JS2 | 0.59 | 0.64 | 0.72 | 0.72 | 0.64 | 0.64 | 0.44 | 0.44 |
JAR2 | 0.63 | 0.68 | 0.73 | 0.73 | 0.59 | 0.59 | 0.53 | 0.53 |
JS3 | 0.57 | 0.63 | 0.71 | 0.71 | 0.67 | 0.67 | 0.38 | 0.38 |
JS4 | 0.55 | 0.61 | 0.69 | 0.69 | 0.69 | 0.69 | 0.32 | 0.32 |
RV2 | 0.66 | 0.73 | 0.74 | 0.74 | 0.71 | 0.71 | 0.54 | 0.54 |
RV1 | 0.66 | 0.72 | 0.75 | 0.75 | 0.72 | 0.72 | 0.53 | 0.53 |
JAR1 | 0.63 | 0.69 | 0.73 | 0.73 | 0.68 | 0.68 | 0.49 | 0.49 |
JAR3 | 0.64 | 0.69 | 0.74 | 0.74 | 0.60 | 0.60 | 0.54 | 0.54 |
JS1 | 0.58 | 0.62 | 0.69 | 0.69 | 0.56 | 0.56 | 0.48 | 0.48 |
JAR5 | 0.62 | 0.67 | 0.72 | 0.72 | 0.56 | 0.56 | 0.54 | 0.54 |
CS2 | 0.58 | 0.63 | 0.67 | 0.67 | 0.58 | 0.58 | 0.49 | 0.49 |
CS4 | 0.53 | 0.57 | 0.63 | 0.63 | 0.50 | 0.50 | 0.44 | 0.44 |
2005 baseline: 0.753 (CD)
KT subset
Algorithm | mean fm | sum fm | KD mean fm | KD sum fm | SD mean fm | SD sum fm | HH mean fm | HH sum fm |
---|---|---|---|---|---|---|---|---|
JS2 | 0.63 | 0.65 | 0.77 | 0.77 | 0.68 | 0.68 | 0.40 | 0.40 |
JAR2 | 0.64 | 0.65 | 0.77 | 0.77 | 0.67 | 0.67 | 0.46 | 0.46 |
JS3 | 0.62 | 0.64 | 0.76 | 0.76 | 0.69 | 0.69 | 0.36 | 0.36 |
JS4 | 0.59 | 0.60 | 0.72 | 0.72 | 0.68 | 0.68 | 0.33 | 0.33 |
RV2 | 0.63 | 0.67 | 0.76 | 0.76 | 0.68 | 0.68 | 0.45 | 0.45 |
RV1 | 0.65 | 0.68 | 0.80 | 0.80 | 0.68 | 0.68 | 0.47 | 0.47 |
JAR1 | 0.64 | 0.65 | 0.75 | 0.75 | 0.67 | 0.67 | 0.45 | 0.45 |
JAR3 | 0.65 | 0.66 | 0.78 | 0.78 | 0.68 | 0.68 | 0.47 | 0.47 |
JS1 | 0.61 | 0.62 | 0.73 | 0.73 | 0.59 | 0.59 | 0.43 | 0.43 |
JAR5 | 0.62 | 0.63 | 0.74 | 0.74 | 0.64 | 0.64 | 0.46 | 0.46 |
CS2 | 0.60 | 0.64 | 0.76 | 0.76 | 0.58 | 0.58 | 0.44 | 0.44 |
CS4 | 0.56 | 0.60 | 0.72 | 0.72 | 0.57 | 0.57 | 0.39 | 0.39 |
2005 baseline: 0.617 (YGO)
RBMA subset
Algorithm | mean fm | sum fm | KD mean fm | KD sum fm | SD mean fm | SD sum fm | HH mean fm | HH sum fm |
---|---|---|---|---|---|---|---|---|
JS2 | 0.57 | 0.60 | 0.80 | 0.80 | 0.47 | 0.47 | 0.39 | 0.39 |
JAR2 | 0.70 | 0.73 | 0.87 | 0.87 | 0.55 | 0.55 | 0.57 | 0.57 |
JS3 | 0.55 | 0.58 | 0.79 | 0.79 | 0.48 | 0.48 | 0.33 | 0.33 |
JS4 | 0.50 | 0.53 | 0.76 | 0.76 | 0.46 | 0.46 | 0.27 | 0.27 |
RV2 | 0.70 | 0.74 | 0.91 | 0.91 | 0.60 | 0.60 | 0.55 | 0.55 |
RV1 | 0.72 | 0.74 | 0.91 | 0.91 | 0.62 | 0.62 | 0.56 | 0.56 |
JAR1 | 0.68 | 0.72 | 0.87 | 0.87 | 0.45 | 0.45 | 0.56 | 0.56 |
JAR3 | 0.69 | 0.73 | 0.87 | 0.87 | 0.56 | 0.56 | 0.57 | 0.57 |
JS1 | 0.56 | 0.60 | 0.76 | 0.76 | 0.43 | 0.43 | 0.43 | 0.43 |
JAR5 | 0.67 | 0.71 | 0.83 | 0.83 | 0.53 | 0.53 | 0.56 | 0.56 |
CS2 | 0.67 | 0.71 | 0.87 | 0.87 | 0.55 | 0.55 | 0.56 | 0.56 |
CS4 | 0.66 | 0.70 | 0.85 | 0.85 | 0.50 | 0.50 | 0.56 | 0.56 |
MDB subset
Algorithm | mean fm | sum fm | KD mean fm | KD sum fm | SD mean fm | SD sum fm | HH mean fm | HH sum fm |
---|---|---|---|---|---|---|---|---|
JS2 | 0.59 | 0.58 | 0.73 | 0.73 | 0.58 | 0.58 | 0.43 | 0.43 |
JAR2 | 0.65 | 0.65 | 0.64 | 0.64 | 0.62 | 0.62 | 0.59 | 0.59 |
JS3 | 0.54 | 0.54 | 0.68 | 0.68 | 0.51 | 0.51 | 0.39 | 0.39 |
JS4 | 0.48 | 0.48 | 0.62 | 0.62 | 0.44 | 0.44 | 0.33 | 0.33 |
RV2 | 0.55 | 0.55 | 0.44 | 0.44 | 0.54 | 0.54 | 0.58 | 0.58 |
RV1 | 0.64 | 0.65 | 0.55 | 0.55 | 0.60 | 0.60 | 0.57 | 0.57 |
JAR1 | 0.68 | 0.66 | 0.66 | 0.66 | 0.61 | 0.61 | 0.60 | 0.60 |
JAR3 | 0.67 | 0.67 | 0.69 | 0.69 | 0.64 | 0.64 | 0.59 | 0.59 |
JS1 | 0.59 | 0.57 | 0.72 | 0.72 | 0.60 | 0.60 | 0.46 | 0.46 |
JAR5 | 0.66 | 0.66 | 0.68 | 0.68 | 0.63 | 0.63 | 0.58 | 0.58 |
CS2 | 0.63 | 0.61 | 0.69 | 0.69 | 0.48 | 0.48 | 0.63 | 0.63 |
CS4 | 0.65 | 0.63 | 0.75 | 0.75 | 0.56 | 0.56 | 0.59 | 0.59 |
MDB-Drums [1]
GEN subset
Algorithm | mean fm | sum fm | KD mean fm | KD sum fm | SD mean fm | SD sum fm | HH mean fm | HH sum fm |
---|---|---|---|---|---|---|---|---|
JS2-3 | 0.73 | 0.75 | 0.87 | 0.87 | 0.78 | 0.78 | 0.33 | 0.33 |
JAR2 | 0.79 | 0.81 | 0.87 | 0.87 | 0.80 | 0.80 | 0.52 | 0.52 |
JS3-3 | 0.75 | 0.77 | 0.88 | 0.88 | 0.79 | 0.79 | 0.36 | 0.36 |
JS4-3 | 0.75 | 0.77 | 0.87 | 0.87 | 0.79 | 0.79 | 0.36 | 0.36 |
RV2 | 0.70 | 0.75 | 0.89 | 0.89 | 0.75 | 0.75 | 0.38 | 0.38 |
RV1 | 0.78 | 0.81 | 0.91 | 0.91 | 0.80 | 0.80 | 0.50 | 0.50 |
JAR1 | 0.81 | 0.83 | 0.87 | 0.87 | 0.83 | 0.83 | 0.55 | 0.55 |
JAR3 | 0.78 | 0.81 | 0.88 | 0.88 | 0.79 | 0.79 | 0.51 | 0.51 |
JS1-3 | 0.70 | 0.72 | 0.84 | 0.84 | 0.76 | 0.76 | 0.34 | 0.34 |
JAR5 | 0.76 | 0.78 | 0.86 | 0.86 | 0.77 | 0.77 | 0.51 | 0.51 |
CS2-3 | 0.68 | 0.75 | 0.92 | 0.92 | 0.75 | 0.75 | 0.38 | 0.38 |
CS4-3 | 0.68 | 0.75 | 0.92 | 0.92 | 0.76 | 0.76 | 0.38 | 0.38 |
8 Class Results
The overall results represent the mean values over all datasets.
Overall
Algorithm | mean fm | sum fm | BD mean fm | BD sum fm | SD mean fm | SD sum fm | TT mean fm | TT sum fm | HH mean fm | HH sum fm | CY mean fm | CY sum fm | RD mean fm | RD sum fm | CB mean fm | CB sum fm | CL mean fm | CL sum fm |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
JS4 | 0.54 | 0.59 | 0.73 | 0.73 | 0.48 | 0.48 | 0.29 | 0.29 | 0.46 | 0.46 | 0.39 | 0.39 | 0.50 | 0.50 | 0.82 | 0.82 | 0.90 | 0.90 |
JS2 | 0.58 | 0.61 | 0.79 | 0.79 | 0.52 | 0.52 | 0.19 | 0.19 | 0.52 | 0.52 | 0.32 | 0.32 | 0.41 | 0.41 | 0.80 | 0.80 | 0.90 | 0.90 |
JS3 | 0.57 | 0.62 | 0.77 | 0.77 | 0.51 | 0.51 | 0.23 | 0.23 | 0.50 | 0.50 | 0.41 | 0.41 | 0.47 | 0.47 | 0.82 | 0.82 | 0.90 | 0.90 |
JS1 | 0.55 | 0.57 | 0.75 | 0.75 | 0.49 | 0.49 | 0.15 | 0.15 | 0.52 | 0.52 | 0.20 | 0.20 | 0.20 | 0.20 | 0.68 | 0.68 | 0.90 | 0.90 |
CS4 | 0.50 | 0.56 | 0.79 | 0.79 | 0.51 | 0.51 | 0.14 | 0.14 | 0.62 | 0.62 | 0.20 | 0.20 | 0.13 | 0.13 | 0.02 | 0.02 | 0.02 | 0.02 |
CS2 | 0.47 | 0.54 | 0.78 | 0.78 | 0.50 | 0.50 | 0.15 | 0.15 | 0.60 | 0.60 | 0.19 | 0.19 | 0.14 | 0.14 | 0.02 | 0.02 | 0.02 | 0.02 |
RV3 | 0.62 | 0.68 | 0.78 | 0.78 | 0.52 | 0.52 | 0.37 | 0.37 | 0.58 | 0.58 | 0.38 | 0.38 | 0.65 | 0.65 | 0.82 | 0.82 | 0.71 | 0.71 |
RBMA subset
Algorithm | mean fm | sum fm | BD mean fm | BD sum fm | SD mean fm | SD sum fm | TT mean fm | TT sum fm | HH mean fm | HH sum fm | CY mean fm | CY sum fm | RD mean fm | RD sum fm | CB mean fm | CB sum fm | CL mean fm | CL sum fm |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
JS4 | 0.47 | 0.50 | 0.71 | 0.71 | 0.28 | 0.28 | 0.15 | 0.15 | 0.43 | 0.43 | 0.61 | 0.61 | 0.73 | 0.73 | 0.82 | 0.82 | 0.76 | 0.76 |
JS2 | 0.50 | 0.54 | 0.80 | 0.80 | 0.28 | 0.28 | 0.08 | 0.08 | 0.48 | 0.48 | 0.33 | 0.33 | 0.55 | 0.55 | 0.79 | 0.79 | 0.76 | 0.76 |
JS3 | 0.50 | 0.53 | 0.77 | 0.77 | 0.28 | 0.28 | 0.10 | 0.10 | 0.47 | 0.47 | 0.61 | 0.61 | 0.67 | 0.67 | 0.82 | 0.82 | 0.76 | 0.76 |
JS1 | 0.47 | 0.51 | 0.77 | 0.77 | 0.25 | 0.25 | 0.08 | 0.08 | 0.48 | 0.48 | 0.10 | 0.10 | 0.16 | 0.16 | 0.73 | 0.73 | 0.76 | 0.76 |
CS4 | 0.44 | 0.48 | 0.85 | 0.85 | 0.34 | 0.34 | 0.14 | 0.14 | 0.57 | 0.57 | 0.02 | 0.02 | 0.03 | 0.03 | 0.02 | 0.02 | 0.05 | 0.05 |
CS2 | 0.40 | 0.43 | 0.86 | 0.86 | 0.34 | 0.34 | 0.13 | 0.13 | 0.52 | 0.52 | 0.02 | 0.02 | 0.04 | 0.04 | 0.02 | 0.02 | 0.05 | 0.05 |
RV3 | 0.55 | 0.58 | 0.85 | 0.85 | 0.27 | 0.27 | 0.13 | 0.13 | 0.48 | 0.48 | 0.64 | 0.64 | 0.79 | 0.79 | 0.82 | 0.82 | 0.52 | 0.52 |
MDB subset
Algorithm | mean fm | sum fm | BD mean fm | BD sum fm | SD mean fm | SD sum fm | TT mean fm | TT sum fm | HH mean fm | HH sum fm | CY mean fm | CY sum fm | RD mean fm | RD sum fm | CB mean fm | CB sum fm | CL mean fm | CL sum fm |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
JS4 | 0.56 | 0.53 | 0.76 | 0.76 | 0.52 | 0.52 | 0.45 | 0.45 | 0.42 | 0.42 | 0.33 | 0.33 | 0.47 | 0.47 | 0.91 | 0.91 | 1.00 | 1.00 |
JS2 | 0.61 | 0.58 | 0.81 | 0.81 | 0.65 | 0.65 | 0.27 | 0.27 | 0.49 | 0.49 | 0.39 | 0.39 | 0.49 | 0.49 | 0.91 | 0.91 | 1.00 | 1.00 |
JS3 | 0.60 | 0.57 | 0.80 | 0.80 | 0.61 | 0.61 | 0.36 | 0.36 | 0.47 | 0.47 | 0.37 | 0.37 | 0.49 | 0.49 | 0.91 | 0.91 | 1.00 | 1.00 |
JS1 | 0.59 | 0.56 | 0.77 | 0.77 | 0.67 | 0.67 | 0.19 | 0.19 | 0.47 | 0.47 | 0.24 | 0.24 | 0.34 | 0.34 | 0.91 | 0.91 | 1.00 | 1.00 |
CS4 | 0.47 | 0.48 | 0.76 | 0.76 | 0.51 | 0.51 | 0.09 | 0.09 | 0.61 | 0.61 | 0.25 | 0.25 | 0.15 | 0.15 | 0.00 | 0.00 | 0.00 | 0.00 |
CS2 | 0.44 | 0.46 | 0.71 | 0.71 | 0.51 | 0.51 | 0.10 | 0.10 | 0.60 | 0.60 | 0.20 | 0.20 | 0.14 | 0.14 | 0.00 | 0.00 | 0.00 | 0.00 |
RV3 | 0.65 | 0.60 | 0.72 | 0.72 | 0.63 | 0.63 | 0.60 | 0.60 | 0.60 | 0.60 | 0.27 | 0.27 | 0.66 | 0.66 | 0.91 | 0.91 | 0.73 | 0.73 |
MDB-Drums [2]
MIDI subset
Algorithm | mean fm | sum fm | BD mean fm | BD sum fm | SD mean fm | SD sum fm | TT mean fm | TT sum fm | HH mean fm | HH sum fm | CY mean fm | CY sum fm | RD mean fm | RD sum fm | CB mean fm | CB sum fm | CL mean fm | CL sum fm |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
JS4 | 0.58 | 0.66 | 0.71 | 0.71 | 0.63 | 0.63 | 0.26 | 0.26 | 0.52 | 0.52 | 0.23 | 0.23 | 0.30 | 0.30 | 0.74 | 0.74 | 0.94 | 0.94 |
JS2 | 0.61 | 0.67 | 0.75 | 0.75 | 0.62 | 0.62 | 0.23 | 0.23 | 0.58 | 0.58 | 0.23 | 0.23 | 0.19 | 0.19 | 0.70 | 0.70 | 0.94 | 0.94 |
JS3 | 0.61 | 0.67 | 0.74 | 0.74 | 0.64 | 0.64 | 0.22 | 0.22 | 0.56 | 0.56 | 0.24 | 0.24 | 0.25 | 0.25 | 0.72 | 0.72 | 0.94 | 0.94 |
JS1 | 0.57 | 0.61 | 0.71 | 0.71 | 0.55 | 0.55 | 0.18 | 0.18 | 0.60 | 0.60 | 0.25 | 0.25 | 0.12 | 0.12 | 0.40 | 0.40 | 0.94 | 0.94 |
CS4 | 0.59 | 0.63 | 0.76 | 0.76 | 0.67 | 0.67 | 0.19 | 0.19 | 0.67 | 0.67 | 0.32 | 0.32 | 0.20 | 0.20 | 0.05 | 0.05 | 0.02 | 0.02 |
CS2 | 0.58 | 0.63 | 0.76 | 0.76 | 0.63 | 0.63 | 0.23 | 0.23 | 0.68 | 0.68 | 0.33 | 0.33 | 0.22 | 0.22 | 0.04 | 0.04 | 0.02 | 0.02 |
RV3 | 0.66 | 0.75 | 0.78 | 0.78 | 0.66 | 0.66 | 0.37 | 0.37 | 0.66 | 0.66 | 0.22 | 0.22 | 0.52 | 0.52 | 0.74 | 0.74 | 0.90 | 0.90 |