2009:Audio Chord Detection

Description

The text of this section is copied from the 2008 page. This task was first run in 2008. Please add your comments and discussions for 2009.

For many applications in music information retrieval, extracting the harmonic structure is very desirable, for example for segmenting pieces into characteristic segments, for finding similar pieces, or for semantic analysis of music.

The extraction of the harmonic structure requires the detection of as many chords as possible in a piece. That includes the characterisation of chords with a key and type as well as a chronological order with onset and duration of the chords.

Although some publications are available on this topic [1,2,3,4,5], comparison of the results is difficult, because different measures are used to assess the performance. To overcome this problem an accurately defined methodology is needed. This includes a repertory of the findable chords, a defined test set along with ground truth and unambiguous calculation rules to measure the performance.

Regarding this we suggest to introduced the new evaluation task Audio Chord Detection.

The deadline for this task is September 14th.

Data

Christopher Harte`s Beatles dataset is used for the evaluations last year. This dataset consists of 12 Beatles albums [6]. An approach for text annotation of musical chords is presented in [6]. This year an extra dataset was donated by Matthias Mauch which consists of 38 songs from Queen and Zweieck. The data will be provided as 44.1 kHz 16bit mono wav. The ground-truth looks like this:

41.2631021 44.2456460 B

44.2456460 45.7201130 E

45.7201130 47.2061900 E:7/3

47.2061900 48.6922670 A

48.6922670 50.1551240 A:min/b3

I/O Format

This year I/O format needs to be changed to evaluate on all triads an quads. We are planning to use the format suggested by Christopher Harte [6]. The chord root is given as a natural (A|B|C|D|E|F|G) followed by optional sharp or flat modifiers (#|b). For the evaluation process we may assume enharmonic equivalence for chord roots. For a given chord type on root X, the chord labels can be given as a list of intervals or as a shorthand notation as shown in the following table:

NAME	INTERVALS	SHORTHAND
major	X:(1,3,5)	X or X:maj
minor	X:(1,b3,5)	X:min
diminished	X:(1,b3,b5)	X:dim
augmented	X:(1,3,#5)	X:aug
suspended4	X:(1,4,5)	X:sus4
possible 6th triad:
suspended2	X:(1,2,5)	X:sus2
*Quads:
major-major7	X:(1,3,5,7)	X:maj7
major-minor7	X:(1,3,5,b7)	X:7
major-add9	X:(1,3,5,9)	X:maj(9)
major-major7-#5	X:(1,3,#5,7)	X:aug(7)
minor-major7	X:(1,b3,5,7)	X:min(7)
minor-minor7	X:(1,b3,5,b7)	X:min7
minor-add9	X:(1,b3,5,9)	X:min(9)
minor 7/b5 (ambiguous - could be either of the following)
minor-major7-b5	X:(1,b3,b5,7)	X:dim(7)
minor-minor7-b5 (a half diminished-7th)	X:(1,b3,b5,b7)	X:hdim7
sus4-major7	X:(1,4,5,7)	X:sus4(7)
sus4-minor7	X:(1,4,5,b7)	X:sus4(b7)
omitted from list on wiki:
diminished7	X:(1,b3,b5,bb7)	X:dim7
No Chord	N

However, we still accept participants who would only like to be evaluated on major/minor and want to use last year`s format which is an integer chord id on range 0-24, where values 0-11 denote the C major, C# major, ..., B major and 12-23 denote the C minor, C# minor, ..., B minor and 24 denotes silence or no-chord segments

Submission Format

Submissions have to conform to the specified format below:

extractFeaturesAndTrain  "/path/to/trainFileList.txt"  "/path/to/scratch/dir"  

Where fileList.txt has the paths to each wav file. The features extracted on this stage can be stored under "/path/to/scratch/dir" The ground truth files for the supervised learning will be in the same path with a ".txt" extension at the end. For example for "/path/to/trainFile1.wav", there will be a corresponding ground truth file called "/path/to/trainFile1.wav.txt" .

For testing:

doChordID.sh "/path/to/testFileList.txt"  "/path/to/scratch/dir" "/path/to/results/dir"  

If there is no training, you can ignore the second argument here. In the results directory, there should be one file for each testfile with same name as the test file + .txt .

Programs can use their working directory if they need to keep temporary cache files or internal debuggin info. Stdout and stderr will be logged.

Evaluation

Algorithms should output text files with a similar format to that used in the ground truth transcriptions. That is to say, they should be flat text files with chord segment labels and times arranged thus:

start_time end_time chord_label

with elements separated by white spaces, times given in seconds, chord labels corresponding to the syntax described in [6] and one chord segment per line.

Please note that two things have changed in the syntax since it was originally described in [6]. The first change is that the root is no longer implied as a voiced element of a chord so a C major chord (notes C, E and G) should be written C:(1,3,5) instead of just C:(3,5) if using the interval list representation. As before, the labels C and C:maj are equivalent to C:(1,3,5). The second change is that the shorthand label "sus2" (intervals 1,2,5) has been added to the available shorthand list.--Chrish 17:05, 9 September 2009 (UTC)

Segmentation Score

The segmentation score will be calculated using directional hamming distance as described in [8]. An over-segmentation value (m) and an under-segmentation value (f) will be calculated and the final segmentation score will be calculated using the worst case from these two i.e:

segmentation score = 1 - max(m,f)

m and f are not independent of each other so combining them this way ensures that a good score in one does not hide a bad score in the other. The combined segmentation score can take values between 0 and 1 with 0 being the worst and 1 being the best result.-- Chrish 17:05, 9 September 2009 (UTC)

Frame-based recall

For recall evaluation, we may define a different chord dictionary for each level of evaluation (dyads, triads, tetrads etc). Each dictionary is a text file containing chord shorthands / interval lists of the chords that will be considered in that evaluation. The following dictionaries are proposed:

For dyad comparison of major/minor chords only:

N
X:maj
X:min

For comparison of standard triad chords:

N
X:maj
X:min
X:aug
X:dim
X:sus2
X:sus4

For comparison of tetrad (quad) chords:

N
X:maj
X:min
X:aug
X:dim
X:sus2
X:sus4
X:maj7
X:7
X:maj(9)
X:aug(7)
X:min(7)
X:min7
X:min(9)
X:dim(7)
X:hdim7
X:sus4(7)
X:sus4(b7)
X:dim7

For each evaluation level, the ground truth annotation is compared against the dictionary. Any chord label not belonging to the current dictionary will be replaced with an "X" in a local copy of the annotation and will not be included in the recall calculation.

Note that the level of comparison in terms of intervals can be varied. For example, in a triad evaluation we can consider the first three component intervals in the chord so that a major (1,3,5) and a major7 (1,3,5,7) will be considered the same chord. For a tetrad (quad) evaluation, we would consider the first 4 intervals so major and major7 would then be considered to be different chords.

For the maj/min evaluation (using the first example dictionary), using an interval comparison of 2 (dyad) will compare only the first two intervals of each chord label. This would map augmented and diminished chords to major and minor respectively (and any other symbols that had a major 3rd or minor 3rd as their first interval). Using an interval comparison of 3 with the same dictionary would keep only those chords that have major and minor triads as their first 3 intervals so augmented and diminished chords would be removed from the evaluation.

After the annotation has been "filtered" using a given dictionary, it can be compared against the machine generated estimates output by the algorithm under test. The chord sequences described in the annotation and estimate text files are sampled at a given frame rate (in this case 10ms per frame) to give two sequences of chord frames which may be compared directly with each other. For calculating a hit or a miss, the chord labels from the current frame in each sequence will be compared. Chord comparison is done by converting each chord label into an ordered list of pitch classes then comparing the two lists element by element. If the lists match to the required number of intervals then a hit is recorded, otherwise the estimate is considered a miss. It should be noted that, by converting to pitch classes in the comparison, this evaluation ignores enharmonic pitch and interval spellings so the following chords (slightly silly example just for illustration) will all evaluate as identical:

C:maj = Dbb:maj = C#:(b1,b3,#4)

Basic recall calculation algorithm:

1) filter annotated transcription using chord dictionary for a defined number of intervals

2) sample annotated transcription and machine estimated transcription at 10ms intervals to create a sequence of annotation frames and estimate frames

3) start at the first frame

4) get chord label for current annotation frame and estimate frame

5) check annotation label:

IF symbol is 'X' (i.e. non-dictionary)

THEN ignore frame (record number of ignored frames)

ELSE compare annotated/estimated chords for the predefined number of intervals
increment hit count if chords match

ENDIF

6) increment frame count

7) go back to 4 until final chord frame --Chrish 17:05, 9 September 2009 (UTC)

Discussions

Points to discuss:

• The Precision measure has not been used last year, and I believe it should not because (unlike in beat extraction) we can assume a contiguous sequence of chords, i.e. all time units should feature a chord label. --Matthias 11:04, 27 June 2009 (UTC)

• I would like to disagree on Matthias' previous point: I think we cannot assume that there is a chord present in every frame, one can think for instance of a drum solo, an acapella break, ethnic music or simply the beginning and ending of a file. In melody extraction or beat detection, there also isn't a continuity assumption. I must say that at the moment, our system isn't able to generate a no-chord either, so it is not in my personal interest to add this to the evaluation, but I feel this should be part of a general chord extraction system. I've also learned from some premature experiments that with the current frame-based evaluation, it is actually not even beneficial to include such a no-chord generator, because of the inequality of prior chances between a chord and a no-chord (14% for a N.C. in our little dataset, I suspect it to be even less for the Beatles set). The consequence is that the chord/no-chord distinction must be very accurate in order to increase the performance. A related, minor topic is the naming of this task. Why isn't it "audio chord extraction" just like "melody extraction". For me "chord detection" is making the distinction between chords and no-chords and "chord extraction" is naming detected chords. Anyway, just nitpicking on that one. --Johan 15:43, 16 July 2009 (CET)

• I think we can assume a contiguous sequence of chords if we treat "no chord" as a chord. --Matthias 16:01, 7 August 2009 (UTC)

• I believe we should move forward in two ways to get a more meaningful evaluation:
  1. evaluate separate recall measures for several chord classes, my proposal is major, minor, diminished, augmented, dominant (meaning major chords with a minor seventh). A final recall score can then be calculated as a (weighted) average of the recall on different chords. --Matthias 11:04, 27 June 2009 (UTC)
  2. We use just triads major, minor, diminished & augmented which I think is a more sensible distinction. Once you start using quads, why limit yourself to dominant 7 and not use minor 7, major 7, full diminished, etc. So I'm more in favour of just triads (maybe add sus too) or more quads. -- Johan 15:48, 16 July 2009 (CET)
  3. Segmentation should be considered. For example, a chord extraction algorithm that has reasonable recall may still be heavily fragmented thus producing an output difficult to read for humans. One measure to check for similarity in segmentation is directional Hamming distance (or divergence). --Matthias 11:04, 27 June 2009 (UTC)
  4. Agree, while the frame-based evaluation is certainly easy, it is not the most musically sensible. An evaluation on note-level or chord-segment basis might be a little too complicated for now, but this is a start. -- Johan 15:51, 16 July 2009 (CET)
  5. Do you think we could consider several evaluation cases for chord detection with various chord dictionnaries? (For instance one with major/minor triads, one with major/minor/diminished/augmented/dominant etc.) so that each particpant can choose a case that can be handled by his/her algorithm? --Helene (IRCAM)
  6. to Helene: Several (maybe two) evaluation cases /could/ be good, but I think everyone's algorithm should be tested on every task, I think choosing the one you want would mean you can't compare it to other people's method.
  7. to Johan: in your chord list, did you mean to give the same list as I did? Anyway, I like to add "dominant" because it is used often, and musically (Jazz harmony theory) there's a functional difference between "dominant" and "major" (not between "minor" and "minor 7", and not between "major" and "major 7" or "major 6"). --Matthias 16:01, 7 August 2009 (UTC)
  8. @Matthias: no I intended to list just the triads (the dominant shouldn't have been copy-pasted, corrected it now) -- Johan 10:49, 25 Aug 2009 (CET)
  9. to Matthias: I think that using the label "dominant" as you suggest here is not the correct use of that musical term in this context. In music theory (both in classical and jazz harmony) it is true that a dominant-seventh chord is always the major triad + minor 7th chord shape i.e. (1,3,5,b7). However, a (1,3,5,b7) chord does not always function as a dominant. Whether such a chord can be labelled dominant or not is entirely based on the chord's position in a given chord sequence relative to other chords which define its function in the progression. It is precisely for that reason that I argue against the use of the term 'dominant' for context-free chord labelling in the ISMIR05 chord labels paper [6]. -- Chrish 17:00, 10 August 2009 (UTC)
  10. It seems counter-intuitive to have an evaluation that includes some quads but not all. It is fair to evaluate across the triad shapes major, minor, augmented, diminished and suspended (although sus2 and sus4 are inversions of each other) as these are the naturally occurring triad shapes in western harmony. It would seem sensible to include all the other quads that are labelled "7th" chords if including the (1,3,5,b7) shape in an evaluation. Given this problem, one possible way to compare algorithms that detect different sets of chord labels would be to split the results between triad recognition and quad/quint etc recognition. All algorithms can be tested on the triad evaluation - any algorithm that can detect quads can be compared directly against an algorithm that deals only with triads simply by taking the equivalent first three intervals of each chord label in the transcription as a triad. Only those algorithms that can recognise quads need to be evaluated in results that include quad chords. To make this process easy, it might be sensible for the labels that chord recognition algorithms produce to be given in terms of the intervals in the chords themselves rather than a chord name - i.e. a C major could be "C:(1,3,5)" and C major seventh "C:(1,3,5,7)" - both would evaluate as C major in a triad evaluation but the second one could also be evaluated in a test that looked at quads as well. This would also take away problems with possible labelling ambiguities that chord name labels could introduce. A list of the triads and quads that are acceptable in each level of evaluation would need to be drawn up but there is a list something like that at the top of this page already. -- Chrish 17:00, 10 August 2009 (UTC)
  11. I think keeping the MIREX2008 evaluation procedure would make sense. I like the idea of a "merged maj/min" score in order to evaluate the root precision, and then, maybe another score taking only into account the root and mode(maj/min). Then, we could progressively extend the chord dictionary, by adding triads and quads (as suggested by Chris and Johan) and calculating new scores based on these extension.-- Thomas 12:45, 12 August 2009 (UTC)
  12. I like the idea of using a segmentation measure based on directional hamming distance - We must make sure that the measure captures both over-segmentation (fragmentation) and under-segmentation though. The fragmentation measure (1-f) based on the inverse directional hamming distance described by Abdallah et al [8] only measures fragmentation by measuring the distance d_MG between the estimated sequence (M) and the ground-truth annotation (G). They also propose an under-segmentation measure (1-m) which uses the forward directional hamming distance d_GM. Using the fragmentation measure alone would mean that a chord recognition algorithm that output one chord label for the entire piece would score 100% correct for fragmentation. Both of these measures give a value between 0 and 1 so we could combine them to give an overall "chord segmentation" measure: 1-max(f,m) (f and m are not independent so it's probably best to just use the worst one rather than combine them geometrically). I think this measure could complement the frame-based recall quite well. -- Chrish 10:00, 11 August 2009 (UTC)
  13. anyone know how to make the LaTeX maths work on the wiki? -- Chrish 17:01, 10 August 2009 (UTC)
• Something to consider when broadening the scope of used chords, is the inequality in prior chance of different chords (much like the problem with chords/no-chords I mentioned above). When looking for augmented and diminished triads in the Beatles set in addition to major and minor, I'm quite positive the (or at least my) overall performance will decrease. Some processing/selecting could level the priors, but just limitting the data set to the duration of the least frequent chord won't leave us with much data, I'm afraid. The thing is of course that the inequality is also there in reality, so I'm not really convinced myself that this should be done. Another option is not changing the data, but letting the evaluation take it into account. -- Johan 16:23, 16 July 2009 (CET)

• Should chord data be expressed in absolute (aka "F major-minor 7") or relative (aka "C: IV major-minor 7") terms?

Absolute is better for the moment - if an algorithm can generate relative information then it can be converted easily to absolute. Going the other way is not easy if the algorithm doesn't estimate the key before trying to recognise chords... -- Chrish 17:04, 9 September 2009 (UTC)

• Should different inversions of chords be considered in the evaluation process?

The evaluation procedure I've outlined above ignores inversion at the moment. -- Chrish 17:04, 9 September 2009 (UTC)

• What temporal resolution should be used for ground truth and results?

If all algorithms output a text file with chord start/end times and labels then the underlying resolutions of different systems can be different. In the evaluation stage I suggested sampling the transcriptions in 10ms frames because that's the limit of what we can perceive in terms of separate onsets anyway.-- Chrish 17:04, 9 September 2009 (UTC)

• How should enharmonic and other confusions of chords be handled?

Although the chord symbol syntax allows enharmonic spellings to be differentiated, the evaluation process converts the symbols into lists of pitch classes. As long as the symbol is a valid one according to the syntax in [6] (with the two caveats I outlined above) then the system will have no trouble accepting any chord symbol.-- Chrish 17:04, 9 September 2009 (UTC)

• How will Ground Truth be determined?
  1. One thing we should consider doing is using the combined results to find possible errors or inconsistencies in the data. I have found that there are some that exists in Harte's labels (this was, after all, extremely difficult). If there are sections that are systematically error-prone, then maybe it is best to reevaluate these labels for future runs Also, we need to start finding other databases. Since we are only using Beatles data each year, we have no measure of generalization. Maybe we should make a requirement that participation includes a requirement to label 2-3 songs? These could be distributed to a few people and for verification as well. Just a thought. -- Jeremy 14:45, 25 August 2009 (UTC)
  2. If anyone finds mistakes in the Beatles data please let me know about them so I can fix them before I release the new versions (this will be in time for ISMIR 2009) -- Chrish 17:04, 9 September 2009 (UTC)

• What degree of chordal/tonal complexity will the music contain?

This can be variable using different chord dictionaries.-- Chrish 17:04, 9 September 2009 (UTC)

• Will we include any atonal or polytonal music in the Ground Truth dataset?

Are there any recognisable chords in these types of music as such? -- Chrish 17:04, 9 September 2009 (UTC)

• What is the maximal acceptable onset deviation between ground truth and result?

If using a frame based recall and a segmentation measure based on directional hamming distance, we do not need to specify an allowable onset deviation. -- Chrish 17:04, 9 September 2009 (UTC)

• What file format should be used for ground truth and output?

Flat text - effectively the same as the .lab files used for the Beatles transcriptions.-- Chrish 17:04, 9 September 2009 (UTC)

Comment`s by MB 08.12

First I want to make clear that we are using the Christopher`s Harte`s Beatles dataset which includes quad chords. Last year the evaluations were based only on major,minor and non chords. This year if there are enough participants we can extend the evaluations to the rest of the triads diminished, augmented suspended and to quads. Please vote:

<poll> How would you like to evaluations to be performed ? Same as last year evaluate on major, minor and non chord All triads: major, minor, diminished, augmented, suspended + non chord All triads + quads + non chord </poll>

If we decide to extend this task, we have to change the I/O format. For ex: C(1,3,5,7) so that we can evaluate same results against triads or quads easily. In terms of evaluations, we performed a really simple one last year. This year we welcome evaluations scripts written by the community.

A possible simplification of the data output could be to use chromatic numeric notation for the intervals. For example, C(1,3,5,7) would be C(0,4,7,11) or to be a bit more pure, something like 0(0,4,7,11) is cool but a bit redundant, leading us to 0,4,7,11. Dr. Downie prefers the chromatic numeric notation as it instantaneously gets rid of the enharmonic spelling problem.

Looking at the last years results for Chord Detection: https://www.music-ir.org/mirex/2008/index.php/Audio_Chord_Detection_Results

the performance increase for doing a training for chord detection seems to be insignificant. Would you consider dropping the test train part of the task this year?

Potential Participants

• Johan Pauwels/Ghent University, Belgium (firstname.lastname@elis.ugent.be) (still interested)
• Matthias Mauch, Centre for Digital Music, Queen Mary, University of London --Matthias 10:33, 27 June 2009 (UTC)
• Laurent Oudre, TELECOM ParisTech, France (firstname.lastname@telecom-paristech.fr) (still interested, probably 2 algorithms)
• Maksim Khadkevich, Fondazione Bruno Kessler, Italy (lastname_at_fbk_dot_eu) (still interested, 1 algorithm)
• Thomas Rocher, LaBRI Universit├⌐ Bordeaux 1, France (firstname.lastname@labri.fr)
• Yushi Ueda, The University of Tokyo, Japan (lastname@hil.t.u-tokyo.ac.jp)
• Christopher Harte, Centre for Digital Music, Queen Mary, University of London (firstname_dot_lastname_at_elec_dot_qmul_dot_ac_dot_uk)
• Helene Papadopoulos, IRCAM (firstname_dot_lastname_at_ircam.fr)
• Adrian Weller, Daniel Ellis and Tony Jebara, Columbia University, NY, USA (aw2506@columbia.edu)
• Your name here

Bibliography

1.Harte,C.A. and Sandler,M.B.(2005). Automatic chord identification using a quantised chromagram. Proceedings of 118th Audio Engineering Society's Convention.

2.Sailer,C. and Rosenbauer K.(2006). A bottom-up approach to chord detection. Proceedings of International Computer Music Conference 2006.

3.Shenoy,A. and Wang,Y.(2005). Key, chord, and rythm tracking of popular music recordings. Computer Music Journal 29(3), 75-86.

4.Sheh,A. and Ellis,D.P.W.(2003). Chord segmentation and recognition using em-trained hidden markov models. Proceedings of 4th International Conference on Music Information Retrieval.

5.Yoshioka,T. et al.(2004). Automatic Chord Transcription with concurrent recognition of chord symbols and boundaries. Proceedings of 5th International Conference on Music Information Retrieval.

6.Harte,C. and Sandler,M. and Abdallah,S. and G├│mez,E.(2005). Symbolic representation of musical chords: a proposed syntax for text annotations. Proceedings of 6th International Conference on Music Information Retrieval.

7.Papadopoulos,H. and Peeters,G.(2007). Large-scale study of chord estimation algorithms based on chroma representation and HMM. Proceedings of 5th International Conference on Content-Based Multimedia Indexing.

8.Samer Abdallah, Katy Noland, Mark Sandler, Michael Casey & Christophe Rhodes: Theory and Evaluation of a Bayesian Music Structure Extractor (pp. 420-425) Proc. 6th International Conference on Music Information Retrieval, ISMIR 2005.