Difference between revisions of "2025:Symbolic Music Generation"

From MIREX Wiki
(Created page with "=Description= Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have...")
 
 
Line 1: Line 1:
 
=Description=
 
=Description=
−
Symbolic music generation is a broad topic. It covers a wide range of tasks, including generation, harmonization, arrangement, instrumentation, and more. We have multiple ways to represent music data, and the evaluation metrics also vary. To define a MIREX challenge within this topic, we need to narrow our focus to specific subtasks that are both relevant to the community and feasible to evaluate effectively.
 
  
−
This year, we select the task to be '''piano accompaniment arrangement from a lead sheet'''. The lead sheet provides information about the melody, and chord progression. The goal is to generate a piano accompaniment that complements the lead melody. The music data consists of 8-measure segments in 4/4 meter, quantized to a sixteenth-note resolution. A more detailed description of the data structure is provided in the data format section. The genre of the lead sheets is broadly within western pop music (refer to the music examples for more detail).
+
This year, the selected task is '''piano music continuation'''. Given a 4-measure piano prompt, the goal is to generate a 12-measure continuation that is musically coherent with the prompt, resulting in a complete 16-measure piece. All music is assumed to be in 4/4 meter, quantized to sixteenth-note resolution, and may belong to any existing piano style, including classical, pop, jazz, and others. Further details on the data structure are provided in the Data Format section.
 +
 
  
 
=Data Format=
 
=Data Format=
−
The input lead sheet consists of 8 bars for the melody and harmony, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing two properties: <code>melody</code> and <code>chords</code>:
+
The input prompt consists of 4 bars of piano music, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing a property: <code>prompt</code>:
  
−
* <code>melody</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.
+
* <code>prompt</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.
  
−
* <code>chords</code>: a list of chords. Each chord contains properties of <code>start</code>, <code>symbol</code>, and <code>duration</code>.
+
The output generation should also follow the JSON format containing one property <code>output</code>:
  
−
The output generation should also follow the JSON format containing one property <code>acc</code>:
+
* <code>output</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.
−
 
 
−
* <code>acc</code>: a list of notes. Each note contains properties of <code>start</code>, <code>pitch</code>, and <code>duration</code>.
 
  
 
'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''  
 
'''Detailed explanation of <code>start</code> and <code>duration</code> attributes.'''  
  
−
# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both melody and chords, onsets and durations are counted in sixteenth notes.  
+
# The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both prompt and output, onsets and durations are counted in sixteenth notes.  
−
# Both onsets and durations are integers ranging from 0 to 9 * 16 - 1 = 143. Notes that end later than the ninth measure (i.e., 9 * 16 = 144th time step) will be truncated to the end of the ninth measure.
+
# Both onsets and durations are integers ranging from 0 to 17 * 16 - 1 = 271. Notes that end later than the sixteenth measure (i.e., 17 * 16 = 272th time step) will be truncated to the end of the ninth measure.  
−
# Melody notes are not allowed to overlap with one another.
 
−
# There should be no gaps or overlaps between chords. Chords must follow one another directly. If there is a blank space where no chord is played, it must be filled with the <code>N</code> chord.  
 
 
# The accompaniment of the pick-up measure should be blank.  
 
# The accompaniment of the pick-up measure should be blank.  
  
Line 26: Line 22:
  
 
# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.
 
# The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.
−
 
−
'''Detailed explanation of the chord <code>symbol</code> attribute.'''
 
−
 
−
# The symbol property of a chord should be a string based on the syntax of (Harte, 2010). In other words, each chord string should be able to be passed as a parameter to mir_eval.chord.encode() without causing an error.
 
  
  
Line 37: Line 29:
 
<pre>
 
<pre>
 
{
 
{
−
   "melody": [
+
   "prompt": [
 
     {"start": 12, "pitch": 72, "duration": 4},
 
     {"start": 12, "pitch": 72, "duration": 4},
 
     {"start": 16, "pitch": 69, "duration": 8},
 
     {"start": 16, "pitch": 69, "duration": 8},
 
     ...
 
     ...
 
   ],
 
   ],
−
  "chords": [
 
−
    {"start": 0, "symbol": "N", "duration": 16},
 
−
    {"start": 16, "symbol": "F", "duration": 16},
 
−
    ...
 
−
  ]
 
 
}
 
}
 
</pre>
 
</pre>
  
  
−
This is an example of the generated accompaniment. The accompaniment is generated using the baseline method WholeSongGen introduced below. Note that the generation starts from the second measure (time step 16).
+
This is an example of the generated continuation. Note that the generation starts from the fifth measure (time step 80).
  
 
<pre>
 
<pre>
 
{
 
{
 
   "acc": [
 
   "acc": [
−
     {"start": 16, "pitch": 41, "duration": 12},
+
     {"start": 80, "pitch": 41, "duration": 12},
−
     {"start": 16, "pitch": 65, "duration": 5},
+
     {"start": 80, "pitch": 65, "duration": 5},
 
     ...
 
     ...
 
   ]
 
   ]
 
}
 
}
 
</pre>
 
</pre>
−
Full data examples can be accessed in [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main/generation_samples this code repository]. MIDI conversion code and MIDI demos are also provided there.
 
  
  
 
=Evaluation and Competition Format=
 
=Evaluation and Competition Format=
 
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
 
We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:
−
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two lead sheets.''' The organizer team will supplement the lead sheet if necessary.  
+
* '''We use a "''potluck''" test set. Before submitting the algorithm, each team is required to submit two prompts.''' The organizer team will supplement the prompts if necessary.  
 
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
 
* There will be '''no live ranking''' because the subjective test will be done after the algorithm submission deadline.
 
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.   
 
* To better handle randomness in the generation algorithm, we '''allow cherry-picking from a fixed number of generated samples'''.   
Line 73: Line 59:
  
 
==Subjective Evaluation Format==
 
==Subjective Evaluation Format==
−
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 arrangements''' for each test sample. The generated results will be returned to each team for cherry-picking.
+
* After each team submits the algorithm, the organizer team will use the algorithm to generate '''16 continuations''' for each test sample. The generated results will be returned to each team for cherry-picking.
 
* Only a subset of the test set will be used for subjective evaluation.
 
* Only a subset of the test set will be used for subjective evaluation.
−
* In the subjective evaluation, we will first ask the subjects to listen to the lead melody with chords and then listen to the generated samples in random order. The order of the samples will be randomized.
+
* In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
 
* The subject will be asked to rate each arrangement based on the following criteria:
 
* The subject will be asked to rate each arrangement based on the following criteria:
−
:* Harmony correctness (5-point scale)
+
:* Coherency (5-point scale)
 
:* Creativity (5-point scale)
 
:* Creativity (5-point scale)
−
:* Naturalness (5-point scale)
+
:* Structuredness (5-point scale)
 
:* Overall musicality (5-point scale)
 
:* Overall musicality (5-point scale)
  
−
==Objective Measurements==
 
−
* We will use objective measurements only as a reference. The correlation between subjective and objective scores will be measured as a reference.
 
−
* The current plan is to compute the Negative Log Likelihood of a large music language model (e.g., Lu et al., 2023).
 
−
* We welcome proposals of the objective measurements.
 
  
−
==Important Dates==
+
==Important Dates (Tentative)==
  
−
* '''Oct 8, 2024''': Submit two lead sheets as a part of the test set.  
+
* '''Aug 7, 2025''': Submit two lead sheets as a part of the test set.  
−
* '''Oct 15, 2024''': Submit the main algorithm.
+
* '''Aug 15, 2025''': Submit the main algorithm.
−
* '''Oct 22, 2024''': Return the generated samples. The cherry-picking phase begins.
+
* '''Aug 20, 2025''': Return the generated samples. The cherry-picking phase begins.
−
* '''Oct 25, 2024''': Submit the cherry-picked sample ids.
+
* '''Aug 24, 2025''': Submit the cherry-picked sample ids.
−
* '''Oct 31 - Nov 3, 2024''': Online subjective evaluation.
+
* '''Aug 30 - Sep 5, 2024''': Online subjective evaluation.
−
* '''Nov 5, 2024''': Announce the final result.
+
* '''Sep 6, 2024''': Announce the final result.
  
  
Line 100: Line 82:
  
 
As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
 
As a generative task with subjective evaluation, the submission process ''differs greatly'' from other MIREX tasks. There are four important stages:
−
# Test set submission (Oct 8, 2024)
+
# Test set submission  
−
# Algorithm submission (Oct 15, 2024)
+
# Algorithm submission
−
# Cherry-picked sample IDs submission (Oct 25, 2024)
+
# Cherry-picked sample IDs submission  
−
# Evaluation form submission (Nov 3, 2024)
+
# Evaluation form submission  
 
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''
 
Please check the Important Dates section for the detailed schedule. '''Failure to participate in any of the stages will result in disqualification.'''
  
  
 
==Algorithm Submission==
 
==Algorithm Submission==
−
Participants must include an <code>batch_acc_gen.sh</code> script in their submission. The task captain will use the script to generate output files according to the following format:
+
To be announced later.
−
 
 
−
'''Usage'''
 
−
 
 
−
<pre>
 
−
acc_gen.sh "/path/to/input.json" "/path/to/output_folder" n_sample
 
−
</pre>
 
−
 
 
−
* Input File: Path to the input .json file.
 
−
* Output Folder: Path to the folder where the generated output files will be saved.
 
−
* n_sample: Number of samples to generate.
 
−
 
 
−
'''Output'''
 
−
* The script should generate n_sample output files in the specified output folder.
 
−
* Output files should be named sequentially as sample_01.json, sample_02.json, ..., up to sample_n_sample.json.
 
−
 
 
−
Participants are free to implement the internal logic of the script, but it must adhere to this format for proper execution during the evaluation process.
 
−
 
 
−
'''Packaging Submissions'''
 
−
* Every submission must be packed into a docker image
 
−
* Every submission will be deployed and evaluated automatically with <code>docker run</code>
 
−
 
 
−
'''Accepted submission form'''
 
−
* Link to public or private Github repository
 
−
* Link to public or private docker hub
 
−
* Shared google drive links
 
−
* If the repository is private, an access token is also required
 
  
  
 
=Baselines=
 
=Baselines=
  
−
To establish a benchmark for this task, we consider the three baseline models in their official implementations:
+
To be announced later.
−
 
 
−
'''WholeSongGen''' (Wang et al., 2024)
 
−
* A denoising diffusion probabilistic model (DDPM) generating piano accompaniments as piano-roll images.
 
  
−
'''Compose & Embellish''' (Wu and Yang, 2023)
 
−
* A Transformer-based architecture generating piano performances in beat-based event sequences.
 
−
 
−
'''AccoMontage''' (Zhao and Xia, 2021)
 
−
* A hybrid algorithm generating piano accompaniments by rule-based search and music representation learning.
 
  
 
=Contacts=
 
=Contacts=
Line 154: Line 102:
 
* Ziyu Wang: ziyu.wang<at>nyu.edu
 
* Ziyu Wang: ziyu.wang<at>nyu.edu
 
* Jingwei Zhao: jzhao<at>u.nus.edu
 
* Jingwei Zhao: jzhao<at>u.nus.edu
−
 
−
 
−
=References=
 
−
* Harte, C. Towards automatic extraction of harmony information from music signals. PhD Diss. 2010.
 
−
* Lu, P., et al. Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110 (2023).
 
−
* Wang, Z., et al. Whole-song hierarchical generation of symbolic music using cascaded diffusion models, in ICLR 2024.
 
−
* Wu, S.-L., & Yang, Y.-H. Compose & Embellish: Well-structured piano performance generation via a two-stage approach, in ICASSP 2023.
 
−
* Zhao, J., & Xia, G. Accomontage: Accompaniment arrangement via phrase selection and style transfer, in ISMIR 2021.
 
−
 
−
* Code and data format samples: [https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main https://github.com/ZZWaang/acc-gen-8bar-wholesong/tree/main]
 

Latest revision as of 10:14, 30 May 2025

Description

This year, the selected task is piano music continuation. Given a 4-measure piano prompt, the goal is to generate a 12-measure continuation that is musically coherent with the prompt, resulting in a complete 16-measure piece. All music is assumed to be in 4/4 meter, quantized to sixteenth-note resolution, and may belong to any existing piano style, including classical, pop, jazz, and others. Further details on the data structure are provided in the Data Format section.


Data Format

The input prompt consists of 4 bars of piano music, with an additional mandatory pickup measure (left blank if not used). The data is prepared in JSON format containing a property: prompt:

  • prompt: a list of notes. Each note contains properties of start, pitch, and duration.

The output generation should also follow the JSON format containing one property output:

  • output: a list of notes. Each note contains properties of start, pitch, and duration.

Detailed explanation of start and duration attributes.

  1. The data is assumed to be in 4/4 meter, quantized to a sixteenth-note resolution. For both prompt and output, onsets and durations are counted in sixteenth notes.
  2. Both onsets and durations are integers ranging from 0 to 17 * 16 - 1 = 271. Notes that end later than the sixteenth measure (i.e., 17 * 16 = 272th time step) will be truncated to the end of the ninth measure.
  3. The accompaniment of the pick-up measure should be blank.

Detailed explanation of the pitch attribute.

  1. The pitch property of a note should be integers ranging from 0 to 127, corresponding to the MIDI pitch numbers.


Data Example

Below is an example of the input lead sheet in the format given above. The lead sheet is the melody of the first phrase of Hey Jude by The Beatles. 

{
  "prompt": [
    {"start": 12, "pitch": 72, "duration": 4},
    {"start": 16, "pitch": 69, "duration": 8},
    ...
  ],
}


This is an example of the generated continuation. Note that the generation starts from the fifth measure (time step 80). 

{
  "acc": [
    {"start": 80, "pitch": 41, "duration": 12},
    {"start": 80, "pitch": 65, "duration": 5},
    ...
  ]
}


Evaluation and Competition Format

We will evaluate the submitted algorithms through an online subjective double-blind test. The evaluation format differs from conventional tasks in the following aspects:

  • We use a "potluck" test set. Before submitting the algorithm, each team is required to submit two prompts. The organizer team will supplement the prompts if necessary.
  • There will be no live ranking because the subjective test will be done after the algorithm submission deadline.
  • To better handle randomness in the generation algorithm, we allow cherry-picking from a fixed number of generated samples.
  • We hope to compute some objective measurements as well, but these will only be reported as a reference.

Subjective Evaluation Format

  • After each team submits the algorithm, the organizer team will use the algorithm to generate 16 continuations for each test sample. The generated results will be returned to each team for cherry-picking.
  • Only a subset of the test set will be used for subjective evaluation.
  • In the subjective evaluation, we will first ask the subjects to listen to the prompt and then listen to the generated samples in random order. The order of the samples will be randomized.
  • The subject will be asked to rate each arrangement based on the following criteria:
  • Coherency (5-point scale)
  • Creativity (5-point scale)
  • Structuredness (5-point scale)
  • Overall musicality (5-point scale)


Important Dates (Tentative)

  • Aug 7, 2025: Submit two lead sheets as a part of the test set.
  • Aug 15, 2025: Submit the main algorithm.
  • Aug 20, 2025: Return the generated samples. The cherry-picking phase begins.
  • Aug 24, 2025: Submit the cherry-picked sample ids.
  • Aug 30 - Sep 5, 2024: Online subjective evaluation.
  • Sep 6, 2024: Announce the final result.


Submission

As a generative task with subjective evaluation, the submission process differs greatly from other MIREX tasks. There are four important stages:

  1. Test set submission
  2. Algorithm submission
  3. Cherry-picked sample IDs submission
  4. Evaluation form submission

Please check the Important Dates section for the detailed schedule. Failure to participate in any of the stages will result in disqualification.


Algorithm Submission

To be announced later.


Baselines

To be announced later.


Contacts

If you any questions or suggestions about the task, please contact:

  • Ziyu Wang: ziyu.wang<at>nyu.edu
  • Jingwei Zhao: jzhao<at>u.nus.edu
Retrieved from "https://www.music-ir.org/mirex/w/index.php?title=2025:Symbolic_Music_Generation&oldid=14657"