Difference between revisions of "2015:GC15UX:JDISC"

From MIREX Wiki
(Dataset)
(Dataset)
Line 15: Line 15:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ skill
 
|+ skill
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id || auto_increment
 
| id || auto_increment
Line 27: Line 30:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ person
 
|+ person
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id || auto_increment
 
| id || auto_increment
Line 55: Line 61:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ person_skill
 
|+ person_skill
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| person || Reference to the person with this skill
 
| person || Reference to the person with this skill
Line 65: Line 74:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ person_ethnicity
 
|+ person_ethnicity
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id ||  auto_increment
 
| id ||  auto_increment
Line 77: Line 89:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ session
 
|+ session
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id ||  auto_increment
 
| id ||  auto_increment
Line 119: Line 134:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ person_session_skill
 
|+ person_session_skill
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| person || Reference to the person playing in this session
 
| person || Reference to the person playing in this session
Line 131: Line 149:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ session_leader
 
|+ session_leader
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| session || Reference to the session being led
 
| session || Reference to the session being led
Line 141: Line 162:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ composition
 
|+ composition
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id ||  auto_increment
 
| id ||  auto_increment
Line 153: Line 177:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ composition_title
 
|+ composition_title
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id ||  auto_increment
 
| id ||  auto_increment
Line 165: Line 192:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ composition_composer
 
|+ composition_composer
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| composition || Reference to the composition composed by this person
 
| composition || Reference to the composition composed by this person
Line 175: Line 205:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ composition_lyricist
 
|+ composition_lyricist
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| composition || Reference to the composition with lyrics by this person
 
| composition || Reference to the composition with lyrics by this person
Line 185: Line 218:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ track
 
|+ track
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id ||  auto_increment
 
| id ||  auto_increment
Line 211: Line 247:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ person_track_skill
 
|+ person_track_skill
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| person || Reference to the person playing (or not) on this track
 
| person || Reference to the person playing (or not) on this track
Line 223: Line 262:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ track_soloist
 
|+ track_soloist
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| track || Reference to the track on which this person had a solo
 
| track || Reference to the track on which this person had a solo
Line 233: Line 275:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ track_composition
 
|+ track_composition
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| track || Reference to the track as which this composition was recorded
 
| track || Reference to the track as which this composition was recorded
Line 243: Line 288:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ issue
 
|+ issue
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| id ||  auto_increment
 
| id ||  auto_increment
Line 309: Line 357:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ issue_leader
 
|+ issue_leader
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| issue || Reference to the release being led
 
| issue || Reference to the release being led
Line 329: Line 380:
 
{| class="wikitable"
 
{| class="wikitable"
 
|+ issue_overdub_people
 
|+ issue_overdub_people
 +
|-
 +
! Column
 +
! Description
 
|-
 
|-
 
| issue || Reference to the release being overdubbed
 
| issue || Reference to the release being overdubbed

Revision as of 00:41, 28 July 2015

Purpose

Holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems.


Goals

  1. To inspire the development of complete MIR systems.
  2. To promote the notion of user experience as a first-class research objective in the MIR community.


Dataset

  1. simple text description (Yun)
  2. more detailed description of the schema (Yun)
skill
Column Description
id auto_increment
name English label for this skill or instrument
jdisc_nid Legacy JDISC node ID for this skill
jdisc_vid Legacy JDISC version ID for this skill
person
Column Description
id auto_increment
name Best-known English name for this person
name_first First or given name for this person
name_last Last or family name for this person
alternate_name Alternative name (usu. birth name) for this person
gender Textual represenation of person’s gender
birth_date_earliest Earliest possible date of person’s birth
birth_date_latest Last possible date of person’s birth
birth_place Textual representation of place of person’s birth, if known
region_origin Textual representation of person’s region of origin, if known
jdisc_nid Legacy JDISC node ID for this skill
jdisc_vid Legacy JDISC version ID for this skill
person_skill
Column Description
person Reference to the person with this skill
skill Reference to the skill held by this person
order_key Ordering of the skills relative to the person
person_ethnicity
Column Description
id auto_increment
person Reference to the person being described
ethnicity Textual description of the person’s ethnicity
order_key Ordering of ethnicities relative to the person
session
Column Description
id auto_increment
name Human-readable label for the session
venue_type Type of venue where this session took place
venue_name Name of the venue
venue_location Location of the venue
label_name Name of the record label recording this session
ensemble_size Name of the size of the ensemble (trio, quartet, etc.)
company_name Name of the company making the recording
parent_company_name Name of the corporate parent of the recording company
producer Reference to the person who produced the session
engineer Reference to the person who was engineer for the session
date_earliest Earliest possible date of the session
date_latest Last possible date of the session
date_precision Date certainty (as single character, y, m, or d)
session_details_value Free text description of the session
session_details_format 1: filtered HTML; 2: raw HTML; 3: PHP code(!)
sources Free text listing of sources for information about this recording
jdisc_nid Legacy JDISC node ID for this session
jdisc_vid Legacy JDISC version ID for this session
person_session_skill
Column Description
person Reference to the person playing in this session
session Reference to the session performed in by this person
skill Reference to the skill performed by this person in this session
order_key Ordering of the people relative to the session
session_leader
Column Description
session Reference to the session being led
person Reference to the person leading the session
order_key Ordering of the leaders relative to the session
composition
Column Description
id auto_increment
title Human-readable primary title for this composition
jdisc_nid Legacy JDISC node ID for this composition
jdisc_vid Legacy JDISC version ID for this composition
composition_title
Column Description
id auto_increment
composition Reference to the composition for which this is an alternative title
title Alternative title for this composition
order_key Ordering of the titles relative to the composition
composition_composer
Column Description
composition Reference to the composition composed by this person
composer Reference to a person who composed this composition
order_key Ordering of this composer, relative to this composition
composition_lyricist
Column Description
composition Reference to the composition with lyrics by this person
lyricist Reference to a person who wrote lyrics for this composition
order_key Ordering of this lyricist, relative to this composition
track
Column Description
id auto_increment
name Human-readable title for the track
session Reference to the session in which this track was recorded, if known
order_key Ordering of the track relative to the session, if known
length Textual representation of track length, mm:ss
matrix_number Track identification code used by recording company
production_status Textual representation of usage of track (master, alternate, etc.)
release_number Textual representation of track usage on a release, possibly redundant with reified release information
splicing Indicates if track was spliced with another recorded track for release
jdisc_nid Legacy JDISC node ID for this track
jdisc_vid Legacy JDISC version ID for this track
person_track_skill
Column Description
person Reference to the person playing (or not) on this track
track Reference to the track performed on (or not) by this person
skill Reference to the skill performed, or absence thereof, by this person in on this track
order_key Ordering of the people relative to the track
track_soloist
Column Description
track Reference to the track on which this person had a solo
soloist Reference to the person who had a solo on this track
order_key Ordering of the soloists relative to the track
track_composition
Column Description
track Reference to the track as which this composition was recorded
composition Reference to the composition recorded as this track
order_key Ordering of the compositions relative to the track
issue
Column Description
id auto_increment
name Human-readable title for this release
alternate_leader Reference to the alternate leader on this release
label_name Name of the issuing label
catalog_number Catalog number given to this release by the label or company
date Date of this release
format Physical medium of this release (LP, CD, etc.)
num_channels Number of audio channels used for this release
length Textual representation of the playing time of this release
status Status of this release (original, reissue, etc.)
producer Reference to the person who produced this release
master_engineer Reference to the person who mastered this release
reissue_producer Reference to the person who produced this reissued release, if applicable
reissue_engineer Reference to the person who engineered this reissued release, if applicable
remaster Indicates that this release was remastered
remaster_engineer Reference to the person who engineered the remastering process, if applicable
remix Indicates that this release is a remix
remix_engineer Reference to the person who engineered the remixing process, if applicable
overdub Indicates that this release has tracks combined (overdubbed) from multiple sessions
method_notes Free text notes on the post-production editing process for this release
editing_facility Name of the facility where this release was edited
liner_notes_author Reference to the person who wrote the notes for this release
language Language of the text associated with this release (e.g., liner notes)
cover_artist Reference to the person who created art for the release cover, if any
design_supervisor Reference to the person who supervised the design of this release
photographer Reference to the person who took photographs included in this release
release_details_value Free text description of the release
release_details_format 1: filtered HTML; 2: raw HTML; 3: PHP code(!)
sources Free text listing of sources for information about this release
jdisc_nid Legacy JDISC node ID for this release
jdisc_vid Legacy JDISC version ID for this release
issue_leader
Column Description
issue Reference to the release being led
person Reference to the person leading the release
order_key Ordering of the leaders relative to the release
issue_track
issue Reference to the release on which this track appears
track Reference to the track appearing on this release
order_key Ordering of the tracks relative to the release
issue_overdub_people
Column Description
issue Reference to the release being overdubbed
person Reference to the person working on the overdubbing
order_key Ordering of the people relative to the release



  1. visualization of the schema (ER diagram) (Chris)

Download the Dataset

  1. Use the linked MySQL script to download
  2. user agreement signed during download?


Participating Systems

Unlike conventional MIREX tasks, participants are not asked to submit their systems. Instead, the systems will be hosted by their developers. All participating systems need to be constructed as websites accessible to users through normal web browsers. Participating teams will submit the URLs to their systems to the GC15UX team.

To ensure a consistent experience, evaluators will see participating systems in fixed size window: 1024x768. Please test your system for this screen size.

See the #Evaluation Webforms below for a better understanding of our E6K-inpsired evaluation system design.

Potential Participants

Please put your names and email contacts in the following table. It is encouraged that you give your team a cool name!

(Cool) Team Name Name(s) Email(s)

Evaluation

As written in the name of the Grand Challenge, the evaluation will be user-centered. All systems will be used by a number of human evaluators and be rated by them on several most important criteria in evaluating user experience.

Criteria

Note that the evaluation criteria or its descriptions may be slightly changed in the months leading up to the submission deadline, as we test it and work to improve it.

Given the GC15UX is all about how users perceive their experiences of the systems, we intend to capture the user perceptions in a minimally intrusive manner and not to burden the users/evaluators with too many questions or required data inputs. The following criteria are grounded on the literature of Human Computer Interaction (HCI) and User Experience (UX), with a careful consideration on striking a balance between being comprehensive and minimizing evaluators' cognitive load.

Evaluators will rate systems on the following criteria:

  • Overall satisfaction: How would you rate your overall satisfaction with the system?

Very unsatisfactory / Unsatisfactory / Slightly unsatisfactory / Neutral / Slightly satisfactory / Satisfactory / Very satisfactory

  • Aesthetics: How would you rate the visual attractiveness of the system?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

  • Ease of use: How easy was it to figure out how to use the system?

Very difficult / Difficult / Slightly difficult / Neutral / Slightly easy / Easy / Very easy

  • Clarity: How well does the system communicate what is going on?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

  • Affordances: How well does the system allow you to perform what you want to do?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

  • Performance: Does the system work efficiently and without bugs/glitches?

Very Poor / Poor / Slightly Poor / Neutral / Slightly Good / Good / Excellent

  • Open Text Feedback: An open-ended question is provided for evaluators to give feedback if they wish to do so.

Evaluators

Evaluators will be users aged 18 and above. For this round, evaluators will be drawn primarily from the MIR community through solicitations via the ISMIR-community mailing list. The #Evaluation Webforms developed by the GC15UX team will ensure all participating systems will get equal number of evaluators.

Tasks for Evaluators

To motivate the evaluators, a defined yet open task is given to the evaluators:

You need to put together a playlist for a particular event (e.g., dinner party at your house, workout session). Try to use the assigned system to make playlists for at least a couple of different events.

The task is to ensure that evaluators have a (more or less) consistent goal when they interact with the systems. The goal is flexible and authentic to the evaluators' purposes ("music for their own situation"). As the task is not too specific, evaluators can potentially look for a wide range of music in terms of genre, mood and other aspects. This allows great flexibility and virtually unlimited possibility in system or service design.

Another important consideration in designing the task is the music collection available for this GC15UX: the Jamando collection. Jamando music is not well-known to most users/evaluators, whereas many more commonly seen music information tasks are more or less influenced by users' familiarity to the songs and song popularity. Through this task of "finding (copyright-free) background music for a self-made video", we strive to minimize the need of looking for familiar or popular music.

Evaluation Results

Statistics of the scores given by all evaluators will be reported: mean, average deviation. Meaningful text comments from the evaluators will also be reported.

Evaluation Webforms

Graders can take as many assignments as they wish in the My Assignments page. They are allowed to go back to the evaluation page anytime by clicking the thumbnail of the submission.

GCUX wireframe my assignments.png
To facilitate the evaluators and minimize their burden, the GC15UX team will provide a set of evaluation forms which wrap around the participating systems. As shown in the following image, the evaluation webforms are for scoring the participating systems, with their client interfaces embedded within an iframe in the left side of the webform.

GCUX wireframe evaluation.png

Organization

Important Dates

This year GC15UX:JDISC adopts the two-phase model with two evaluations. The first phase will end by the ISMIR conference and we will disclose preliminary results at the conference. Then, phase II will start. Participating developers can continue improving their systems based on the feedback from the first phase and another round of evaluation will be conducted in February. We believe that this model serves the developers well since it is in accordance with the iterative nature of user-centered design. In this way, the developers will also have enough time to develop their complete MIR systems.

  • July ?: announce the GC15UX:JDISC
  • Sep. 28st: the first deadline for system submission
  • Feb. 28st: the second deadline for system submission

What to Submit

A URL to the participanting system.

Contacts

The GC15UX team consists of:

J. Stephen Downie, University of Illinois (MIREX director)
Xiao Hu, University of Hong Kong (ISMIR2014 co-chair)
Jin Ha Lee, University of Washington (ISMIR2014 program co-chair)
Yi-Hsuan (Eric) Yang, Academic Sinica, Taiwan (ISMIR2014 program co-chair)
David Bainbridge, Waikato University, New Zealand
Christopher R. Maden, University of Illinois
Kahyun Choi, University of Illinois
Peter Organisciak, University of Illinois
Yun Hao, University of Illinois


Inquiries, suggestions, questions, comments are all highly welcome! Please contact Prof. Downie [1] or anyone in the team.