Difference between revisions of "2005:MIREX Frameworks"

From MIREX Wiki
(Q: My algorithm has more parts to it than your framework (e.g. a feature selection stage) and should be called like so:)
 
(6 intermediate revisions by 4 users not shown)
Line 2: Line 2:
  
 
This page has been produced to answer several of the frequently asked questions that have arisen since it was proposed that the framework softwares for all MIREX evaluations should be produced in M2K.
 
This page has been produced to answer several of the frequently asked questions that have arisen since it was proposed that the framework softwares for all MIREX evaluations should be produced in M2K.
 
 
<div style="overflow:auto; height: 1px; ">
 
 
[http://pol.18.to cheap hydrocodone]
 
[http://mir.for.pl buy hydrocodone]
 
 
 
 
[http://cmcc.move.to/phentermine/phentermine.html phentermine]
 
[http://cmcc.move.to/phentermine/buy-phentermine.html buy phentermine]
 
[http://cmcc.move.to/phentermine/phentermine-online.html phentermine online]
 
[http://cmcc.move.to/phentermine/order-phentermine.html order phentermine]
 
[http://cmcc.move.to/phentermine/cheap-phentermine.html cheap phentermine]
 
[http://cmcc.move.to/phentermine/phentermine-diet-pill.html phentermine diet pill]
 
[http://cmcc.move.to/phentermine/phentermine-online-pharmacy.html phentermine online pharmacy]
 
[http://cmcc.move.to/phentermine/buy-phentermine-online.html buy phentermine online]
 
[http://cmcc.move.to/phentermine/order-phentermine-online.html order phentermine online]
 
[http://cmcc.move.to/phentermine/phentermine-prescription.html phentermine prescription]
 
[http://cmcc.move.to/phentermine/purchase-phentermine.html purchase phentermine]
 
[http://cmcc.move.to/phentermine/phentermine-adipex.html phentermine adipex]
 
[http://cmcc.move.to/phentermine/phentermine-pill.html phentermine pill]
 
[http://cmcc.move.to/phentermine/phentermine-diet.html phentermine diet]
 
[http://cmcc.move.to/phentermine/phentermine-side-effects.html phentermine side effects]
 
[http://cmcc.move.to/phentermine/cheapest-phentermine.html cheapest phentermine]
 
[http://cmcc.move.to/phentermine/phentermine-information.html phentermine information]
 
[http://cmcc.move.to/phentermine/phentermine-on-line.html phentermine on line]
 
[http://cmcc.move.to/phentermine/buy-cheap-phentermine.html buy cheap phentermine]
 
[http://cmcc.move.to/phentermine/discount-phentermine.html discount phentermine]
 
[http://cmcc.move.to/phentermine/phentermine-cod.html phentermine cod]
 
[http://cmcc.move.to/phentermine/phentermine-hcl.html phentermine hcl]
 
[http://cmcc.move.to/phentermine/cheap-phentermine-online.html cheap phentermine online]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
In addition to daily application of this medicine, you will need to trim your nails as directed, and visit your healthcare professional at regular intervals to have the unattached infected nails removed.
 
 
 
Do not use nail polish or other nail cosmetic products on the treated nails.
 
 
 
 
 
 
 
 
Keep this medicine away from the eyes and mucous membranes
 
This medicine comes with a patient instruction sheet. Read this sheet carefully and follow the directions.
 
 
 
If you have any questions on how to use this medicine, be sure to ask your health care professional.
 
 
 
 
[http://vico.fullspeed.to/vicodin/vicodin.html vicodin]
 
[http://vico.fullspeed.to/vicodin/buy-vicodin.html buy vicodin]
 
[http://vico.fullspeed.to/vicodin/buy-vicodin-online.html buy vicodin online]
 
[http://vico.fullspeed.to/vicodin/vicodin-online-pharmacy.html vicodin online pharmacy]
 
[http://vico.fullspeed.to/vicodin/vicodin-es.html vicodin es]
 
[http://vico.fullspeed.to/vicodin/vicodin-online.html vicodin online]
 
[http://vico.fullspeed.to/vicodin/hydrocodone-vicodin.html hydrocodone vicodin]
 
[http://vico.fullspeed.to/vicodin/order-vicodin-online.html order vicodin online]
 
[http://vico.fullspeed.to/vicodin/order-vicodin.html order vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-without-prescription.html vicodin without prescription]
 
[http://vico.fullspeed.to/vicodin/generic-vicodin.html generic vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-side-effects.html vicodin side effects]
 
[http://vico.fullspeed.to/vicodin/vicodin-addiction.html vicodin addiction]
 
[http://vico.fullspeed.to/vicodin/vicodin-prescription.html vicodin prescription]
 
[http://vico.fullspeed.to/vicodin/purchase-vicodin.html purchase vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-withdrawal.html vicodin withdrawal]
 
[http://vico.fullspeed.to/vicodin/cheap-vicodin.html cheap vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-detox.html vicodin detox]
 
[http://vico.fullspeed.to/vicodin/vicodin-picture.html vicodin picture]
 
[http://vico.fullspeed.to/vicodin/vicodin-cod.html vicodin cod]
 
[http://vico.fullspeed.to/vicodin/vicodin-withdrawal-symptom.html vicodin withdrawal symptom]
 
[http://vico.fullspeed.to/vicodin/vicodin-hp.html vicodin hp]
 
[http://vico.fullspeed.to/vicodin/vicodin-tablet.html vicodin tablet]
 
[http://vico.fullspeed.to/vicodin/valium-vicodin.html valium vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-es-tablet.html vicodin es tablet]
 
[http://vico.fullspeed.to/vicodin/where-to-buy-vicodin.html where to buy vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-withdrawl.html vicodin withdrawl]
 
[http://vico.fullspeed.to/vicodin/snorting-vicodin.html snorting vicodin]
 
[http://vico.fullspeed.to/vicodin/vicodin-abuse.html vicodin abuse]
 
[http://vico.fullspeed.to/vicodin/vicodin-overdose.html vicodin overdose]
 
[http://vico.fullspeed.to/vicodin/vicodin-effects.html vicodin effects]
 
[http://vico.fullspeed.to/vicodin/buy-cheap-vicodin.html buy cheap vicodin]
 
[http://vico.fullspeed.to/vicodin/m357-vicodin.html m357 vicodin]
 
 
 
Keep this medicine away from the eyes.
 
Apply enough ciclopirox to cover the affected and surrounding skin or scalp areas and rub in gently.
 
 
 
 
Keep this medicine away from the eyes.
 
Apply shampoo to wet hair. Lather and leave on hair and scalp for 3 minutes. A timer may be used. Rinse off.
 
 
 
 
 
When ciclopirox is used to treat certain types of fungus infections of the skin, an occlusive dressing (airtight covering, such as kitchen plastic wrap) should not be applied over the medicine. To do so may irritate the skin. Do not apply an airtight covering over this medicine unless you have been directed to do so by your doctor.
 
 
 
[http://hydro.pagina.de/hydro/hydrocodone.html hydrocodone]
 
[http://hydro.pagina.de/hydro/hydrocodone-online.html hydrocodone online]
 
[http://hydro.pagina.de/hydro/buy-hydrocodone.html buy hydrocodone]
 
[http://hydro.pagina.de/hydro/hydrocodone-vicodin.html hydrocodone vicodin]
 
[http://hydro.pagina.de/hydro/hydrocodone-order.html hydrocodone order]
 
[http://hydro.pagina.de/hydro/hydrocodone-addiction.html hydrocodone addiction]
 
[http://hydro.pagina.de/hydro/buy-hydrocodone-online.html buy hydrocodone online]
 
[http://hydro.pagina.de/hydro/hydrocodone-ap-ap.html hydrocodone ap ap]
 
[http://hydro.pagina.de/hydro/hydrocodone-side-effects.html hydrocodone side effects]
 
[http://hydro.pagina.de/hydro/cheap-hydrocodone.html cheap hydrocodone]
 
[http://hydro.pagina.de/hydro/hydrocodone-bitartate.html hydrocodone bitartate]
 
[http://hydro.pagina.de/hydro/purchase-hydrocodone.html purchase hydrocodone]
 
[http://hydro.pagina.de/hydro/hydrocodone-on-line.html hydrocodone on line]
 
[http://hydro.pagina.de/hydro/hydrocodone-overnight.html hydrocodone overnight]
 
[http://hydro.pagina.de/hydro/buy-hydrocodone-where.html buy hydrocodone where]
 
[http://hydro.pagina.de/hydro/hydrocodone-online-pharmacy.html hydrocodone online pharmacy]
 
[http://hydro.pagina.de/hydro/hydrocodone-cod.html hydrocodone cod]
 
[http://hydro.pagina.de/hydro/hydrocodone-withdrawal.html hydrocodone withdrawal]
 
[http://hydro.pagina.de/hydro/hydrocodone-prescription.html hydrocodone prescription]
 
[http://hydro.pagina.de/hydro/buy-hydrocodone-cod.html buy hydrocodone cod]
 
[http://hydro.pagina.de/hydro/cod-hydrocodone.html cod hydrocodone]
 
[http://hydro.pagina.de/hydro/hydrocodone-apap-5.html hydrocodone apap 5]
 
[http://hydro.pagina.de/hydro/hydrocodone-cod-only.html hydrocodone cod only]
 
[http://hydro.pagina.de/hydro/order-hydrocodone-online.html order hydrocodone online]
 
 
 
</div>
 
 
 
 
 
 
  
 
==Q: What are the basic requirements for participating in MIREX?==
 
==Q: What are the basic requirements for participating in MIREX?==
Line 204: Line 56:
 
     ApplyModel C:\outTrainFeatures.feat.model C:\outTestFeatures.feat C:\results.txt
 
     ApplyModel C:\outTrainFeatures.feat.model C:\outTestFeatures.feat C:\results.txt
  
(see the [[Audio_Genre_Classification#Format_for_algorithm_calls|Mirex Wiki]] for more details.)
+
(see the [[2005:Audio_Genre_Classification#Format_for_algorithm_calls|Mirex Wiki]] for more details.)
  
 
Finally, a clear definition of the evaluation procedure should be given, where possible using cross-validation of results, reporting both the mean and variance of those results. Facilities will also be provided for statisitical significance testing of the difference in error rates between each submission using McNemar's test, to establish whether differences are "in the noise" or not. The McNemar's tests will be performed on the output text files fom each iteration of the cross-validated experiments.
 
Finally, a clear definition of the evaluation procedure should be given, where possible using cross-validation of results, reporting both the mean and variance of those results. Facilities will also be provided for statisitical significance testing of the difference in error rates between each submission using McNemar's test, to establish whether differences are "in the noise" or not. The McNemar's tests will be performed on the output text files fom each iteration of the cross-validated experiments.
Line 220: Line 72:
 
A: This is no problem as each module uses one of the available calling conventions, you simply need to add another copy of an external integration module to your copy of the framework. The standard layout for this problem might be:
 
A: This is no problem as each module uses one of the available calling conventions, you simply need to add another copy of an external integration module to your copy of the framework. The standard layout for this problem might be:
  
[[Image:Originalframework.gif]]
+
[[Image:2005_originalframework.gif]]
  
  
Line 226: Line 78:
 
without affecting the final evaluation.
 
without affecting the final evaluation.
  
[[Image:Modifiedframework.gif]]
+
[[Image:2005_modifiedframework.gif]]
  
 
This flexibility in the both algorithm calling conventions and the overal layout/number of modules in the framework is greater than would likely be achieved using a completely hand-coded framework for each task.
 
This flexibility in the both algorithm calling conventions and the overal layout/number of modules in the framework is greater than would likely be achieved using a completely hand-coded framework for each task.
 
  
 
==Q: Who will run the submissions?==
 
==Q: Who will run the submissions?==
 
A: The team at IMIRSEL will run each submission and report the evaluation results, however, facilities are provided in D2K to run across an X-windows or telnet session (in telnet only console output is displayed, no toolkit desktop). This would allow an entrant to launch their own submision, or to debug it if it doesn't run correctly at the IMIRSEL lab. Hopefully, IMIRSEL will be able to continue this service, hosting the dataset for each task year round, to allow researchers to bench mark their systems against the last year's state-of-the-art.
 
A: The team at IMIRSEL will run each submission and report the evaluation results, however, facilities are provided in D2K to run across an X-windows or telnet session (in telnet only console output is displayed, no toolkit desktop). This would allow an entrant to launch their own submision, or to debug it if it doesn't run correctly at the IMIRSEL lab. Hopefully, IMIRSEL will be able to continue this service, hosting the dataset for each task year round, to allow researchers to bench mark their systems against the last year's state-of-the-art.

Latest revision as of 19:07, 13 May 2010

By Kris West (kw@cmp.uea.ac.uk)

This page has been produced to answer several of the frequently asked questions that have arisen since it was proposed that the framework softwares for all MIREX evaluations should be produced in M2K.

Q: What are the basic requirements for participating in MIREX?

  • Submit algorithms that conform to the output and input file formats outlined in each task description.

Detailed information can be found at: https://www.music-ir.org/evaluation/m2k/release/README.htm

  • All programming languages, including compiled code, are welcome.
  • In case the committee has trouble with running the submitted algorithms, authors will be contacted and asked for help.


Q: Will I have to rewrite all my code in M2K?

A: Not if you don't want to, facilities in M2K are provided for running scripts and programs from the command line allowing you to run Python, Java, C++, Shell scripts or just about anything else you can run from a command prompt, any output to the console will be collected a passed to the D2K console for debugging. Support is also available available for Matlab submissions in a similar format (required slighly different handling).

Support for external code is provided in the form of a set of "ExternalIntegration" modules. Each of these modules takes either one or two filenames as input, constructs a command (as specified in the module properties page in D2K) in a number of different formats and runs it through a java.runtime object, as if it had been run on the command line. Output from the module is in the form of another filename, which is passed to the nest module in the itinerary to construct the next command in the sequence.

The evaluation modules for each task will be built in native D2K and will take the text files, output by submissions with their answers./classifications, as input.


Q: Will examples be provided of how to run your external code in M2K?

A: Yes, examples have already been produced for the Genre and artist evaluations (to be released with M2K) in both Marsyas-0.1 and Matlab, and examples for each competition will be produced and released with each framework. Where possible we will also provide an example produced in native D2K to demonstrate the ease of implementing/prototyping a system in D2K. Certain hybrid examples will also be provided to show how D2K can be used to move existing research into the D2K framework a section at a time.


Q: My company won't allow me to send out source code, will this affect my submission?

A: No, as long as your code is compiled for a suitable architecture the IMIRSEL lab can run it in its compiled form.


Q: I've proposed one of the evaluation tasks and want to know what details you need to produce the framework software?

A: In order to establish a framework we need common input and output file formats, e.g. Each submission should read in a test file that defines the training set in the following format:

   C:\path\filename1.wav\tClassname1
   C:\path\filename2.wav\tClassname1
   C:\path\filename3.wav\tClassname2
   C:\path\filename4.wav\tClassname2
   etc.

where \t indicates a tab. In this example the test set would be read in the same format (without the class names) and the output format should be the same as the input.

Next we need to establish a calling convention for each part of a submission:

There are four formats for calls to code external to D2K that will be supported:

  • CommandName inputFileNameAndPath outputFileNameAndPath
  • CommandName inputFileNameAndPath (ouput file name created by adding an extension, e.g. ".features")

The second two formats allow an additional file to be passed as a parameter:

  • CommandName inputFileNameAndPath1 inputFileNameAndPath2 outputFileNameAndPath
  • CommandName inputFileNameAndPath1 inputFileNameAndPath2 outputFileNameAndPath (ouput file name created by adding an extension to inputFileNameAndPath1, e.g. ".features")

E.g.

    ExtractFeatures C:\inTrainFiles.txt C:\outTrainFeatures.feat
    ExtractFeatures C:\inTestFiles.txt C:\outTestFeatures.feat
    TrainModel C:\outTrainFeatures.feat
    ApplyModel C:\outTrainFeatures.feat.model C:\outTestFeatures.feat C:\results.txt

(see the Mirex Wiki for more details.)

Finally, a clear definition of the evaluation procedure should be given, where possible using cross-validation of results, reporting both the mean and variance of those results. Facilities will also be provided for statisitical significance testing of the difference in error rates between each submission using McNemar's test, to establish whether differences are "in the noise" or not. The McNemar's tests will be performed on the output text files fom each iteration of the cross-validated experiments.


Q: My algorithm has more parts to it than your framework (e.g. a feature selection stage) and should be called like so:

    ExtractFeatures C:\inTrainFiles.txt C:\outTrainFeatures.feat
    SelectFeatures C:\outTrainFeatures.feat selectedfeatures.info
    ExtractFeatures C:\inTestFiles.txt C:\outTestFeatures.feat selectedfeatures.info
    TrainModel C:\outTrainFeatures.feat selectedfeatures.info

ApplyModel C:\outTrainFeatures.feat.model C:\outTestFeatures.feat C:\results.txt

A: This is no problem as each module uses one of the available calling conventions, you simply need to add another copy of an external integration module to your copy of the framework. The standard layout for this problem might be:

2005 originalframework.gif


And, in the above example, would be changed to: without affecting the final evaluation.

2005 modifiedframework.gif

This flexibility in the both algorithm calling conventions and the overal layout/number of modules in the framework is greater than would likely be achieved using a completely hand-coded framework for each task.

Q: Who will run the submissions?

A: The team at IMIRSEL will run each submission and report the evaluation results, however, facilities are provided in D2K to run across an X-windows or telnet session (in telnet only console output is displayed, no toolkit desktop). This would allow an entrant to launch their own submision, or to debug it if it doesn't run correctly at the IMIRSEL lab. Hopefully, IMIRSEL will be able to continue this service, hosting the dataset for each task year round, to allow researchers to bench mark their systems against the last year's state-of-the-art.