org.imirsel.m2k.evaluation
Class MelodySimilarityEvaluator1

java.lang.Object
  java.util.Observable
      ncsa.d2k.core.modules.RootModule
          ncsa.d2k.core.modules.ConfigurableModule
              ncsa.d2k.core.modules.EmbeddedPipeModule
                  ncsa.d2k.core.modules.ExecModule
                      ncsa.d2k.core.modules.ComputeModule
                          org.imirsel.m2k.evaluation.MelodySimilarityEvaluator1

All Implemented Interfaces:

java.lang.Cloneable, ncsa.d2k.core.modules.Module, java.io.Serializable, ncsa.d2k.core.modules.SystemModule

public class MelodySimilarityEvaluator1

extends ncsa.d2k.core.modules.ComputeModule

This module evaluates symbolic melody similarity algorithms based on ground truth (evaluation 1). Input is a file containing ground truth and a file containing test result. The measures returned are: 1. Precisions at group boundaries 2. Average precision (non-interpolated) over all relevance docs This is done by computing the precision after every retrieved relevant document and then averaging these precisions over the total number of retrieved relevant documents for that topic. These topic averages are then combined (averaged) across all topics in the appropriate set to create the non-interpolated average precision for that set. 3. Interpolated Recall - Precision (precisions at different recall levels) 4. Precision at different document level 5. R-Precision (precision after R (= num_rel for a query, 50 in this evaluation) docs retrieved. 6. For evaluation2: an average of the precision for each topic after 100 documents have been retrieved for that topic. This measure is useful because it reflects a clearly comprehended retrieval point. It took on added importance in the TREC environment because only the top 100 documents retrieved for each topic were actually assessed. For this reason it produces a guaranteed evaluation point for each system. Finally, evaluation result will be output to a text file, specified as a parameter.

Author:

Xiao

See Also:

Serialized Form

Field Summary

Fields inherited from class ncsa.d2k.core.modules.ConfigurableModule

addedInputInfo, addedInputNames, addedInputTypes, addedOutputInfo, addedOutputNames, addedOutputTypes, addPortListeners

Fields inherited from class ncsa.d2k.core.modules.RootModule

alias, children, DEBUG_LEVEL, EMPTY_INPUT, ERROR_LEVEL, executionManager, FATAL_LEVEL, HAVE_INPUT, INFO_LEVEL, iPipes, iPipesIndex, iTriggers, oPipes, oPipesIndex, oTriggers, parents, pipesFullManager, stats, triggerChildren, triggerParents, WARN_LEVEL

Fields inherited from interface ncsa.d2k.core.modules.SystemModule

BLOCKED_STATE, IDLE_STATE, MEDIUM, MEDIUM_RARE, MEDIUM_WELL, RARE, READY_STATE, WELL, WORKING_STATE

Constructor Summary

MelodySimilarityEvaluator1()
Creates a new instance of MelodySimilarityEvaluator1

Method Summary

void	beginExecution() Performs operations at the beginning of itinerary execution.
protected double[]	compADR() compute ADR (average dynmaic recall)
protected double[]	compAvgPrec() Calculate non-interpolated average precision for each query
protected double[][]	compDocPrec() calculate Precision at different document levels (each query has 50 ground truth), and thus, we calculate precisions at 5, 10, 15, 20, 30, 40 50, 60,80,100 retrieved documents
protected double[]	compNormRec(java.util.Vector[] presAtGroupBoundary) compute normalized recall for all queries of each run
protected java.util.Vector[]	compPrecAtGroupBdr() Calculate precision at group boundaries
protected double[][]	compRecPrec() Calculate Interpolated Recall - Precision (precisions at different recall levels)
protected int	compRelRet() Calculate the total number relevant retrieved documents across all topics
void	doit() This module takes as input two File Objects.
void	endExecution() Performs operations at the end of itinerary execution.
java.lang.String	getInputInfo(int i) Returns a text description for the indicated input.
java.lang.String	getInputName(int i) Returns a text name for the given input.
java.lang.String[]	getInputTypes() Returns an array of strings containing the Java data types of the input.
java.lang.String	getModuleInfo() Returns information about the module.
java.lang.String	getModuleName() Returns the name of the module.
java.lang.String	getOutputDirectory() Returns the output directory
java.lang.String	getOutputInfo(int i) Returns a text description for the given output.
java.lang.String	getOutputName(int i) Returns a text name for the given output.
java.lang.String[]	getOutputTypes() Returns an array of strings containing the Java data types of the outputs.
ncsa.d2k.core.modules.PropertyDescription[]	getPropertiesDescriptions() Returns an array of description objects for each property of the Module.
boolean	getVerbose() Returns the value of the verbose output flag
boolean	isReady() Returns a boolean indicating if the module is ready to execute or not.
protected int	readInAlgResult(java.io.File arFile) Read in algorithm results
protected int	readInGroundTruth(java.io.File gtFile) Read in Ground Truth
void	setOutputDirectory(java.lang.String value) Sets the output directory
void	setVerbose(boolean value) Sets the value of the verbose output flag

Methods inherited from class ncsa.d2k.core.modules.ExecModule

execute, setExecutionManager

Methods inherited from class ncsa.d2k.core.modules.ConfigurableModule

addAddPortListener, addInput, addInputTrigger, addOutput, addOutputTrigger, getAddedInputTypes, getAddedOutputTypes, getAddPortListeners, insertInput, insertOutput, removeAddPortListener, removeInput, removeInputTrigger, removeOutput, removeOutputTrigger, setAddedInputTypes, setInputType, setOutputType

Methods inherited from class ncsa.d2k.core.modules.RootModule

activateTriggers, begin, canRun, clone, debug, debug, disconnectInputPipe, disconnectInputTriggers, disconnectOutputPipe, disconnectOutputTriggers, end, error, error, fatal, fatal, fetchInputs, getAlias, getChildIndex, getChildMux, getChildren, getExecutionManager, getFile, getFlags, getFullPipeManager, getGuiComponent, getImage, getInputPipes, getInputPipeSize, getInputTriggers, getModuleStatistics, getNumInputs, getNumOutputs, getOutputCounts, getOutputPipes, getOutputPipeSize, getOutputTriggers, getParentIndex, getParentMux, getParents, getPipesFull, getPriority, getPropertyEditor, getResource, getRootName, getState, info, info, initModule, isAborting, isHead, isInputPipeConnected, isOutputPipeConnected, pullInput, pushOutput, resetInputs, setAlias, setBlocked, setD2KModulesLoggingLevel, setFlags, setFullPipeManager, setIdle, setInputPipe, setLogLevel, setModuleStatistics, setModuleStatisticsByMachine, setOutputCounts, setOutputPipe, setPipesFull, setReady, setResource, setState, setThePriority, setWorking, trigger, triggersActivated, warn, warn

Methods inherited from class java.util.Observable

addObserver, clearChanged, countObservers, deleteObserver, deleteObservers, hasChanged, notifyObservers, notifyObservers, setChanged

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MelodySimilarityEvaluator1

public MelodySimilarityEvaluator1()

Creates a new instance of MelodySimilarityEvaluator1

Method Detail

setVerbose

public void setVerbose(boolean value)

Sets the value of the verbose output flag

Parameters:

value - the value which verbose is set to

See Also:

#verbose

getVerbose

public boolean getVerbose()

Returns the value of the verbose output flag

Returns:

verbose

See Also:

#verbose

setOutputDirectory

public void setOutputDirectory(java.lang.String value)

Sets the output directory

Parameters:

value - string representation of the output directory

See Also:

#outputDirectory

getOutputDirectory

public java.lang.String getOutputDirectory()

Returns the output directory

Returns:

the output directory

See Also:

#outputDirectory

getPropertiesDescriptions

public ncsa.d2k.core.modules.PropertyDescription[] getPropertiesDescriptions()

Returns an array of description objects for each property of the Module.

Returns:

an array of description objects for each property of the Module.

getModuleName

public java.lang.String getModuleName()

Returns the name of the module.

Returns:

the module name

getModuleInfo

public java.lang.String getModuleInfo()

Returns information about the module.

Returns:

Module information

getInputName

public java.lang.String getInputName(int i)

Returns a text name for the given input.

Parameters:

i - the index of the input

Returns:

the name of the indexed input

getInputInfo

public java.lang.String getInputInfo(int i)

Returns a text description for the indicated input.

Parameters:

i - the index of the input

Returns:

a text description of the indexed input

getInputTypes

public java.lang.String[] getInputTypes()

Returns an array of strings containing the Java data types of the input.

Returns:

the fully qualified java types for each of the inputs

getOutputName

public java.lang.String getOutputName(int i)

Returns a text name for the given output.

Parameters:

i - the index of the output

Returns:

the name of the indexed output

getOutputInfo

public java.lang.String getOutputInfo(int i)

Returns a text description for the given output.

Parameters:

i - the index of the output

Returns:

the name of the indexed output

getOutputTypes

public java.lang.String[] getOutputTypes()

Returns an array of strings containing the Java data types of the outputs.

Returns:

the fully qualified java types for each of the outputs.

beginExecution

public void beginExecution()

Performs operations at the beginning of itinerary execution.

See Also:

endExecution()

endExecution

public void endExecution()

Performs operations at the end of itinerary execution.

See Also:

beginExecution()

isReady

public boolean isReady()

Returns a boolean indicating if the module is ready to execute or not.

Returns:

true if the module is ready to execute, false if not.

doit

public void doit()
          throws java.lang.Exception

This module takes as input two File Objects. It checks to see if the input are null objects. If one input is a null, it passes a null to the output, and exits the method through a return statement. If the input object are not nulls, check if the two Files contain enough queries (specified as parameter), if not, report error; if yes, caculate evaluation measures. Finally, output. The input data is not modified.

Throws:

java.lang.Exception

readInGroundTruth

protected int readInGroundTruth(java.io.File gtFile)

Read in Ground Truth

Parameters:

gtFile - File handle of ground truth file

Returns:

total number of relevant docs

readInAlgResult

protected int readInAlgResult(java.io.File arFile)

Read in algorithm results

Parameters:

arFile - File handle of algorithm result file

Returns:

total number of retrieved docs

compPrecAtGroupBdr

protected java.util.Vector[] compPrecAtGroupBdr()
                                         throws java.lang.Exception

Calculate precision at group boundaries

Returns:

Vector array of calculated precisions

Throws:

java.lang.Exception - if algorithm result doesn't contain all queries in groundTruth

compAvgPrec

protected double[] compAvgPrec()
                        throws java.lang.Exception

Calculate non-interpolated average precision for each query

Returns:

double array of calulated precision

Throws:

java.lang.Exception - exception if algorithm result doesn't contain all queries in groundTruth

compRecPrec

protected double[][] compRecPrec()
                          throws java.lang.Exception

Calculate Interpolated Recall - Precision (precisions at different recall levels)

Returns:

recall-precision for all queries

Throws:

java.lang.Exception - exception if algorithm result doesn't contain all queries in groundTruth

compDocPrec

protected double[][] compDocPrec()
                          throws java.lang.Exception

calculate Precision at different document levels (each query has 50 ground truth), and thus, we calculate precisions at 5, 10, 15, 20, 30, 40 50, 60,80,100 retrieved documents

Returns:

precisions at different document levels for all queries

Throws:

java.lang.Exception - exception if algorithm result doesn't contain all queries in groundTruth

compRelRet

protected int compRelRet()
                  throws java.lang.Exception

Calculate the total number relevant retrieved documents across all topics

Returns:

the total number relevant retrieved documents

Throws:

java.lang.Exception - exception if algorithm result doesn't contain all queries in groundTruth

compNormRec

protected double[] compNormRec(java.util.Vector[] presAtGroupBoundary)

compute normalized recall for all queries of each run

Parameters:

presAtGroupBoundary - recalls at group boundaries

Returns:

normalized recall

compADR

protected double[] compADR()

compute ADR (average dynmaic recall)

Returns:

ADR for all queries

Throws:

java.lang.Exception - if algorithm result doesn't contain all queries in groundtruth