Commonly used coreference resolution evaluation metrics can only be applied to key mentions (i.e.already annotated mentions). We here propose two variants of the BCubed (Bagga and Baldwin, 1998) and CEAF (Luo, 2005) coreference resolution evaluation algorithms which can be used to evaluate coreference resolution systems dealing with system mentions (i.e. automatically determined mentions).The algorithms and relevant analysis can be found in details in our SIGDIAL 2010 paper.Both BCubedsys and CEAFsys java classes are available for download, along with a couple of necessary parent classes. They should be used within the BART framework, whose repository is available here.
Click here to download BCubedsys and CEAFsys.
Publications related to evaluation metrics for end-to-end coreference resolution