Pair Hidden Markov Model for Named Entity Matching

This paper introduces a pair-Hidden Markov Model (pair-HMM) for the task of evaluating the similarity between bilingual named entities. The pair-HMM is adapted from Mackay and Kondrak [1] who used it on the task of cognate identification and was later adapted by Wieling et al. [5] for Dutch dialect comparison. When using the pair-HMM for evaluating named entities, we do not consider the phonetic representation step as is the case with most named-entity similarity measurement systems. We instead consider the original orthographic representation of the input data and introduce into the pair-HMM representation for diacritics or accents to accommodate for pronunciation variations in the input data. We have first adapted the pair-HMM on measuring the similarity between named entities from languages (French and English) that use the same writing system (the Roman alphabet) and languages (English and Russian) that use a different writing system. The results are encouraging as we propose to extend the pair-HMM to more application oriented named-entity recognition and generation tasks.