From language identification to language distance

In this paper, we define two quantitative distances to measure how far apart two languages are. The distance measure that we have identifed as more accurate is based on the perplexity of n-gram models extracted from text corpora. An experiment to compare forty-four European languages has been performed. For this purpose, we computed the distances for all the possible language pairs and built a network whose nodes are languages and edges are distances. The network we have built on the basis of linguistic distances represents the current map of similarities and divergences among the main languages of Europe.

keywords: Language Distance, N-Gram Models, Perplexity, Corpus-Based Linguistics, Natural Language Processing, Language Identifcation