Diachronic Explorer

This is a system that allows users to search for and view the lexical changes of tens of thousands of Spanish words over time, specifically across the 1900–2009 timeline, using as its data source the semantic representations built from Google's Spanish n-grams (45 billion). The user searches for a word and a time period (between 1 and N years), and the system returns the meaning of the word for each year in the selected interval. A word's meaning is represented by the set of words that are most similar to it in semantic and distributional terms. For example, the word “cáncer“ was closely associated with “tuberculosis“ and “sífilis“ in 1910, but by 1960 the closest terms were “tumor“ and “carcinoma“.

The system's input is a data structure in which words are associated with other words for each year by similarity scores (cosine). These data were recently generated by the ProLNat@GE team (Pablo Gamallo, Marcos Garcia) using Natural Language Processing techniques and modules. Specifically, we performed semantic processing on 45 billion n-grams, made available after the digitization of more than 1 million books from the “Google Books“ project. Semantic processing consists of transforming the n-grams into 'word-context' distributional matrices. One matrix was generated per year, where each word is a vector of contexts. Finally, the similarity between vectors (words) is calculated, and the 20 most similar words are selected for each word and year. In total, a data structure of more than more than 300M was generated; this is the input to the demonstrator.

The diachronic explorer can be used in two ways: by accessing the website or through its API.

Installation

Prerequisites

It should be clarified that the diachronic explorer retrieves word-similarity data from a MongoDB database. Therefore, you need an active and accessible Mongo server as a data source.

In addition, the explorer assumes that the clouds and cache folders, as well as the tasks and log files, exist. Their path is passed as an argument when running the program.

Dependencies

Secondly, the explorer relies on several libraries to provide some of its functionality. To make installation easier, a requirements.txt file in the appropriate format is included in the project root so that the libraries can be imported directly using the pip command.

pip install -r requirements.txt

Deployment

The explorer is built in two distinct parts: the server component, which queries the data source, processes the data, and formats it; and the web interface, which simply queries the data provided by the server.

To start the server, go to the services folder and run the main.py file (passing as an argument the path to the required files). This will start the CherryPy server and connect to a MongoDB database using the default parameters. The server will then be running at localhost:8080/, so you can make queries by passing parameters in the URL, for example: http://localhost:8080/busca/simple/fumar/1905/1910

cd services python main.py /ruta/archivos/proyecto

To run the web interface, you need a server with PHP enabled. Once you have one, simply access index.php to use the query system.