Comparison

As mentioned before, PROVEE is a user-friendly tool for 2D high scalability embedding projections, but it is not the only tool that aims to visualize embeddings. To build the optimal embedding projector we have looked at existing tools and examined what worked well using these tools. But most importantly: what was missing? Other’s mistakes can be a fruitful sources of information. Mistakes can only imPROVEE our library!

There are several libraries and tools that try to visualize embeddings or parts of embeddings, but only a few have visualization of embeddings as the main purpose. The most important libraries/tools we have looked at to create PROVEE:

• Vec2graph

• Tensorflow Embedding Projector

• Whatlies

• Parallax

Libraries and tools were examined for their scalability, user-friendliness, responsiveness and advantages and disadvantages. We were also interested in their back-end and data storage and transfer.

Vec2graph

Vec2graph is a library by Katricheva et al.(2020) for visualizing word embeddings as graphs. The 2D graph is created based on cosine similarity between points and between neighbours. Edges between the nodes display semantic similarity between the embeddings. Graphs can contain nodes that link to other graphs and are displayed using an .html file. Vec2graph shines in its ease of use as graphs can be displayed using 2 or 3 lines of code, but the projections are not suited to display many data points at once.

Whatlies

Whatlies is a project developed by Rasa. The goal of the project is to create an API that supports many languages, such as SpaCy, Gensim and FastText to display word or sentence embeddings. Whatlies enables users to easily display data in interactive 2D graphs using Jupyter Notebooks. The axes can be defined by dimension reduction methods, such as PCA or UMAP, but also special queries using vector arithmetics. For instance, ‘king - man’ can be the y-axis and ‘woman’ can be the x-axis. The special thing about Whatlies is the support for vector arithmetic on embeddings, which can be visualized directly in the interactive plots (they are inspired by Parallax!). It has a high scalability of the input, but the overall goal is to visualize smaller groups of words. It is not able to display more than 5000 embeddings as well. Furthermore, the tool requires little programming knowledge and has a clear documentation on their Github.io page, including examples on how to use the tool.

Tensorflow Embedding Projector

The Embedding Projector is part of the Tensorboard. It can graphically represent embeddings in a 2D or 3D space. These embeddings can be anything, as long as they can be converted to a tab separated file. The tool has a high scalability and is suited to display many data points at once. It is possible to use the Embedding Projector locally or directly using the web-browser. The user can interactively explore the embedding space, varying many parameters, easily switching from PCA to UMAP, 2D to 3D. A disadvantage of the tool is the limitation of dimensionality reduction methods as preprocessing method to display the embeddings.

Parallax

Last but not least, there is Parallax. Parallax is a tool to display word embedding spaces, suited for many embeddings with high dimensions. The tool is suited to display many data points at once. Most interesting is the idea on vector arithmetics as axes in the article accompanying the tool of Molino et al. (2019). The axes of their tool can be obtained using PCA or t-SNE, but they propose a special approach for the specification of the axes: axes that are the result of vector arithmetics formulas on these embeddings. This results in axes such as the average of two words or the most frequently occuring word in the data set. The major drawback of the tool is the slow loading and reaction time when using >10.000 points. Another difficulty of the tool is the required programming exprience. Some knowledge of programming is required.

PROVEE

Our current project PROVEE tries to overcome all the major drawbacks of the studied tools, by designing a user-friendly, responsive tool, suitable for many embeddings with high dimensionality. The highest priority of our tool is providing a high scalability of the projection: thousands of points with multidimensional embeddings. The embeddings can be anything. Not only word embeddings, but image embeddings, DNA embeddings, sentence embeddigns, you name it. Other priorities are low memory footprint and ease of use: programming experience is not required, which makes the tool easy to use.

Overview

In this section we would like to give an overview of the tools, including characteristics we think of as important in developing PROVEE.

	Vec2graph	Tensorflow Embedding Projector	Whatlies	Parallax	PROVEE
2D interactive graph		✅	✅	✅	✅
PCA, t-SNE, UMAP		✅	✅	✅	✅
Vector Arithmetics axes			✅	✅	✅
Easy to use	✅		✅		✅
Display > 100.000 points		✅			✅
Other than word embeddings		✅			✅
Load multiple data sets					✅

Analytical Process: when to use which tool?

Context Differences

Words can have different meanings in different contexts. As a result, the embeddings of the same words in different contexts are different. It is possible to investigate these differences. PROVEE has the option to load in different data sets. Using the same words on the axes, but from different embedding sets, can display the meanings of similar words in diverse contexts.

Molino et al. (2019) display that Parallax is able to perform a similar task, using the co-occurence matrix. Whatlies and the Embedding Projector are likely to perform the task as well, but no tool is able to load in two data sets separately.

Local Embedding Inspection

Although the high scalability of projection is the main drive of PROVEE, the tool is suited to explore local embedding neighbourhoods. PROVEE displays embeddings in an interactive 2D visualization, meaning that users are able to zoom in or out as they desire. Inspecting the local neighbourhood of one specific embedding is enabled by zooming into the embedding space.

Tools designed especially to display local neighbourhood are Vec2graph and Whatlies. These tools have a low scalability of projection, because they aim to visualize the direct neighbourhood of a point. PROVEE is capable of providing such a local view, but this is not the direct goal of the tool. The Embedding Projector and Parallax are constructed in a similar way: not designed to display local neighbourhoods directly, but it is possible using these tools.

Bias Detection

A full set of embeddings can be displayed in 2D using specific axes to detect bias. The most common example is to put ‘man’ and ‘woman’ on the x- and y-axis for the detection of gender bias. Word embeddings are placed in the embedding space, relating their dimensions to the word ‘man’ and ‘woman’, as gender bias can be found by a the direction in the word embedding Bolukbasi et al. (2016). Words that are biased towards ‘man’ lie below the diagonal of the embedding space; words that are biased towards ‘woman’ lie above this diagonal.

Not only PROVEE is suited for bias detection. One could also use Parallax, Whatlies or the Tensorflow Embedding Projector. PROVEE is able to display more points than Whatlies and display these points faster than Parallax, which makes PROVEE suitable for bias detection in a data set containing many embeddings.

Analogy Task

The analogy task can be seen as the similarity between pairs of words. Already clear from the fact that bias can be detected in word embeddings, Mikolov et al. (2013) showed that embeddings contain linguistic regularities and patterns. These patterns become visible in the example ‘Madrid’ - ‘Spain’ + ‘France’, which is very close to ‘Paris’ (shown by Mikolov et al. (2013)). We can use PROVEE for this analogy task. Placing ‘Madrid’ - ‘Spain’ on one axis and ‘France’ on the other, we can see which embeddings are suited for this analogy. .

Molino et al. (2019) provides an example in the paper of the implementation of the analogy task for using Parallax. As Whatlies supports these vector arithmetics, this tool is suited for the analogy task as well. A similar argument holds: PROVEE is able to display more points faster than Whatlies and Parallax.

Summary

A table summarizing the tasks and the suitable tools. It is possible to think of other tasks as well, such as investigating polysemous embeddings or contextual embeddings using BERT embeddings. We can see that Whatlies and Parallax can perform all tasks, but not in a similar way as PROVEE (less points, not able to load in two data sets etc.).

	Vec2graph	Tensorflow Embedding Projector	Whatlies	Parallax	PROVEE
Context differences	✅	✅	✅	✅	✅
Local embedding inspection	✅	✅	✅	✅	✅
Bias detection		✅	✅	✅	✅
Analogy Task			✅	✅	✅