Synonyms using Glove

If you get a set of pretrained word embeddings like Glove from Stanford you can create a synonym generator very easily.

First use Location Sensitive Hashing to split up the vector space of all words in your vocabulary into a number of partitions. This is as simple as determining the range of each of your embeddings and picking uniform random point between these ends. A set of these random points - one for each of your embedding dimensions - constitutes a hyperplane. You can multiply the word vector of a word by this hyperplane and sum the results. The sign of the result -1, +1 or 0 gives you the key part for that hyperplane. Repeat for all of the other hyperplanes and you can form a tuple of these key parts. Key a dict with these and a list of the associated words and you are good to go. 

So when you want to find a synonym you can find the bucket of words with the same hash key and calculate the Cosine Similarity for each of them. Sort by these similarities and you have your synonyms.

Note that this is approximate nearest neighbours. If you carry out the cosine similarity with all of your word vectors as a (slow) experiment you will see that you get a fuller list of similar words the different elements of which were excluded by the approximate nature of this algorithm.

Initially I tried this out with a Glove vector set with a 400k vocab. I will try with a much larger set ~1.2 MM to see if I get more refined results.

Comments

Post a Comment

Popular posts from this blog

Execute Jupyter notebooks line by line in VS Code

Normal notebook execution is cell by cell in Jupyter notebooks in the browser. This frequently results in cells needing to be split and rejoined at various points in the dev process. VS Code allows line by line execution of code within those cells. This feels a lot like running SQL code in MS SQL Analyzer and can be a very natural way to work. In addition VS Code gives you a nice view of the data returned from a cell or line execution with its Data Viewer. In WSL Ubuntu to get the line by line stuff working you will need to install the Jupyter Extension and the Python extension. Once these are in place the process of getting VS Code to find your correct python environment is very straightforward. You can find more about line by line debugging here: https://github.com/microsoft/vscode-jupyter/wiki/Setting-Up-Run-by-Line-and-Debugging-for-Notebooks Once the python extension is installed VS Code will find your virtual environments and you pyenv environments.
Read more

Using TensorFlow Serving

Having converted the front end to React the next step is to get a model making predictions from the server. TensorFlow Serving seems like the best way to do this. Instructions are here  for using Serving with Docker which is the recommended approach.  Some tips on docker: After you have run docker pull to get the TensorFlow Serving container you will want to list your containers: docker ps To run a container use docker run as per the instructions.  docker run -p 8501:8501 --mount type=bind,source=$(pwd)/data/model/pos_checkpoint,target=/models/pos_checkpoint -e MODEL_NAME=pos_checkpoint -t tensorflow/serving Then use docker stop XX where XX is enough of the tag from docker ps to identify that container.  You will need to put your SavedModel into an integer folder for the version.  The command to run a prediction using the REST endpoint is: curl -d '{"instances":[1,2,3]}' -X POST http://localhost:8501/v1/models/pos_checkpoint:predict That will error because the ...
Read more

Text Summarisation with BERT

Hugging Face provides a wide range of pretrained models for lots of machine learning tasks. Summarising the chapters of our novel - Bleak House - would be an interesting challenge. This could also be a very useful mnemonic technique if you were reviewing a long piece of work.  So here is the Hugging Face instructions on this. pip install transformers Then in python: from transformers import pipeline summarizer = pipeline("summarizer") Give it the text to summarize along with some params summarizer(ARTICLE, max_length=130, min_length=30, do_sample=False) BERT is a large model and takes about 15 minutes to download for me. However it is then a pip package and so will be available next time you try to run it. 
Read more