Creating our own word embeddings with Glove

The Stanford Glove website gives details on how to train your own set of embeddings. This is pretty straightforward using the tools they have provided.

I am interested in seeing if I can get embeddings trained for some of the words in our book which are not in even the large corpus Glove models. Bleak House is a Dickens novel, so it has an old fashioned vocabulary in many ways. 

Just using the original text of Bleak House and tokenizing it gives 46 word embeddings that were not captured in the original 1.9MM vocab of Glove. Extending this to cover a much larger range of Dickens novels - any maybe victorian novels in general would probably produce a much better result.

The new words are a strange lot. But maybe that is even more of a reason to be able to provide synonyms.

"'eart",
 "'my",
 "'now",
 "'ouse",
 "'prentices",
 "'t",
 "'this",
 "'what",
 "'you",
 'a-moving',
 'ap-kerrig',
 'berryin',
 'bogsby',
 'borrioboola-gha',
 'borrioboolan',
 'boythorn',
 'chadbands',
 'coavinses',
 'coodle',
 'dedlocks',
 "for'ard",
 'gownd',
 'growlery',
 'grubble',
 "guppy's",
 "guv'ner",
 'inkwhich',
 'law-stationer',
 'law-writer',
 'pardiggle',
 'sangsby',
 'shop-door',
 'squod',
 'swosser',
 'terewth',
 'thavies',
 'tom-all-alone',
 'toughey',
 'turveydrop',
 'unfortnet',
 've-ry',
 'weevle',
 'wiglomeration',
 'wiolinceller',
 "won't",
 "wouldn't"

Comments

Post a Comment

Popular posts from this blog

Execute Jupyter notebooks line by line in VS Code

Normal notebook execution is cell by cell in Jupyter notebooks in the browser. This frequently results in cells needing to be split and rejoined at various points in the dev process. VS Code allows line by line execution of code within those cells. This feels a lot like running SQL code in MS SQL Analyzer and can be a very natural way to work. In addition VS Code gives you a nice view of the data returned from a cell or line execution with its Data Viewer. In WSL Ubuntu to get the line by line stuff working you will need to install the Jupyter Extension and the Python extension. Once these are in place the process of getting VS Code to find your correct python environment is very straightforward. You can find more about line by line debugging here: https://github.com/microsoft/vscode-jupyter/wiki/Setting-Up-Run-by-Line-and-Debugging-for-Notebooks Once the python extension is installed VS Code will find your virtual environments and you pyenv environments.
Read more

Using TensorFlow Serving

Having converted the front end to React the next step is to get a model making predictions from the server. TensorFlow Serving seems like the best way to do this. Instructions are here  for using Serving with Docker which is the recommended approach.  Some tips on docker: After you have run docker pull to get the TensorFlow Serving container you will want to list your containers: docker ps To run a container use docker run as per the instructions.  docker run -p 8501:8501 --mount type=bind,source=$(pwd)/data/model/pos_checkpoint,target=/models/pos_checkpoint -e MODEL_NAME=pos_checkpoint -t tensorflow/serving Then use docker stop XX where XX is enough of the tag from docker ps to identify that container.  You will need to put your SavedModel into an integer folder for the version.  The command to run a prediction using the REST endpoint is: curl -d '{"instances":[1,2,3]}' -X POST http://localhost:8501/v1/models/pos_checkpoint:predict That will error because the ...
Read more

Text Summarisation with BERT

Hugging Face provides a wide range of pretrained models for lots of machine learning tasks. Summarising the chapters of our novel - Bleak House - would be an interesting challenge. This could also be a very useful mnemonic technique if you were reviewing a long piece of work.  So here is the Hugging Face instructions on this. pip install transformers Then in python: from transformers import pipeline summarizer = pipeline("summarizer") Give it the text to summarize along with some params summarizer(ARTICLE, max_length=130, min_length=30, do_sample=False) BERT is a large model and takes about 15 minutes to download for me. However it is then a pip package and so will be available next time you try to run it. 
Read more