Simple POS tagger with 92% accuracy

My goal here is to build a POS tagger using Keras. I initially did some very poor attempts at the tagger with things like bigrams. Next I tried Sequence models. Could not get them to work. They may not be inherently a bad idea for this, but everything about the code felt like overkill. Also I had yet to get it to work, so that really convinced me to drop that idea. 

My most recent effort is based on using the features described here for building a tagger. 

The encoding of the tags and words had a few gotchas. Although the input features consist of a mixture of words, suffixes, prefixes and tags from other parts of the context, the target is just a POS tag. Leaving the one hot encoding of the target at the overall vocab size - about 47k tokens would have given very poor results. There are less than 50 tags. Needle in haystack stuff. So 2 vocabularies were needed.

I needed to cover the full range of input features in the vocab, so this involved adding tokens for the various word fragments in there too. Not a problem as the training set is very large. ~1MM words.

So features and target appropriately encoded I could build the network. 

An Embedding layer which has an input dimension of the sum of the 2 vocabularies (word bits and pos tags).

A Flatten layer so that the output of the Embedding matches the final Dense layer. That dense has a dimension of the POS vocabulary - about 45 in this case. 

Training on just a fraction of the total data gave accuracy on the val set of about 92%. This is sufficient for my needs, so I won't take the POS tagger any further for now. 

Next up I am going to convert the model to TFLite so that I can consume it in some javascript.

You can find the full code of the project here.

As a rough check that everything was as I expected I ran prediction on the first 1000 examples in the validation set and checked them against ground truth. Printing out the accuracy at every 100 is shown below.



Comments

Post a Comment

Popular posts from this blog

Execute Jupyter notebooks line by line in VS Code

Normal notebook execution is cell by cell in Jupyter notebooks in the browser. This frequently results in cells needing to be split and rejoined at various points in the dev process. VS Code allows line by line execution of code within those cells. This feels a lot like running SQL code in MS SQL Analyzer and can be a very natural way to work. In addition VS Code gives you a nice view of the data returned from a cell or line execution with its Data Viewer. In WSL Ubuntu to get the line by line stuff working you will need to install the Jupyter Extension and the Python extension. Once these are in place the process of getting VS Code to find your correct python environment is very straightforward. You can find more about line by line debugging here: https://github.com/microsoft/vscode-jupyter/wiki/Setting-Up-Run-by-Line-and-Debugging-for-Notebooks Once the python extension is installed VS Code will find your virtual environments and you pyenv environments.
Read more

Using TensorFlow Serving

Having converted the front end to React the next step is to get a model making predictions from the server. TensorFlow Serving seems like the best way to do this. Instructions are here  for using Serving with Docker which is the recommended approach.  Some tips on docker: After you have run docker pull to get the TensorFlow Serving container you will want to list your containers: docker ps To run a container use docker run as per the instructions.  docker run -p 8501:8501 --mount type=bind,source=$(pwd)/data/model/pos_checkpoint,target=/models/pos_checkpoint -e MODEL_NAME=pos_checkpoint -t tensorflow/serving Then use docker stop XX where XX is enough of the tag from docker ps to identify that container.  You will need to put your SavedModel into an integer folder for the version.  The command to run a prediction using the REST endpoint is: curl -d '{"instances":[1,2,3]}' -X POST http://localhost:8501/v1/models/pos_checkpoint:predict That will error because the ...
Read more

Text Summarisation with BERT

Hugging Face provides a wide range of pretrained models for lots of machine learning tasks. Summarising the chapters of our novel - Bleak House - would be an interesting challenge. This could also be a very useful mnemonic technique if you were reviewing a long piece of work.  So here is the Hugging Face instructions on this. pip install transformers Then in python: from transformers import pipeline summarizer = pipeline("summarizer") Give it the text to summarize along with some params summarizer(ARTICLE, max_length=130, min_length=30, do_sample=False) BERT is a large model and takes about 15 minutes to download for me. However it is then a pip package and so will be available next time you try to run it. 
Read more