Sequence models are not the right fit for POS taggers

So I was trying to train a sequence model to predict the POS tag for a word. The idea I was going with was that POS tags in a body of text follow some sort of sequence - much the same way as words do. This sequential nature of Natural Language is a cornerstone of NLP. 

On reflection - and having tried unsuccessfully to train a Sequence model - I have come to the conclusion that this does not work.

Here are some of the things I tried

I changed the labelled training data into a long list of tuples of Word and POS tag. Stuff like ('run', 'VB'). I split this up into a windowed dataset - i.e. I created subsequences of n input tuples and 1 target tuple, n being the sequence length which I arbitrarily set to 20.  

Then I created dictionaries for the various vocabularies - these were not word vocabularies, but tuple vocabularies. So an index number for each of the (word, POS) tuples in the training data. I translated the training data into a set of integer lists in this way. 

I also created a vocabulary for  the POS tags themselves. This was so I could encode the targets which of course were not words. Finally I added the word which we want to find the tag for to the end of the training data list. 

So it all ends up looking something like this:

x = [(word1, POS1), (word2, POS2)...(wordn)]

y = [pos for wordn]

but encoded as integers.

This gave even worse results than the previous (very basic) probabilistic methods. There are probably significant improvements I could make to this approach to get better results, but overall I think this level of complexity should be unnecessary for a problem like POS tagging.

As I mentioned the NLTK POS tagger gives excellent results. I decided to take a look at what they had done. That lead me here. Excellent resource.

So my task for tomorrow is to build a simpler feed forward neural network using the same features as the NLTK tagger and see how I get on.

Some learnings

Google Colaboratory provides free GPU instances for experimentation. For larger models you are likely to run out of memory, but colab significantly outperforms my new macbook pro. Their TPUs are even faster, but can consume all of your free limit very quickly, so best to steer away from TPU unless you are prepared to upgrade for continued service.

I learned quite a bit about Tensorflow Datasets on another Coursera course that I did last year. However having not used it in a while I found it difficult to get back into. A powerful technology, especially for larger datasets - where it managers memory very well - but a steep learning curve. Down which I appear to have slid :( 

Keras is a great library. There is still quite a bit that I don't understand about it, but it seems to be one that is here for the long haul. Well worth the time invested learning it. 


Comments

Post a Comment

Popular posts from this blog

Execute Jupyter notebooks line by line in VS Code

Normal notebook execution is cell by cell in Jupyter notebooks in the browser. This frequently results in cells needing to be split and rejoined at various points in the dev process. VS Code allows line by line execution of code within those cells. This feels a lot like running SQL code in MS SQL Analyzer and can be a very natural way to work. In addition VS Code gives you a nice view of the data returned from a cell or line execution with its Data Viewer. In WSL Ubuntu to get the line by line stuff working you will need to install the Jupyter Extension and the Python extension. Once these are in place the process of getting VS Code to find your correct python environment is very straightforward. You can find more about line by line debugging here: https://github.com/microsoft/vscode-jupyter/wiki/Setting-Up-Run-by-Line-and-Debugging-for-Notebooks Once the python extension is installed VS Code will find your virtual environments and you pyenv environments.
Read more

Using TensorFlow Serving

Having converted the front end to React the next step is to get a model making predictions from the server. TensorFlow Serving seems like the best way to do this. Instructions are here  for using Serving with Docker which is the recommended approach.  Some tips on docker: After you have run docker pull to get the TensorFlow Serving container you will want to list your containers: docker ps To run a container use docker run as per the instructions.  docker run -p 8501:8501 --mount type=bind,source=$(pwd)/data/model/pos_checkpoint,target=/models/pos_checkpoint -e MODEL_NAME=pos_checkpoint -t tensorflow/serving Then use docker stop XX where XX is enough of the tag from docker ps to identify that container.  You will need to put your SavedModel into an integer folder for the version.  The command to run a prediction using the REST endpoint is: curl -d '{"instances":[1,2,3]}' -X POST http://localhost:8501/v1/models/pos_checkpoint:predict That will error because the ...
Read more

Text Summarisation with BERT

Hugging Face provides a wide range of pretrained models for lots of machine learning tasks. Summarising the chapters of our novel - Bleak House - would be an interesting challenge. This could also be a very useful mnemonic technique if you were reviewing a long piece of work.  So here is the Hugging Face instructions on this. pip install transformers Then in python: from transformers import pipeline summarizer = pipeline("summarizer") Give it the text to summarize along with some params summarizer(ARTICLE, max_length=130, min_length=30, do_sample=False) BERT is a large model and takes about 15 minutes to download for me. However it is then a pip package and so will be available next time you try to run it. 
Read more