Revising My Transformer Plan
As a result of some unexpected changes in my schedule, I was able to work a little bit on my transformer! Progress has been incredibly slow, but I am proud that I have something to write about. After deciding last week that I was going to write my model using the Hugging Face transformers library, I started this week off by researching how I was going to do that.
First off, I needed to figure out what sort of model I was going to use, as hugging face had a variety of different model types available. After a little bit of Googling, I ended up going with the BERT model due to its previous use with classification problems. Previous transformer models I was looking at revolved around being able to either predict text or translate it into a different version of itself. BERT on the other hand is primarily used for the classification through the use of an appended neural network at the end of the model. This model would take the output from the BERT encoders, and then figure how the different elements of the BERT output corresponded with the labels for the texts. This implementation of a transformer matched my use case, so I decided to go with it.
The main bit of research revolved around model deployment because this model needs to be put into a mobile application; and not just any mobile application, but a PWA running on top of CapictorJS. As such, I needed to make sure that I could port the final model and sentence tokenizer to Javascript so that I could run it in the mobile application. However, after researching for a couple of hours, I realized that I likely wouldn’t be able to use this library for my application. While I could likely export the needed model, I wouldn’t have been able to create a tokenizer in Javascript that I could use to create the word vector inputs needed for the model.
So, going into this week my hope is that I can figure out how to write the model from scratch so that way I can make sure it works in both Javascript and Python.
Adios
