Get started with the transformers package from Hugging Face for sentiment analysis, translation, zero-shot text classification, summarization, and named-entity recognition (English and French)
Transformers are certainly among the hottest deep learning models at the moment.
Originally, transformers were introduced as a novel architecture for language translation [Attention is All You Need, 2017]
And in fact, they are still mainly used for natural language processing. Here are a few examples:
In this article, you will use pre-trained transformer models to perform some of these tasks.
We are going to use the Hugging Face library, which provides an easy, high-level interface to either TensorFlow or Pytorch for transformers.
Hugging Face also maintains a hub, which makes it easy to share and access models and datasets.
So let's get started!
You can run this tutorial on Google Colab by clicking here.
There is no need for a specific background in programming or machine learning, you'll see that Hugging Face makes it very easy for anybody to use pre-trained transformer models.
The transformers package is not installed by default on Google Colab. So let's install it with pip:
!pip install transformers[sentencepiece]
In this article, we will use the high-level pipeline interface, which makes it extremely easy to use pre-trained transformer models.
Basically, we just need to tell the pipeline what we want to do, and possibly to tell it which model to use for this task.
Here we're going to do sentiment analysis in English, so we select the sentiment-analysis task, and the default model:
from transformers import pipeline
classifier = pipeline("sentiment-analysis")
The pipeline is ready, and we can now use it:
classifier(["this is a great tutorial, thank you",
"your content just sucks"])
We sent two sentences through the pipeline. The first one is predicted to be positive and the second one negative with very high confidence.
Sounds good!
Now let's see what happens if we send in french sentences:
classifier(["Ton tuto est vraiment bien",
"il est complètement nul"])
This time, the classification does not work...
Indeed, the second sentence, which means "this tutorial is complete crap", is classified as positive.
That's not a surprise: the default model for the sentiment analysis task has been trained on English text, so it does not understand French.
So what can you do if you want to work with text in another language, say French?
You just need to search the hub for a french classification model.
Several models are available, and I decided to select nlptown/bert-base-multilingual-uncased-sentiment.
We can specify this model as the one to be used when we create our sentiment-analysis pipeline:
multilang_classifier = pipeline("sentiment-analysis",
model="nlptown/bert-base-multilingual-uncased-sentiment")
multilang_classifier(["Ton tuto est vraiment bien",
"il est complètement nul"])
And it worked! The second sentence is properly classified as very negative.
You might be wondering while the confidence for the first sentence is lower. I'm pretty sure that it's because this sentence scores high on '4 stars' as well.
Now let's try with an actual review for a restaurant near my place:
import pprint
sentence="Contente de pouvoir retourner au restaurant... Quelle déception... L accueil peu chaleureux... Un plat du jour plus disponible à 12h45...rien à me proposer à la place... Une pizza pas assez cuite et pour finir une glace pleine de glaçons... Et au gout très fade... Je pensais que les serveuses seraient plus aimable à l idée de retrouver leur clientèle.. Dommage"
pprint.pprint(sentence)
multilang_classifier([sentence])
2 stars! on Google Review, this review has 1 star. Not a bad prediction.
Let's try and do a bit of translation, from English to French.
Again, we search the hub, and we end up with this pipeline:
en_to_fr = pipeline("translation_en_to_fr",
model="Helsinki-NLP/opus-mt-en-fr")
en_to_fr("your tutorial is really good")
This works well. Let's translate in the other direction. For this, we need to change the task and the model:
fr_to_en = pipeline("translation_fr_to_en",
model="Helsinki-NLP/opus-mt-fr-en")
fr_to_en("ton tutoriel est super")
Excellent translation!
Nowadays, very large deep-learning models are trained on very large datasets collected from the internet.
These models know a lot already, so they don't need to learn much more.
Typically, it's possible to fine tune these models to a specific use case like text classification with a very small additional specific dataset. This is called few-shots learning.
And sometimes, we can even do zero-shot learning: specific tasks can be performed without any specific training. This is what we're going to do now.
We search the hub for a french zero-shot classification model, and we create this pipeline:
classifier = pipeline("zero-shot-classification",
model="BaptisteDoyen/camembert-base-xlni")
In the example below, I propose a sequence to classify into categories, and I also specify the categories.
It's important to note that the model has not been trained with these categories, you could change them at will!
sequence = "Colin est en train d'écrire un article au sujet du traitement du langage naturel"
candidate_labels = ["science","politique","education", "news"]
classifier(sequence, candidate_labels)
The predicted probabilities are actually quite sound! This sentence is indeed about science, news, and education. And not at all related to politics.
But if we try this one:
sequence = "Laurent Wauquiez reconduit à la tête de la région Rhône-Alpes-Auvergne à la suite du deuxième tour des élections."
candidate_labels = ["politique", "musique"]
classifier(sequence, candidate_labels)
It works!
Feel free to try other sentences and other categories. You can also change the model if you wish to do zero-shot classification in English or in another language.
Summarizing text is an interesting application of transformers.
Here, we use a model trained on a dataset obtained by scraping https://actu.orange.fr/, again found on the Hugging Face hub:
summarizer = pipeline("summarization",
model="moussaKam/barthez-orangesum-title")
Let's use the first two paragraphs of an article about Covid-19 read in Le Monde:
import pprint
sentence = "La pandémie ne marque pas le pas. Le variant Delta poursuit son essor planétaire au grand dam de pays impatients de retrouver une vie normale. La pandémie a fait près de quatre millions de morts dans le monde depuis que le bureau de l’Organisation mondiale de la santé (OMS) en Chine a fait état de l’apparition de la maladie fin décembre 2019, selon un bilan établi par l’Agence France-Presse (AFP) à partir de sources officielles, lundi à 12 heures. Les Etats-Unis sont le pays le plus touché tant en nombre de morts (603 967) que de cas. Le Brésil, qui compte 513 474 morts, est suivi par l’Inde (396 730), le Mexique (232 564) et le Pérou (191 899), le pays qui déplore le plus de morts par rapport à sa population. Ces chiffres, qui reposent sur les bilans quotidiens des autorités nationales de santé, sont globalement sous-évalués. L’Organisation mondiale de la santé (OMS) estime que le bilan de la pandémie pourrait être deux à trois fois plus élevé que celui officiellement calculé."
pprint.pprint(sentence)
summarizer(sentence, max_length=80)
Terse, but not bad!
Named entity recognition can serve as the basis of many interesting apps!
For example, one could analyse financial reports looking for dates, prices, company names.
Let's see how to do this.
Here, we use a french equivalent of BERT, called CamemBERT, fine-tuned for NER:
ner = pipeline("token-classification", model="Jean-Baptiste/camembert-ner")
nes = ner("Colin est parti à Saint-André acheter de la mozzarella")
pprint.pprint(nes)
We need to do a bit or post-processing to aggregate named entities with the same type.
Here is a simple algorithm to do so (it can certainly be improved!)
cur = None
agg = []
for ne in nes:
entity=ne['entity']
if entity != cur:
if cur is None:
cur = entity
if agg:
print(cur, ner.tokenizer.convert_tokens_to_string(agg))
agg = []
cur = entity
agg.append(ne['word'])
print(cur, ner.tokenizer.convert_tokens_to_string(agg))
We found two named entities:
In this article, you have learned how easy it is to use a pre-trained transformer, using the transformers package from Hugging Face.
This is possible thanks to all the researchers who designed and trained the models, and who shared them on the Hugging Face hub.
If you want to know more about transformers, I encourage you to follow the Hugging Face course. This should take you no more than a day, and probably less if you're already familiar with deep learning.
In the next article, we will see how to fine tune a transformer to a specific task, with transfer learning.
Please let me know what you think in the comments! I’ll try and answer all questions.
And if you liked this article, you can subscribe to my mailing list to be notified of new posts (no more than one mail per week I promise.)
You can join my mailing list for new posts and exclusive content: