Tokenization

Before you feed your text into BERT, you need to turn it into numbers. That’s the role of a tokenizer. Some tokenizers split text on spaces, so that each token corresponds to a word. That would result however in a huge vocabulary, which makes training a model more difficult, so instead BERT relies on sub-word tokenization. Let’s see how it works in code.

Each pre-trained model comes with a pre-trained tokenizer (we can’t separate them), so we need to download it as well. Let’s use it then to tokenize a line of text and see the output.

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'I like to practice kungfu.'
tokens = tokenizer.encode(text)
print(tokens)

[101, 1045, 2066, 2000, 3218, 18577, 11263, 1012, 102]

Each token is a number that corresponds to a word (or subword) in the vocabulary. The most frequent words are represented as a whole word, while less frequent words are divided in sub-words. That ensures that we can map the entire corpus to a fixed size vocabulary without unknown tokens (in reality, they may still come up in rare cases). Let’s see the length of our model’s vocabulary, and how the tokens corresponds to words.

print(f'Length of BERT base vocabulary: {len(tokenizer.vocab)}')
print(f'Text: {text}')
for t in tokens:
  print(f'Token: {t}, subword: {tokenizer.decode([t])}')

Length of BERT base vocabulary: 30522
Text: I like to practice kungfu.
Token: 101, subword: [CLS]
Token: 1045, subword: i
Token: 2066, subword: like
Token: 2000, subword: to
Token: 3218, subword: practice
Token: 18577, subword: kung
Token: 11263, subword: ##fu
Token: 1012, subword: .
Token: 102, subword: [SEP]

In the example, you can see how the tokenizer split a less common word ‘kungfu’ into 2 subwords: ‘kung’ and ‘##fu’. The ‘##’ characters inform us that this subword occurs in the middle of a word. BERT tokenizer also added 2 special tokens for us, that are expected by the model: [CLS] which comes at the beginning of every sequence, and [SEP] that comes at the end. [SEP] may optionally also be used to separate two sequences, for example between question and context in a question answering scenario. Another example of a special token is [PAD], we need to use it to pad shorter sequences in a batch, because BERT expects each example in a batch to have the same amount of tokens.

Outputs

Let’s download a pretrained model now, run our text through it, and see what comes out. We will first need to convert the tokens into tensors, and add the batch size dimension (here, we will work with batch size 1).

model = BertModel.from_pretrained('bert-base-uncased')

inputs = torch.tensor(tokens).unsqueeze(0) # Batch size 1
outputs = model(inputs)
print(f'output type: {type(outputs)}, output length: {len(outputs)}')
print(f'first item shape: {outputs[0].shape}')
print(f'second item shape: {outputs[1].shape}')

output type: <class 'tuple'>, output length: 2
first item shape: torch.Size([1, 9, 768])
second item shape: torch.Size([1, 768])

In the examples above, we used BERT to handle some useful tasks, such as text classification, named entity recognition, or question answering. For each of those tasks, a task-specific model head was added on top of raw model outputs. Here, we are dealing with the raw model outputs - we need to understand them to be able to add custom heads to solve our own, specific tasks.

The model outputs a tuple. The first item of the tuple has the following shape: 1 (batch size) x 9 (sequence length) x 768 (the number of hidden units). This is called the sequence output, and it provides the representation of each token in the context of other tokens in the sequence. If we’d like to fine-tune our model for named entity recognition, we will use this output and expect the 768 numbers representing each token in a sequence to inform us if the token corresponds to a named entity.

The second item in the tuple has the shape: 1 (batch size) x 768 (the number of hidden units). It is called the pooled output, and in theory it should represent the entire sequence. It corresponds to the first token in a sequence (the [CLS] token). We can use it in a text classification task - for example when we fine-tune the model for sentiment classification, we’d expect the 768 hidden units of the pooled output to capture the sentiment of the text.

In practice, we may want to use some other way to capture the meaning of the sequence, for example by averaging the sequence output, or even concatenating the hidden states from lower levels.

Pre-training

In order for a model to solve an NLP task, like sentiment classification, it needs to understand a lot about language. Most of the labelled datasets that we have available are too small to teach our model enough about language. Ideally, we’d like to use all the text we have available, for example all books and the internet. Because it’s hard to label so much text, we create ‘fake tasks’ that will help us achieve our goal without manual labelling.

BERT is trained on a very large corpus using two ‘fake tasks’: masked language modeling (MLM) and next sentence prediction (NSP). In MLM, we randomly hide some tokens in a sequence, and ask the model to predict which tokens are missing. In NSP, we provide our model with two sentences, and ask it to predict if the second sentence follows the first one in our corpus. The intent of these tasks is for our model to be able to represent the meaning of both individual words, and the entire sentences.

nlp = pipeline("fill-mask")
preds = nlp(f"I am exhausted, it's been a very {nlp.tokenizer.mask_token} day.")
print('I am exhausted, it\'s been a very ***** day.')
for p in preds: print(nlp.tokenizer.decode([p['token']]))
preds = nlp(f"I am excited, it's been a very {nlp.tokenizer.mask_token} day.")
print('I am excited, it\'s been a very ***** day.')
for p in preds: print(nlp.tokenizer.decode([p['token']]))

I am exhausted, it's been a very ***** day.
 busy
 exhausting
 stressful
 taxing
 rough
I am excited, it's been a very ***** day.
 busy
 exciting
 productive
 good
 nice

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')

first_sentence = "I cut my finger."
second_sentence_right = "The blood started flowing."
second_sentence_wrong = "This website uses cookies."

right = tokenizer.encode_plus(first_sentence, text_pair=second_sentence_right)
wrong = tokenizer.encode_plus(first_sentence, text_pair=second_sentence_wrong)

r1, r2, r3 = torch.tensor(right['input_ids']).unsqueeze(0), torch.tensor(right['token_type_ids']).unsqueeze(0), torch.tensor(right['attention_mask']).unsqueeze(0)
w1, w2, w3 = torch.tensor(wrong['input_ids']).unsqueeze(0), torch.tensor(wrong['token_type_ids']).unsqueeze(0), torch.tensor(wrong['attention_mask']).unsqueeze(0)

right_outputs = model(input_ids=r1, token_type_ids=r2, attention_mask=r3)
right_seq_relationship_scores = right_outputs[0]
wrong_outputs = model(input_ids=w1, token_type_ids=w2, attention_mask=w3)
wrong_seq_relationship_scores = wrong_outputs[0]

print(first_sentence + ' ' + second_sentence_right)
print(f'Next sentence prediction: {right_seq_relationship_scores.detach().numpy().flatten()[0] > 0}')
print(first_sentence + ' ' + second_sentence_wrong)
print(f'Next sentence prediction: {wrong_seq_relationship_scores.detach().numpy().flatten()[0] > 0}')

I cut my finger. The blood started flowing.
Next sentence prediction: True
I cut my finger. This website uses cookies.
Next sentence prediction: False

Finetuning

As we can see from the examples above, BERT has learned quite a lot about language during pretraining. That knowledge is represented in its outputs - the hidden units corresponding to tokens in a sequence. We can use that knowledge by adding our own, custom layers on top of BERT outputs, and further training (finetuning) it on our own data.