Easy access of language models and ULMFiT

NLP model creation and training

The main thing here is RNNLearner. There are also some utility functions to help create and update text models.

Quickly get a learner

language_model_learner[source][test]

language_model_learner(data:DataBunch, arch, config:dict=None, drop_mult:float=1.0, pretrained:bool=True, pretrained_fnames:OptStrTuple=None, **learn_kwargs) → LanguageLearner

Tests found for language_model_learner:

  • pytest -sv tests/test_text_train.py::test_qrnn_works_if_split_fn_provided [source]
  • pytest -sv tests/test_text_train.py::test_qrnn_works_with_no_split [source]

To run tests please refer to this guide.

Create a Learner with a language model from data and arch.

The model used is given by arch and config. It can be:

They each have a default config for language modelling that is in {lower_case_class_name}\_lm\_config if you want to change the default parameter. At this stage, only the AWD LSTM and Tranformer support pretrained=True but we hope to add more pretrained models soon. drop_mult is applied to all the dropouts weights of the config, learn_kwargs are passed to the Learner initialization.

If your data is backward, the pretrained model downloaded will also be a backward one (only available for AWD_LSTM).

path = untar_data(URLs.IMDB_SAMPLE)
data = TextLMDataBunch.from_csv(path, 'texts.csv')
learn = language_model_learner(data, AWD_LSTM, drop_mult=0.5)

text_classifier_learner[source][test]

text_classifier_learner(data:DataBunch, arch:Callable, bptt:int=70, max_len:int=1400, config:dict=None, pretrained:bool=True, drop_mult:float=1.0, lin_ftrs:Collection[int]=None, ps:Collection[float]=None, **learn_kwargs) → TextClassifierLearner

Tests found for text_classifier_learner:

  • pytest -sv tests/test_text_train.py::test_classifier [source]
  • pytest -sv tests/test_text_train.py::test_order_preds [source]

To run tests please refer to this guide.

Create a Learner with a text classifier from data and arch.

Here again, the backbone of the model is determined by arch and config. The input texts are fed into that model by bunch of bptt and only the last max_len activations are considered. This gives us the backbone of our model. The head then consists of:

  • a layer that concatenates the final outputs of the RNN with the maximum and average of all the intermediate outputs (on the sequence length dimension),
  • blocks of (nn.BatchNorm1d, nn.Dropout, nn.Linear, nn.ReLU) layers.

The blocks are defined by the lin_ftrs and drops arguments. Specifically, the first block will have a number of inputs inferred from the backbone arch and the last one will have a number of outputs equal to data.c (which contains the number of classes of the data) and the intermediate blocks have a number of inputs/outputs determined by lin_ftrs (of course a block has a number of inputs equal to the number of outputs of the previous block). The dropouts all have the same value ps if you pass a float, or the corresponding values if you pass a list. Default is to have an intermediate hidden size of 50 (which makes two blocks model_activation -> 50 -> n_classes) with a dropout of 0.1.

path = untar_data(URLs.IMDB_SAMPLE)
data = TextClasDataBunch.from_csv(path, 'texts.csv')
learn = text_classifier_learner(data, AWD_LSTM, drop_mult=0.5)

class RNNLearner[source][test]

RNNLearner(data:DataBunch, model:Module, split_func:OptSplitFunc=None, clip:float=None, alpha:float=2.0, beta:float=1.0, metrics=None, **learn_kwargs) :: Learner

No tests found for RNNLearner. To contribute a test please refer to this guide and this discussion.

Basic class for a Learner in NLP.

Handles the whole creation from data and a model with a text data using a certain bptt. The split_func is used to properly split the model in different groups for gradual unfreezing and differential learning rates. Gradient clipping of clip is optionally applied. alpha and beta are all passed to create an instance of RNNTrainer. Can be used for a language model or an RNN classifier. It also handles the conversion of weights from a pretrained model as well as saving or loading the encoder.

get_preds[source][test]

get_preds(ds_type:DatasetType=<DatasetType.Valid: 2>, activ:Module=None, with_loss:bool=False, n_batch:Optional[int]=None, pbar:Union[MasterBar, ProgressBar, NoneType]=None, ordered:bool=True) → List[Tensor]

No tests found for get_preds. To contribute a test please refer to this guide and this discussion.

Return predictions and targets on the valid, train, or test set, depending on ds_type.

If ordered=True, returns the predictions in the order of the dataset, otherwise they will be ordered by the sampler (from the longest text to the shortest). The other arguments are passed to Learner.get_preds.

class TextClassificationInterpretation[source][test]

TextClassificationInterpretation(learn:Learner, preds:Tensor, y_true:Tensor, losses:Tensor, ds_type:DatasetType=<DatasetType.Valid: 2>) :: ClassificationInterpretation

No tests found for TextClassificationInterpretation. To contribute a test please refer to this guide and this discussion.

Provides an interpretation of classification based on input sensitivity. This was designed for AWD-LSTM only for the moment, because Transformer already has its own attentional model.

The darker the word-shading in the below example, the more it contributes to the classification. Results here are without any fitting. After fitting to acceptable accuracy, this class can show you what is being used to produce the classification of a particular case.

import matplotlib.cm as cm

txt_ci = TextClassificationInterpretation.from_learner(learn)
test_text = "Zombiegeddon was perhaps the GREATEST movie i have ever seen!"
txt_ci.show_intrinsic_attention(test_text,cmap=cm.Purples)
xxbos xxmaj xxunk was perhaps the xxup greatest movie i have ever seen !

You can also view the raw attention values with .intrinsic_attention(text)

txt_ci.intrinsic_attention(test_text)[1]

tensor([0.6078, 0.4961, 0.4707, 0.4946, 0.5228, 0.5393, 0.5656, 0.6153, 0.6893,
        0.8047, 0.9329, 1.0000, 0.9080, 0.5786], device='cuda:0')

Create a tabulation showing the first k texts in top_losses along with their prediction, actual,loss, and probability of actual class. max_len is the maximum number of tokens displayed. If max_len=None, it will display all tokens.

txt_ci.show_top_losses(5)
Text Prediction Actual Loss Probability
xxbos i have to agree with what many of the other reviewers concluded . a subject which could have been thought - provoking and shed light on a reversed double - standard , failed miserably . \n \n xxmaj rape being a crime of violence and forced abusive control , the scenes here were for the most part pathetic . xxmaj it would have been a better idea to pos neg 8.25 0.00
xxbos xxmaj betty xxmaj sizemore ( xxmaj renee xxmaj zellweger ) lives her life through soap xxmaj opera " a xxmaj reason to xxmaj love " as a way to escape her slob husband and dull life . xxmaj after a shocking incident involving two hit men ( xxmaj morgan xxmaj freeman and xxmaj chris xxmaj rock ) , xxmaj betty goes into shock and travels to xxup la , pos pos 7.71 1.00
xxbos xxmaj when people harp on about how " they do n't make 'em like they used to " then just point them towards this fantastically entertaining , and quaint - looking , comedy horror from writer - director xxmaj glenn mcquaid . \n \n xxmaj it 's a tale of graverobbers ( played by xxmaj dominic xxmaj monaghan and xxmaj larry xxmaj fessenden ) who end up digging pos pos 7.47 1.00
xxbos i have to agree with all the previous xxunk -- this is simply the best of all frothy comedies , with xxmaj bardot as sexy as xxmaj marilyn xxmaj monroe ever was , and definitely with a prettier face ( maybe there 's less mystique , but look how xxmaj marilyn paid for that . ) i do n't think i 've ever seen such a succulent - looking pos pos 6.55 1.00
xxbos i will freely admit that i have n't seen the original movie , but i 've read the play , so i 've some background with the " original . " xxmaj if you shuck off the fact that this is a remake of an old classic , this movie is smart , witty , fresh , and hilarious . xxmaj yes , the casting decisions may seem strange pos pos 6.38 1.00

Loading and saving

load_encoder[source][test]

load_encoder(name:str, device:device=None)

No tests found for load_encoder. To contribute a test please refer to this guide and this discussion.

Load the encoder name from the model directory.

save_encoder[source][test]

save_encoder(name:str)

No tests found for save_encoder. To contribute a test please refer to this guide and this discussion.

Save the encoder to name inside the model directory.

load_pretrained[source][test]

load_pretrained(wgts_fname:str, itos_fname:str, strict:bool=True)

No tests found for load_pretrained. To contribute a test please refer to this guide and this discussion.

Load a pretrained model and adapts it to the data vocabulary.

Opens the weights in the wgts_fname of self.model_dir and the dictionary in itos_fname then adapts the pretrained weights to the vocabulary of the data. The two files should be in the models directory of the learner.path.

Utility functions

convert_weights[source][test]

convert_weights(wgts:Weights, stoi_wgts:Dict[str, int], itos_new:StrList) → Weights

No tests found for convert_weights. To contribute a test please refer to this guide and this discussion.

Convert the model wgts to go with a new vocabulary.

Uses the dictionary stoi_wgts (mapping of word to id) of the weights to map them to a new dictionary itos_new (mapping id to word).

Get predictions

class LanguageLearner[source][test]

LanguageLearner(data:DataBunch, model:Module, split_func:OptSplitFunc=None, clip:float=None, alpha:float=2.0, beta:float=1.0, metrics=None, **learn_kwargs) :: RNNLearner

No tests found for LanguageLearner. To contribute a test please refer to this guide and this discussion.

Subclass of RNNLearner for predictions.

predict[source][test]

predict(text:str, n_words:int=1, no_unk:bool=True, temperature:float=1.0, min_p:float=None, sep:str=' ', decoder='decode_spec_tokens')

No tests found for predict. To contribute a test please refer to this guide and this discussion.

Return text and the n_words that come after

If no_unk=True the unknown token is never picked. Words are taken randomly with the distribution of probabilities returned by the model. If min_p is not None, that value is the minimum probability to be considered in the pool of words. Lowering temperature will make the texts less randomized.

beam_search(text:str, n_words:int, no_unk:bool=True, top_k:int=10, beam_sz:int=1000, temperature:float=1.0, sep:str=' ', decoder='decode_spec_tokens')

No tests found for beam_search. To contribute a test please refer to this guide and this discussion.

Return the n_words that come after text using beam search.

Basic functions to get a model

get_language_model[source][test]

get_language_model(arch:Callable, vocab_sz:int, config:dict=None, drop_mult:float=1.0)

No tests found for get_language_model. To contribute a test please refer to this guide and this discussion.

Create a language model from arch and its config, maybe pretrained.

get_text_classifier[source][test]

get_text_classifier(arch:Callable, vocab_sz:int, n_class:int, bptt:int=70, max_len:int=1400, config:dict=None, drop_mult:float=1.0, lin_ftrs:Collection[int]=None, ps:Collection[float]=None, pad_idx:int=1) → Module

No tests found for get_text_classifier. To contribute a test please refer to this guide and this discussion.

Create a text classifier from arch and its config, maybe pretrained.

This model uses an encoder taken from the arch on config. This encoder is fed the sequence by successive bits of size bptt and we only keep the last max_seq outputs for the pooling layers.

The decoder use a concatenation of the last outputs, a MaxPooling of all the outputs and an AveragePooling of all the outputs. It then uses a list of BatchNorm, Dropout, Linear, ReLU blocks (with no ReLU in the last one), using a first layer size of 3*emb_sz then following the numbers in n_layers. The dropouts probabilities are read in drops.

Note that the model returns a list of three things, the actual output being the first, the two others being the intermediate hidden states before and after dropout (used by the RNNTrainer). Most loss functions expect one output, so you should use a Callback to remove the other two if you're not using RNNTrainer.