--- title: utils keywords: fastai sidebar: home_sidebar summary: "Various utility functions used by the blurr package." description: "Various utility functions used by the blurr package." nb_path: "nbs/00_utils.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}
torch.cuda.set_device(1)
print(f'Using GPU #{torch.cuda.current_device()}: {torch.cuda.get_device_name()}')
Using GPU #1: GeForce GTX 1080 Ti
{% endraw %} {% raw %}
{% endraw %} {% raw %}

str_to_class[source]

str_to_class(classname)

converts string representation to class

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class Singleton[source]

Singleton()

{% endraw %}

Singleton functions as python decorator. Use this above any class to turn that class into a singleton (see here for more info on the singleton pattern).

{% raw %}
@Singleton
class TestSingleton: pass

a = TestSingleton()
b = TestSingleton()
test_eq(a,b)
{% endraw %}

ModelHelper

{% raw %}
{% endraw %} {% raw %}

Singleton object at 0x7f6a5c12db90>[source]

Singleton object at 0x7f6a5c12db90>(*args, **kwargs)

{% endraw %}

ModelHelper is a Singleton (there exists only one instance, and the same instance is returned upon subsequent instantiation requests). You can get at via the BLURR_MODEL_HELPER constant below.

{% raw %}
mh = ModelHelper()
mh2 = ModelHelper()
test_eq(mh, mh2)
{% endraw %}

Provide global helper constant

Users of this library can simply use BLURR_MODEL_HELPER to access all the ModelHelper capabilities without having to fetch an instance themselves.

{% raw %}
{% endraw %} {% raw %}

ModelHelper.get_architectures[source]

ModelHelper.get_architectures()

Used to get all the architectures supported by your Transformers install

{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_architectures())
['albert', 'auto', 'bart', 'bert', 'bert_japanese', 'camembert', 'ctrl', 'distilbert', 'dpr', 'electra', 'encoder_decoder', 'flaubert', 'gpt2', 'longformer', 'marian', 'mbart', 'mmbt', 'mobilebert', 'openai', 'pegasus', 'reformer', 'retribert', 'roberta', 't5', 'transfo_xl', 'utils_base', 'utils_fast', 'xlm', 'xlm_roberta', 'xlnet']
{% endraw %}

We'll also create an enum for downstream tasks

{% raw %}
{% endraw %} {% raw %}
print(L(HF_ARCHITECTURES))
(#30) [<HF_ARCHITECTURES.albert: 1>,<HF_ARCHITECTURES.auto: 2>,<HF_ARCHITECTURES.bart: 3>,<HF_ARCHITECTURES.bert: 4>,<HF_ARCHITECTURES.bert_japanese: 5>,<HF_ARCHITECTURES.camembert: 6>,<HF_ARCHITECTURES.ctrl: 7>,<HF_ARCHITECTURES.distilbert: 8>,<HF_ARCHITECTURES.dpr: 9>,<HF_ARCHITECTURES.electra: 10>...]
{% endraw %} {% raw %}

ModelHelper.get_config[source]

ModelHelper.get_config(arch)

Used the locate the name of the configuration class for a given architecture

{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_config('bert'))
<class 'transformers.configuration_bert.BertConfig'>
{% endraw %} {% raw %}

ModelHelper.get_tokenizers[source]

ModelHelper.get_tokenizers(arch)

Used to get the available huggingface tokenizers for a given architecture. Note: There may be multiple tokenizers and so this returns a list.

{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_tokenizers('electra'))
[<class 'transformers.tokenization_electra.ElectraTokenizer'>, <class 'transformers.tokenization_electra.ElectraTokenizerFast'>]
{% endraw %} {% raw %}

ModelHelper.get_tasks[source]

ModelHelper.get_tasks(arch=None)

Get the type of tasks for which there is a custom model for (optional: by architecture). There are a number of customized models built for specific tasks like token classification, question/answering, LM, etc....

{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_tasks())
print('')
print(BLURR_MODEL_HELPER.get_tasks('bart'))
['CausalLM', 'Classification', 'ConditionalGeneration', 'LMHead', 'LMHeadModel', 'MaskedLM', 'MultipleChoice', 'NextSentencePrediction', 'PreTraining', 'QuestionAnswering', 'QuestionAnsweringSimple', 'Seq2SeqLM', 'SequenceClassification', 'TokenClassification']

['ConditionalGeneration', 'QuestionAnswering', 'SequenceClassification']
{% endraw %}

We'll create an enum for tasks as well, one for all tasks and another for tasks available via huggingface's AutoModel capabilities

{% raw %}
{% endraw %} {% raw %}
print('--- all tasks ---')
print(L(HF_TASKS_ALL))
print('\n--- auto only ---')
print(L(HF_TASKS_AUTO))
--- all tasks ---
(#14) [<HF_TASKS_ALL.CausalLM: 1>,<HF_TASKS_ALL.Classification: 2>,<HF_TASKS_ALL.ConditionalGeneration: 3>,<HF_TASKS_ALL.LMHead: 4>,<HF_TASKS_ALL.LMHeadModel: 5>,<HF_TASKS_ALL.MaskedLM: 6>,<HF_TASKS_ALL.MultipleChoice: 7>,<HF_TASKS_ALL.NextSentencePrediction: 8>,<HF_TASKS_ALL.PreTraining: 9>,<HF_TASKS_ALL.QuestionAnswering: 10>...]

--- auto only ---
(#9) [<HF_TASKS_AUTO.CausalLM: 1>,<HF_TASKS_AUTO.LMHead: 2>,<HF_TASKS_AUTO.MaskedLM: 3>,<HF_TASKS_AUTO.MultipleChoice: 4>,<HF_TASKS_AUTO.PreTraining: 5>,<HF_TASKS_AUTO.QuestionAnswering: 6>,<HF_TASKS_AUTO.Seq2SeqLM: 7>,<HF_TASKS_AUTO.SequenceClassification: 8>,<HF_TASKS_AUTO.TokenClassification: 9>]
{% endraw %} {% raw %}
HF_TASKS_ALL.Classification
<HF_TASKS_ALL.Classification: 2>
{% endraw %} {% raw %}

ModelHelper.get_models[source]

ModelHelper.get_models(arch=None, task=None)

The transformer models available for use (optional: by architecture | task)

{% endraw %} {% raw %}
print(L(BLURR_MODEL_HELPER.get_models()))
(#151) [<class 'transformers.modeling_transfo_xl.AdaptiveEmbedding'>,<class 'transformers.modeling_albert.AlbertForMaskedLM'>,<class 'transformers.modeling_albert.AlbertForMultipleChoice'>,<class 'transformers.modeling_albert.AlbertForPreTraining'>,<class 'transformers.modeling_albert.AlbertForQuestionAnswering'>,<class 'transformers.modeling_albert.AlbertForSequenceClassification'>,<class 'transformers.modeling_albert.AlbertForTokenClassification'>,<class 'transformers.modeling_albert.AlbertModel'>,<class 'transformers.modeling_albert.AlbertPreTrainedModel'>,<class 'transformers.modeling_auto.AutoModel'>...]
{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_models(arch='bert'))
[<class 'transformers.modeling_bert.BertForMaskedLM'>, <class 'transformers.modeling_bert.BertForMultipleChoice'>, <class 'transformers.modeling_bert.BertForNextSentencePrediction'>, <class 'transformers.modeling_bert.BertForPreTraining'>, <class 'transformers.modeling_bert.BertForQuestionAnswering'>, <class 'transformers.modeling_bert.BertForSequenceClassification'>, <class 'transformers.modeling_bert.BertForTokenClassification'>, <class 'transformers.modeling_bert.BertLMHeadModel'>, <class 'transformers.modeling_bert.BertLayer'>, <class 'transformers.modeling_bert.BertModel'>, <class 'transformers.modeling_bert.BertPreTrainedModel'>]
{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_models(task='TokenClassification'))
[<class 'transformers.modeling_albert.AlbertForTokenClassification'>, <class 'transformers.modeling_auto.AutoModelForTokenClassification'>, <class 'transformers.modeling_bert.BertForTokenClassification'>, <class 'transformers.modeling_camembert.CamembertForTokenClassification'>, <class 'transformers.modeling_distilbert.DistilBertForTokenClassification'>, <class 'transformers.modeling_electra.ElectraForTokenClassification'>, <class 'transformers.modeling_flaubert.FlaubertForTokenClassification'>, <class 'transformers.modeling_longformer.LongformerForTokenClassification'>, <class 'transformers.modeling_mobilebert.MobileBertForTokenClassification'>, <class 'transformers.modeling_roberta.RobertaForTokenClassification'>, <class 'transformers.modeling_xlm.XLMForTokenClassification'>, <class 'transformers.modeling_xlm_roberta.XLMRobertaForTokenClassification'>, <class 'transformers.modeling_xlnet.XLNetForTokenClassification'>]
{% endraw %} {% raw %}
print(BLURR_MODEL_HELPER.get_models(arch='bert', task='TokenClassification'))
[<class 'transformers.modeling_bert.BertForTokenClassification'>]
{% endraw %} {% raw %}

ModelHelper.get_classes_for_model[source]

ModelHelper.get_classes_for_model(model_name_or_cls)

Get tokenizers, config, and model for a given model name / class

{% endraw %} {% raw %}
config, tokenizers, model = BLURR_MODEL_HELPER.get_classes_for_model('RobertaForSequenceClassification')

print(config)
print(tokenizers[0])
print(model)
<class 'transformers.configuration_roberta.RobertaConfig'>
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
<class 'transformers.modeling_roberta.RobertaForSequenceClassification'>
{% endraw %} {% raw %}
config, tokenizers, model = BLURR_MODEL_HELPER.get_classes_for_model(DistilBertModel)

print(config)
print(tokenizers[0])
print(model)
<class 'transformers.configuration_distilbert.DistilBertConfig'>
<class 'transformers.tokenization_distilbert.DistilBertTokenizer'>
<class 'transformers.modeling_distilbert.DistilBertModel'>
{% endraw %} {% raw %}

ModelHelper.get_model_architecture[source]

ModelHelper.get_model_architecture(model_name_or_enum)

Get the architecture for a given model name / enum

{% endraw %} {% raw %}
BLURR_MODEL_HELPER.get_model_architecture('RobertaForSequenceClassification')
'roberta'
{% endraw %}

Methods for loading pre-trained (configs, tokenizer, model) hugginface classes

{% raw %}

ModelHelper.get_hf_objects[source]

ModelHelper.get_hf_objects(pretrained_model_name_or_path, task=None, config=None, tokenizer_cls=None, model_cls=None, config_kwargs={}, tokenizer_kwargs={}, model_kwargs={}, cache_dir=None)

Returns the architecture (str), config (obj), tokenizer (obj), and model (obj) given at minimum a pre-trained model name or path. Specify a task to ensure the right "AutoModelFor" is used to create the model.

Optionally, you can pass a config (obj), tokenizer (class), and/or model (class) (along with any related kwargs for each) to get as specific as you want w/r/t what huggingface objects are returned.

{% endraw %} {% raw %}
arch, config, tokenizer, model = BLURR_MODEL_HELPER.get_hf_objects("bert-base-cased-finetuned-mrpc",
                                                                   task=HF_TASKS_AUTO.LMHead)

print(arch)
print(type(config))
print(type(tokenizer))
print(type(model))
/home/wgilliam/anaconda3/envs/blurr/lib/python3.7/site-packages/transformers/modeling_auto.py:821: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
Some weights of the model checkpoint at bert-base-cased-finetuned-mrpc were not used when initializing BertForMaskedLM: ['classifier.weight', 'classifier.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-base-cased-finetuned-mrpc and are newly initialized: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
bert
<class 'transformers.configuration_bert.BertConfig'>
<class 'transformers.tokenization_bert.BertTokenizer'>
<class 'transformers.modeling_bert.BertForMaskedLM'>
{% endraw %} {% raw %}
arch, tokenizer, config, model = BLURR_MODEL_HELPER.get_hf_objects("fmikaelian/flaubert-base-uncased-squad",
                                                                   task=HF_TASKS_AUTO.QuestionAnswering)

print(arch)
print(type(config))
print(type(tokenizer))
print(type(model))
Some weights of the model checkpoint at fmikaelian/flaubert-base-uncased-squad were not used when initializing FlaubertForQuestionAnsweringSimple: ['qa_outputs.start_logits.dense.weight', 'qa_outputs.start_logits.dense.bias', 'qa_outputs.end_logits.dense_0.weight', 'qa_outputs.end_logits.dense_0.bias', 'qa_outputs.end_logits.LayerNorm.weight', 'qa_outputs.end_logits.LayerNorm.bias', 'qa_outputs.end_logits.dense_1.weight', 'qa_outputs.end_logits.dense_1.bias', 'qa_outputs.answer_class.dense_0.weight', 'qa_outputs.answer_class.dense_0.bias', 'qa_outputs.answer_class.dense_1.weight']
- This IS expected if you are initializing FlaubertForQuestionAnsweringSimple from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing FlaubertForQuestionAnsweringSimple from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of FlaubertForQuestionAnsweringSimple were not initialized from the model checkpoint at fmikaelian/flaubert-base-uncased-squad and are newly initialized: ['transformer.position_ids', 'qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
flaubert
<class 'transformers.tokenization_flaubert.FlaubertTokenizer'>
<class 'transformers.configuration_flaubert.FlaubertConfig'>
<class 'transformers.modeling_flaubert.FlaubertForQuestionAnsweringSimple'>
{% endraw %} {% raw %}
arch, tokenizer, config, model = BLURR_MODEL_HELPER.get_hf_objects("bert-base-cased-finetuned-mrpc",
                                                                   config=None,
                                                                   tokenizer_cls=BertTokenizer, 
                                                                   model_cls=BertForNextSentencePrediction)
print(arch)
print(type(config))
print(type(tokenizer))
print(type(model))
Some weights of the model checkpoint at bert-base-cased-finetuned-mrpc were not used when initializing BertForNextSentencePrediction: ['classifier.weight', 'classifier.bias']
- This IS expected if you are initializing BertForNextSentencePrediction from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForNextSentencePrediction from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForNextSentencePrediction were not initialized from the model checkpoint at bert-base-cased-finetuned-mrpc and are newly initialized: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
bert
<class 'transformers.tokenization_bert.BertTokenizer'>
<class 'transformers.configuration_bert.BertConfig'>
<class 'transformers.modeling_bert.BertForNextSentencePrediction'>
{% endraw %}

Cleanup