{% raw %}

{% endraw %} {% raw %}

torch.cuda.set_device(1)
print(f'Using GPU #{torch.cuda.current_device()}: {torch.cuda.get_device_name()}')

Using GPU #1: GeForce GTX 1080 Ti

{% endraw %}

Base tokenization, batch transform, and DataBlock methods

{% raw %}

`class` `HF_BaseInput`[source]

HF_BaseInput(x, **kwargs) :: TensorBase

A Tensor which support subclass pickling, and maintains metadata when casting or after methods

{% endraw %} {% raw %}

{% endraw %}

A HF_BaseInput object is returned from the decodes method of HF_AfterBatchTransform as a means to customize @typedispatched functions like DataLoaders.show_batch and Learner.show_results. It uses the "input_ids" of a huggingface object as the representative tensor for show methods

{% raw %}

`class` `HF_BeforeBatchTransform`[source]

HF_BeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model, max_length=None, padding=True, truncation=True, is_split_into_words=False, tok_kwargs={}, **kwargs) :: Transform

Handles everything you need to assemble a mini-batch of inputs and targets, as well as decode the dictionary produced as a byproduct of the tokenization process in the encodes method.

{% endraw %} {% raw %}

{% endraw %}

HF_BeforeBatchTransform was inspired by this article. It handles both the tokenization and numericalization traditionally split apart in the fastai text DataBlock API. For huggingface tokenizers that require a prefix space, it will be included automatically. In its current incarnation, HF_BeforeBatchTransform can be used to tokenize multiple inputs (common in seq2seq models for tasks like summarization) and even apply different tokenizers and arguments to each.

Inputs can come in as a string or a list of tokens, the later being for tasks like Named Entity Recognition (NER), where you want to predict the label of each token.

Notes re: on-the-fly batch-time tokenization: The previous version of the library performed the tokenization/numericalization as a type transform when the raw data was read, and included a couple batch transforms to prepare the data for collation (e.g., to be made into a mini-batch). With this update, everything is done in a single batch transform. Why? Part of the inspiration had to do with the mechanics of the huggingrace tokenizer, in particular how by default it returns a collated mini-batch of data given a list of sequences. And where do we get a list of examples with fastai? In the batch transforms! So I thought, hey, why not do everything dynamically at batch time? And with a bit of tweaking, I got everything to work pretty well. The result is less code, faster mini-batch creation, less RAM utilization and time spent tokenizing (really helps with very large datasets), and more flexibility.

{% raw %}

`class` `HF_AfterBatchTransform`[source]

HF_AfterBatchTransform(hf_tokenizer, input_return_type=HF_BaseInput) :: Transform

Delegates (__call__,decode,setup) to (encodes,decodes,setups) if split_idx matches

{% endraw %} {% raw %}

{% endraw %}

With fastai 2.1.5, before batch transforms no longer have a decodes method ... and so, I've introduced a standard batch transform here, HF_AfterBatchTransform, that will do the decoding for us.

{% raw %}

`class` `HF_TextBlock`[source]

HF_TextBlock(hf_arch=None, hf_config=None, hf_tokenizer=None, hf_model=None, before_batch_tfm=None, after_batch_tfm=None, max_length=None, padding=True, truncation=True, is_split_into_words=False, input_return_type=HF_BaseInput, dl_type=SortedDL, before_batch_kwargs={}, after_batch_kwargs={}, tok_kwargs={}, **kwargs) :: TransformBlock

A basic wrapper that links defaults transforms for the data block API

{% endraw %} {% raw %}

{% endraw %}

A basic wrapper that links defaults transforms for the data block API

HF_TextBlock has been dramatically simplified from it's predecessor. It handles setting up your HF_BeforeBatchTransform and HF_AfterBatchTransform transforms regardless of data source (e.g., this will work with files, DataFrames, whatever). You must either pass in your own instance of a HF_BeforeBatchTransform class or the huggingface architecture and tokenizer via the hf_arch and hf_tokenizer (the other args are optional).

{% raw %}

`get_blurr_tfm`[source]

get_blurr_tfm(tfms_list, tfm_class=HF_BeforeBatchTransform)

{% endraw %} {% raw %}

{% endraw %}

Sequence classification

Below demonstrates how to contruct your DataBlock for a sequence classification task (e.g., a model that requires a single text input)

{% raw %}

path = untar_data(URLs.IMDB_SAMPLE)

model_path = Path('models')
imdb_df = pd.read_csv(path/'texts.csv')

{% endraw %} {% raw %}

imdb_df.head()

	label	text	is_valid
0	negative	Un-bleeping-believable! Meg Ryan doesn't even look her usual pert lovable self in this, which normally makes me forgive her shallow ticky acting schtick. Hard to believe she was the producer on this dog. Plus Kevin Kline: what kind of suicide trip has his career been on? Whoosh... Banzai!!! Finally this was directed by the guy who did Big Chill? Must be a replay of Jonestown - hollywood style. Wooofff!	False
1	positive	This is a extremely well-made film. The acting, script and camera-work are all first-rate. The music is good, too, though it is mostly early in the film, when things are still relatively cheery. There are no really superstars in the cast, though several faces will be familiar. The entire cast does an excellent job with the script.<br /><br />But it is hard to watch, because there is no good end to a situation like the one presented. It is now fashionable to blame the British for setting Hindus and Muslims against each other, and then cruelly separating them into two countries. There is som...	False
2	negative	Every once in a long while a movie will come along that will be so awful that I feel compelled to warn people. If I labor all my days and I can save but one soul from watching this movie, how great will be my joy.<br /><br />Where to begin my discussion of pain. For starters, there was a musical montage every five minutes. There was no character development. Every character was a stereotype. We had swearing guy, fat guy who eats donuts, goofy foreign guy, etc. The script felt as if it were being written as the movie was being shot. The production value was so incredibly low that it felt li...	False
3	positive	Name just says it all. I watched this movie with my dad when it came out and having served in Korea he had great admiration for the man. The disappointing thing about this film is that it only concentrate on a short period of the man's life - interestingly enough the man's entire life would have made such an epic bio-pic that it is staggering to imagine the cost for production.<br /><br />Some posters elude to the flawed characteristics about the man, which are cheap shots. The theme of the movie "Duty, Honor, Country" are not just mere words blathered from the lips of a high-brassed offic...	False
4	negative	This movie succeeds at being one of the most unique movies you've seen. However this comes from the fact that you can't make heads or tails of this mess. It almost seems as a series of challenges set up to determine whether or not you are willing to walk out of the movie and give up the money you just paid. If you don't want to feel slighted you'll sit through this horrible film and develop a real sense of pity for the actors involved, they've all seen better days, but then you realize they actually got paid quite a bit of money to do this and you'll lose pity for them just like you've alr...	False

{% endraw %}

There are a bunch of ways we can get at the four huggingface elements we need (e.g., architecture name, tokenizer, config, and model). We can just create them directly, or we can use one of the helper methods available via BLURR.

{% raw %}

model_cls = AutoModelForSequenceClassification

pretrained_model_name = "roberta-base" # "distilbert-base-uncased" "bert-base-uncased"
hf_arch, hf_config, hf_tokenizer, hf_model = BLURR.get_hf_objects(pretrained_model_name, model_cls=model_cls)

{% endraw %}

Once you have those elements, you can create your DataBlock as simple as the below.

{% raw %}

blocks = (HF_TextBlock(hf_arch, hf_config, hf_tokenizer, hf_model), CategoryBlock)
dblock = DataBlock(blocks=blocks, get_x=ColReader('text'), get_y=ColReader('label'), splitter=ColSplitter())

{% endraw %} {% raw %}

dls = dblock.dataloaders(imdb_df, bs=4)

{% endraw %} {% raw %}

b = dls.one_batch()

{% endraw %} {% raw %}

b = dls.one_batch(); len(b), len(b[0]['input_ids']), b[0]['input_ids'].shape, len(b[1])

(2, 4, torch.Size([4, 512]), 4)

{% endraw %}

Let's take a look at the actual types represented by our batch

{% raw %}

explode_types(b)

{tuple: [dict, fastai.torch_core.TensorCategory]}

{% endraw %} {% raw %}

dls.show_batch(dataloaders=dls, max_n=2, trunc_at=500)

	text	category
0	Raising Victor Vargas: A Review<br /><br />You know, Raising Victor Vargas is like sticking your hands into a big, steaming bowl of oatmeal. It's warm and gooey, but you're not sure if it feels right. Try as I might, no matter how warm and gooey Raising Victor Vargas became I was always aware that something didn't quite feel right. Victor Vargas suffers from a certain overconfidence on the director's part. Apparently, the director thought that the ethnic backdrop of a Latino family on the lower	negative
1	Many neglect that this isn't just a classic due to the fact that it's the first 3D game, or even the first shoot-'em-up. It's also one of the first stealth games, one of the only(and definitely the first) truly claustrophobic games, and just a pretty well-rounded gaming experience in general. With graphics that are terribly dated today, the game thrusts you into the role of B.J.(don't even think I'm going to attempt spelling his last name!), an American P.O.W. caught in an underground bunker.	positive

{% endraw %}

Tests

The tests below to ensure the core DataBlock code above works for all pretrained sequence classification models available in huggingface. These tests are excluded from the CI workflow because of how long they would take to run and the amount of data that would be required to download.

Note: Feel free to modify the code below to test whatever pretrained classification models you are working with ... and if any of your pretrained sequence classification models fail, please submit a github issue (or a PR if you'd like to fix it yourself)

{% raw %}

[ model_type for model_type in BLURR.get_models(task='SequenceClassification') 
 if (not model_type.__name__.startswith('TF')) ]

[transformers.models.albert.modeling_albert.AlbertForSequenceClassification,
 transformers.models.bart.modeling_bart.BartForSequenceClassification,
 transformers.models.bert.modeling_bert.BertForSequenceClassification,
 transformers.models.big_bird.modeling_big_bird.BigBirdForSequenceClassification,
 transformers.models.ctrl.modeling_ctrl.CTRLForSequenceClassification,
 transformers.models.camembert.modeling_camembert.CamembertForSequenceClassification,
 transformers.models.convbert.modeling_convbert.ConvBertForSequenceClassification,
 transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification,
 transformers.models.deberta_v2.modeling_deberta_v2.DebertaV2ForSequenceClassification,
 transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification,
 transformers.models.electra.modeling_electra.ElectraForSequenceClassification,
 transformers.models.flaubert.modeling_flaubert.FlaubertForSequenceClassification,
 transformers.models.funnel.modeling_funnel.FunnelForSequenceClassification,
 transformers.models.gpt2.modeling_gpt2.GPT2ForSequenceClassification,
 transformers.models.ibert.modeling_ibert.IBertForSequenceClassification,
 transformers.models.led.modeling_led.LEDForSequenceClassification,
 transformers.models.layoutlm.modeling_layoutlm.LayoutLMForSequenceClassification,
 transformers.models.longformer.modeling_longformer.LongformerForSequenceClassification,
 transformers.models.mbart.modeling_mbart.MBartForSequenceClassification,
 transformers.models.mpnet.modeling_mpnet.MPNetForSequenceClassification,
 transformers.models.mobilebert.modeling_mobilebert.MobileBertForSequenceClassification,
 transformers.models.openai.modeling_openai.OpenAIGPTForSequenceClassification,
 transformers.models.reformer.modeling_reformer.ReformerForSequenceClassification,
 transformers.models.roberta.modeling_roberta.RobertaForSequenceClassification,
 transformers.models.squeezebert.modeling_squeezebert.SqueezeBertForSequenceClassification,
 transformers.models.tapas.modeling_tapas.TapasForSequenceClassification,
 transformers.models.transfo_xl.modeling_transfo_xl.TransfoXLForSequenceClassification,
 transformers.models.xlm.modeling_xlm.XLMForSequenceClassification,
 transformers.models.xlm_roberta.modeling_xlm_roberta.XLMRobertaForSequenceClassification,
 transformers.models.xlnet.modeling_xlnet.XLNetForSequenceClassification]

{% endraw %} {% raw %}

pretrained_model_names = [
    'albert-base-v1',
    'facebook/bart-base',
    'bert-base-uncased',
    'sshleifer/tiny-ctrl',
    'camembert-base',
    'microsoft/deberta-base',
    'distilbert-base-uncased',
    'monologg/electra-small-finetuned-imdb',
    'flaubert/flaubert_small_cased', 
    'huggingface/funnel-small-base',
    'gpt2',
    'allenai/led-base-16384',
    'allenai/longformer-base-4096',
    'sshleifer/tiny-mbart', 
    'microsoft/mpnet-base',
    'google/mobilebert-uncased',
    'openai-gpt',
    #'reformer-enwik8',                  (see model card; does not work with/require a tokenizer so no bueno here)
    'roberta-base',
    'squeezebert/squeezebert-uncased',
    #'google/tapas-base',                (requires pip install torch-scatter)
    'transfo-xl-wt103', 
    'xlm-mlm-en-2048',
    'xlm-roberta-base',
    'xlnet-base-cased'
]

{% endraw %} {% raw %}

path = untar_data(URLs.IMDB_SAMPLE)

model_path = Path('models')
imdb_df = pd.read_csv(path/'texts.csv')

{% endraw %} {% raw %}

#hide_output
model_cls = AutoModelForSequenceClassification
bsz = 2
seq_sz = 128

test_results = []
for model_name in pretrained_model_names:
    error=None
    
    print(f'=== {model_name} ===\n')

    hf_arch, hf_config, hf_tokenizer, hf_model = BLURR.get_hf_objects(model_name, model_cls=model_cls)    
    print(f'architecture:\t{hf_arch}\ntokenizer:\t{type(hf_tokenizer).__name__}\n')
    
    # not all architectures include a native pad_token (e.g., gpt2, ctrl, etc...), so we add one here
    if (hf_tokenizer.pad_token is None): 
        hf_tokenizer.add_special_tokens({'pad_token': '<pad>'})  
        hf_config.pad_token_id = hf_tokenizer.get_vocab()['<pad>']
        hf_model.resize_token_embeddings(len(hf_tokenizer))   
    
    blocks = (
        HF_TextBlock(hf_arch, hf_config, hf_tokenizer, hf_model, padding='max_length', max_length=seq_sz), 
        CategoryBlock
    )

    dblock = DataBlock(blocks=blocks, get_x=ColReader('text'), get_y=ColReader('label'), splitter=ColSplitter())
    dls = dblock.dataloaders(imdb_df, bs=bsz) 
    b = dls.one_batch()
    
    try:
        print('*** TESTING DataLoaders ***\n')
        test_eq(len(b), 2)
        test_eq(len(b[0]['input_ids']), bsz)
        test_eq(b[0]['input_ids'].shape, torch.Size([bsz, seq_sz]))
        test_eq(len(b[1]), bsz)

        if (hasattr(hf_tokenizer, 'add_prefix_space') and hf_arch not in ['led']):
             test_eq(hf_tokenizer.add_prefix_space, True)
            
        test_results.append((hf_arch, type(hf_tokenizer).__name__, model_name, 'PASSED', ''))
        dls.show_batch(dataloaders=dls, max_n=2, trunc_at=250)
        
    except Exception as err:
        test_results.append((hf_arch, type(hf_tokenizer).__name__, model_name, 'FAILED', err))

{% endraw %} {% raw %}

	arch	tokenizer	model_name	result	error
0	albert	AlbertTokenizerFast	albert-base-v1	PASSED
1	bart	BartTokenizerFast	facebook/bart-base	PASSED
2	bert	BertTokenizerFast	bert-base-uncased	PASSED
3	ctrl	CTRLTokenizer	sshleifer/tiny-ctrl	PASSED
4	camembert	CamembertTokenizerFast	camembert-base	PASSED
5	deberta	DebertaTokenizer	microsoft/deberta-base	FAILED	==:\nFalse\nTrue
6	distilbert	DistilBertTokenizerFast	distilbert-base-uncased	PASSED
7	electra	ElectraTokenizerFast	monologg/electra-small-finetuned-imdb	PASSED
8	flaubert	FlaubertTokenizer	flaubert/flaubert_small_cased	PASSED
9	funnel	FunnelTokenizerFast	huggingface/funnel-small-base	PASSED
10	gpt2	GPT2TokenizerFast	gpt2	PASSED
11	led	LEDTokenizerFast	allenai/led-base-16384	PASSED
12	longformer	LongformerTokenizerFast	allenai/longformer-base-4096	PASSED
13	mbart	MBartTokenizerFast	sshleifer/tiny-mbart	PASSED
14	mpnet	MPNetTokenizerFast	microsoft/mpnet-base	PASSED
15	mobilebert	MobileBertTokenizerFast	google/mobilebert-uncased	PASSED
16	openai	OpenAIGPTTokenizerFast	openai-gpt	PASSED
17	roberta	RobertaTokenizerFast	roberta-base	PASSED
18	squeezebert	SqueezeBertTokenizerFast	squeezebert/squeezebert-uncased	PASSED
19	transfo_xl	TransfoXLTokenizer	transfo-xl-wt103	PASSED
20	xlm	XLMTokenizer	xlm-mlm-en-2048	PASSED
21	xlm_roberta	XLMRobertaTokenizerFast	xlm-roberta-base	PASSED
22	xlnet	XLNetTokenizerFast	xlnet-base-cased	PASSED

{% endraw %}

Base tokenization, batch transform, and DataBlock methods

class HF_BaseInput[source]

class HF_BeforeBatchTransform[source]

class HF_AfterBatchTransform[source]

class HF_TextBlock[source]

get_blurr_tfm[source]

Sequence classification

Tests

Cleanup

`class` `HF_BaseInput`[source]

`class` `HF_BeforeBatchTransform`[source]

`class` `HF_AfterBatchTransform`[source]

`class` `HF_TextBlock`[source]

`get_blurr_tfm`[source]