--- title: modeling.core keywords: fastai sidebar: home_sidebar summary: "This module contains core custom models, loss functions, and a default layer group splitter for use in applying discriminiative learning rates to your huggingface models trained via fastai" description: "This module contains core custom models, loss functions, and a default layer group splitter for use in applying discriminiative learning rates to your huggingface models trained via fastai" nb_path: "nbs/02_modeling-core.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}
torch.cuda.set_device(1)
print(f'Using GPU #{torch.cuda.current_device()}: {torch.cuda.get_device_name()}')
Using GPU #1: GeForce GTX 1080 Ti
{% endraw %}

Base splitter, model wrapper, and model callback

{% raw %}
{% endraw %} {% raw %}

hf_splitter[source]

hf_splitter(m)

Splits the huggingface model based on various model architecture conventions

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class HF_BaseModelWrapper[source]

HF_BaseModelWrapper(hf_model) :: Module

Same as nn.Module, but no need for subclasses to call super().__init__

{% endraw %}

Note that HF_baseModelWrapper includes some nifty code for just passing in the things your model needs, as not all transformer architectures require/use the same information.

{% raw %}
{% endraw %} {% raw %}

class HF_BaseModelCallback[source]

HF_BaseModelCallback(before_fit=None, before_epoch=None, before_train=None, before_batch=None, after_pred=None, after_loss=None, before_backward=None, after_backward=None, after_step=None, after_cancel_batch=None, after_batch=None, after_cancel_train=None, after_train=None, before_validate=None, after_cancel_validate=None, after_validate=None, after_cancel_epoch=None, after_epoch=None, after_cancel_fit=None, after_fit=None) :: Callback

Basic class handling tweaks of the training loop by changing a Learner in various events

{% endraw %}

We use a Callback for handling what is returned from the huggingface model ... "the huggingface model will return a tuple in outputs, with the actual predictions and some additional activations (should we want to use them is some regularization scheme)" - from the fastai Transformer's Tutorial

Sequence classification

Below demonstrates how to setup your blurr pipeline for a sequence classification task (e.g., a model that requires a single text input)

{% raw %}
path = untar_data(URLs.IMDB_SAMPLE)
imdb_df = pd.read_csv(path/'texts.csv')
{% endraw %} {% raw %}
imdb_df.head()
label text is_valid
0 negative Un-bleeping-believable! Meg Ryan doesn't even look her usual pert lovable self in this, which normally makes me forgive her shallow ticky acting schtick. Hard to believe she was the producer on this dog. Plus Kevin Kline: what kind of suicide trip has his career been on? Whoosh... Banzai!!! Finally this was directed by the guy who did Big Chill? Must be a replay of Jonestown - hollywood style. Wooofff! False
1 positive This is a extremely well-made film. The acting, script and camera-work are all first-rate. The music is good, too, though it is mostly early in the film, when things are still relatively cheery. There are no really superstars in the cast, though several faces will be familiar. The entire cast does an excellent job with the script.<br /><br />But it is hard to watch, because there is no good end to a situation like the one presented. It is now fashionable to blame the British for setting Hindus and Muslims against each other, and then cruelly separating them into two countries. There is som... False
2 negative Every once in a long while a movie will come along that will be so awful that I feel compelled to warn people. If I labor all my days and I can save but one soul from watching this movie, how great will be my joy.<br /><br />Where to begin my discussion of pain. For starters, there was a musical montage every five minutes. There was no character development. Every character was a stereotype. We had swearing guy, fat guy who eats donuts, goofy foreign guy, etc. The script felt as if it were being written as the movie was being shot. The production value was so incredibly low that it felt li... False
3 positive Name just says it all. I watched this movie with my dad when it came out and having served in Korea he had great admiration for the man. The disappointing thing about this film is that it only concentrate on a short period of the man's life - interestingly enough the man's entire life would have made such an epic bio-pic that it is staggering to imagine the cost for production.<br /><br />Some posters elude to the flawed characteristics about the man, which are cheap shots. The theme of the movie "Duty, Honor, Country" are not just mere words blathered from the lips of a high-brassed offic... False
4 negative This movie succeeds at being one of the most unique movies you've seen. However this comes from the fact that you can't make heads or tails of this mess. It almost seems as a series of challenges set up to determine whether or not you are willing to walk out of the movie and give up the money you just paid. If you don't want to feel slighted you'll sit through this horrible film and develop a real sense of pity for the actors involved, they've all seen better days, but then you realize they actually got paid quite a bit of money to do this and you'll lose pity for them just like you've alr... False
{% endraw %} {% raw %}
task = HF_TASKS_AUTO.SequenceClassification

pretrained_model_name = "roberta-base" # "distilbert-base-uncased" "bert-base-uncased"
hf_arch, hf_config, hf_tokenizer, hf_model = BLURR_MODEL_HELPER.get_hf_objects(pretrained_model_name, task=task)
{% endraw %} {% raw %}
blocks = (HF_TextBlock(hf_arch=hf_arch, hf_tokenizer=hf_tokenizer), CategoryBlock)

dblock = DataBlock(blocks=blocks, 
                   get_x=ColReader('text'), 
                   get_y=ColReader('label'), 
                   splitter=ColSplitter(col='is_valid'))
{% endraw %} {% raw %}
dls = dblock.dataloaders(imdb_df, bs=4)
{% endraw %} {% raw %}
dls.show_batch(dataloaders=dls, max_n=2)
text category
0 Raising Victor Vargas: A Review<br /><br />You know, Raising Victor Vargas is like sticking your hands into a big, steaming bowl of oatmeal. It's warm and gooey, but you're not sure if it feels right. Try as I might, no matter how warm and gooey Raising Victor Vargas became I was always aware that something didn't quite feel right. Victor Vargas suffers from a certain overconfidence on the director's part. Apparently, the director thought that the ethnic backdrop of a Latino family on the lower east side, and an idyllic storyline would make the film critic proof. He was right, but it didn't fool me. Raising Victor Vargas is the story about a seventeen-year old boy called, you guessed it, Victor Vargas (Victor Rasuk) who lives his teenage years chasing more skirt than the Rolling Stones could do in all the years they've toured. The movie starts off in `Ugly Fat' Donna's bedroom where Victor is sure to seduce her, but a cry from outside disrupts his plans when his best-friend Harold (Kevin Rivera) comes-a-looking for him. Caught in the attempt by Harold and his sister, Victor Vargas runs off for damage control. Yet even with the embarrassing implication that he's been boffing the homeliest girl in the neighborhood, nothing dissuades young Victor from going off on the hunt for more fresh meat. On a hot, New York City day they make way to the local public swimming pool where Victor's eyes catch a glimpse of the lovely young nymph Judy (Judy Marte), who's not just pretty, but a strong and independent too. The relationship that develops between Victor and Judy becomes the focus of the film. The story also focuses on Victor's family that is comprised of his grandmother or abuelita (Altagracia Guzman), his brother Nino (also played by real life brother to Victor, Silvestre Rasuk) and his sister Vicky (Krystal Rodriguez). The action follows Victor between scenes with Judy and scenes with his family. Victor tries to cope with being an oversexed pimp-daddy, his feelings for Judy and his grandmother's conservative Catholic upbringing.<br /><br />The problems that arise from Raising Victor Vargas are a few, but glaring errors. Throughout the film you get to know certain characters like Vicky, Nino, Grandma, negative
1 THE SHOP AROUND THE CORNER is one of the sweetest and most feel-good romantic comedies ever made. There's just no getting around that, and it's hard to actually put one's feeling for this film into words. It's not one of those films that tries too hard, nor does it come up with the oddest possible scenarios to get the two protagonists together in the end. In fact, all its charm is innate, contained within the characters and the setting and the plot... which is highly believable to boot. It's easy to think that such a love story, as beautiful as any other ever told, *could* happen to you... a feeling you don't often get from other romantic comedies, however sweet and heart-warming they may be. <br /><br />Alfred Kralik (James Stewart) and Clara Novak (Margaret Sullavan) don't have the most auspicious of first meetings when she arrives in the shop (Matuschek & Co.) he's been working in for the past nine years, asking for a job. They clash from the very beginning, mostly over a cigarette box that plays music when it's opened--he thinks it's a ludicrous idea; she makes one big sell of it and gets hired. Their bickering takes them through the next six months, even as they both (unconsciously, of course!) fall in love with each other when they share their souls and minds in letters passed through PO Box 237. This would be a pretty thin plotline to base an entire film on, except that THE SHOP AROUND THE CORNER is expertly fleshed-out with a brilliant supporting cast made up of entirely engaging characters, from the fatherly but lonely Hugo Matuschek (Frank Morgan) himself, who learns that his shop really is his home; Pirovitch (Felix Bressart), Kralik's sidekick and friend who always skitters out of the room when faced with the possibility of being asked for his honest opinion; smarmy pimp-du-jour Vadas (Joseph Schildkraut) who ultimately gets his comeuppance from a gloriously righteous Kralik; and ambitious errand boy Pepi Katona (William Tracy) who wants nothing more than to be promoted to the position of clerk for Matuschek & Co. The unpretentious love story between 'Dear Friends' is played out in this little shop in positive
{% endraw %}

Training

We'll also add in custom summary methods for blurr learners/models that work with dictionary inputs

{% raw %}
model = HF_BaseModelWrapper(hf_model)

learn = Learner(dls, 
                model,
                opt_func=partial(Adam),
                loss_func=CrossEntropyLossFlat(),
                metrics=[accuracy],
                cbs=[HF_BaseModelCallback],
                splitter=hf_splitter)

learn.create_opt()             # -> will create your layer groups based on your "splitter" function
learn.freeze()
{% endraw %}

.to_fp16() requires a GPU so had to remove for tests to run on github. Let's check that we can get predictions.

{% raw %}
b = dls.one_batch()
{% endraw %} {% raw %}
learn.model(b[0])
(tensor([[0.0173, 0.0322],
         [0.0076, 0.0268],
         [0.0125, 0.0255],
         [0.0141, 0.0351]], device='cuda:1', grad_fn=<AddmmBackward>),)
{% endraw %} {% raw %}
{% endraw %} {% raw %}

blurr_module_summary[source]

blurr_module_summary(learn, *xb)

Print a summary of model using xb

{% endraw %} {% raw %}
{% endraw %} {% raw %}

Learner.blurr_summary[source]

Learner.blurr_summary()

Print a summary of the model, optimizer and loss function.

{% endraw %}

We have to create our own summary methods above because fastai only works where things are represented by a single tensor. But in the case of huggingface transformers, a single sequence is represented by multiple tensors (in a dictionary).

The change to make this work is so minor I think that the fastai library can/will hopefully be updated to support this use case.

{% raw %}
 
{% endraw %} {% raw %}
print(len(learn.opt.param_groups))
4
{% endraw %} {% raw %}
learn.lr_find(suggestions=True)
SuggestedLRs(lr_min=1.584893179824576e-05, lr_steep=0.03981071710586548)
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=1e-3)
epoch train_loss valid_loss accuracy time
0 0.290000 0.270782 0.900000 00:21
{% endraw %}

Showing results

And here we creat a @typedispatched impelmentation of Learner.show_results.

{% raw %}
{% endraw %} {% raw %}
learn.show_results(learner=learn, max_n=2)
text category target
0 The trouble with the book, "Memoirs of a Geisha" is that it had Japanese surfaces but underneath the surfaces it was all an American man's way of thinking. Reading the book is like watching a magnificent ballet with great music, sets, and costumes yet performed by barnyard animals dressed in those costumes—so far from Japanese ways of thinking were the characters.<br /><br />The movie isn't about Japan or real geisha. It is a story about a few American men's mistaken ideas about Japan and geisha filtered through their own ignorance and misconceptions. So what is this movie if it isn't about Japan or geisha? Is it pure fantasy as so many people have said? Yes, but then why make it into an American fantasy?<br /><br />There were so many missed opportunities. Imagine a culture where there are no puritanical hang-ups, no connotations of sin about sex. Sex is natural and normal. How is sex handled in this movie? Right. Like it was dirty. The closest thing to a sex scene in the movie has Sayuri wrinkling up her nose and grimacing with distaste for five seconds as if the man trying to mount her had dropped a handful of cockroaches on her crotch. <br /><br />Does anyone actually enjoy sex in this movie? Nope. One character is said to be promiscuous but all we see is her pushing away her lover because it looks like she doesn't want to get caught doing something dirty. Such typical American puritanism has no place in a movie about Japanese geisha.<br /><br />Did Sayuri enjoy her first ravishing by some old codger after her cherry was auctioned off? Nope. She lies there like a cold slab of meat on a chopping block. Of course she isn't supposed to enjoy it. And that is what I mean about this movie. Why couldn't they have given her something to enjoy? Why does all the sex have to be sinful and wrong?<br /><br />Behind Mameha the Chairman was Sayuri's secret patron, and as such he was behind the auction of her virginity. He could have rigged the auction and won her himself. Nobu didn't even bid. So why did the Chairman let that old codger win her and, reeking of old-man stink, get his fingers all over her naked body? Would any woman ever really forgive a man for that?<br /><br />Let's negative negative
1 <br /><br />I'm sure things didn't exactly go the same way in the real life of Homer Hickam as they did in the film adaptation of his book, Rocket Boys, but the movie "October Sky" (an anagram of the book's title) is good enough to stand alone. I have not read Hickam's memoirs, but I am still able to enjoy and understand their film adaptation. The film, directed by Joe Johnston and written by Lewis Colick, records the story of teenager Homer Hickam (Jake Gyllenhaal), beginning in October of 1957. It opens with the sound of a radio broadcast, bringing news of the Russian satellite Sputnik, the first artificial satellite in orbit. We see a images of a blue-gray town and its people: mostly miners working for the Olga Coal Company. One of the miners listens to the news on a hand-held radio as he enters the elevator shaft, but the signal is lost as he disappears into the darkness, losing sight of the starry sky above him. A melancholy violin tune fades with this image. We then get a jolt of Elvis on a car radio as words on the screen inform us of the setting: October 5, 1957, Coalwood, West Virginia. Homer and his buddies, Roy Lee Cook (William Lee Scott) and Sherman O'Dell (Chad Lindberg), are talking about football tryouts. Football scholarships are the only way out of the town, and working in the mines, for these boys. "Why are the jocks the only ones who get to go to college," questions Homer. Roy Lee replies, "They're also the only ones who get the girls." Homer doesn't make it in football like his older brother, so he is destined for the mines, and to follow in his father's footsteps as mine foreman. Until he sees the dot of light streaking across the October sky. Then he wants to build a rocket. "I want to go into space," says Homer. After a disastrous attempt involving a primitive rocket and his mother's (Natalie Canerday) fence, Homer enlists the help of the nerdy Quentin Wilson (Chris Owen). Quentin asks Homer, "What do you want to know about rockets?" Homer quickly anwers, "Everything." His science teacher at Big Creek High School, Miss Frieda Riley (Laura Dern) greatly supports Homer, and the four boys work on building rockets in Homer's basement. His father, positive positive
{% endraw %} {% raw %}
{% endraw %} {% raw %}

Learner.blurr_predict[source]

Learner.blurr_predict(item, rm_type_tfms=None, with_input=False)

{% endraw %}

Same as with summary, we need to replace fastai's Learner.predict method with the one above which is able to work with inputs that are represented by multiple tensors included in a dictionary.

{% raw %}
learn.blurr_predict('I really liked the movie')
('positive', tensor(1), tensor([0.0763, 0.9237]))
{% endraw %} {% raw %}
learn.unfreeze()
{% endraw %} {% raw %}
learn.fit_one_cycle(3, lr_max=slice(1e-7, 1e-4))
epoch train_loss valid_loss accuracy time
0 0.184161 0.222415 0.920000 00:34
1 0.167082 0.214611 0.920000 00:33
2 0.144612 0.213530 0.925000 00:34
{% endraw %} {% raw %}
learn.recorder.plot_loss()
{% endraw %} {% raw %}
learn.show_results(learner=learn, max_n=2)
text category target
0 The trouble with the book, "Memoirs of a Geisha" is that it had Japanese surfaces but underneath the surfaces it was all an American man's way of thinking. Reading the book is like watching a magnificent ballet with great music, sets, and costumes yet performed by barnyard animals dressed in those costumes—so far from Japanese ways of thinking were the characters.<br /><br />The movie isn't about Japan or real geisha. It is a story about a few American men's mistaken ideas about Japan and geisha filtered through their own ignorance and misconceptions. So what is this movie if it isn't about Japan or geisha? Is it pure fantasy as so many people have said? Yes, but then why make it into an American fantasy?<br /><br />There were so many missed opportunities. Imagine a culture where there are no puritanical hang-ups, no connotations of sin about sex. Sex is natural and normal. How is sex handled in this movie? Right. Like it was dirty. The closest thing to a sex scene in the movie has Sayuri wrinkling up her nose and grimacing with distaste for five seconds as if the man trying to mount her had dropped a handful of cockroaches on her crotch. <br /><br />Does anyone actually enjoy sex in this movie? Nope. One character is said to be promiscuous but all we see is her pushing away her lover because it looks like she doesn't want to get caught doing something dirty. Such typical American puritanism has no place in a movie about Japanese geisha.<br /><br />Did Sayuri enjoy her first ravishing by some old codger after her cherry was auctioned off? Nope. She lies there like a cold slab of meat on a chopping block. Of course she isn't supposed to enjoy it. And that is what I mean about this movie. Why couldn't they have given her something to enjoy? Why does all the sex have to be sinful and wrong?<br /><br />Behind Mameha the Chairman was Sayuri's secret patron, and as such he was behind the auction of her virginity. He could have rigged the auction and won her himself. Nobu didn't even bid. So why did the Chairman let that old codger win her and, reeking of old-man stink, get his fingers all over her naked body? Would any woman ever really forgive a man for that?<br /><br />Let's negative negative
1 <br /><br />I'm sure things didn't exactly go the same way in the real life of Homer Hickam as they did in the film adaptation of his book, Rocket Boys, but the movie "October Sky" (an anagram of the book's title) is good enough to stand alone. I have not read Hickam's memoirs, but I am still able to enjoy and understand their film adaptation. The film, directed by Joe Johnston and written by Lewis Colick, records the story of teenager Homer Hickam (Jake Gyllenhaal), beginning in October of 1957. It opens with the sound of a radio broadcast, bringing news of the Russian satellite Sputnik, the first artificial satellite in orbit. We see a images of a blue-gray town and its people: mostly miners working for the Olga Coal Company. One of the miners listens to the news on a hand-held radio as he enters the elevator shaft, but the signal is lost as he disappears into the darkness, losing sight of the starry sky above him. A melancholy violin tune fades with this image. We then get a jolt of Elvis on a car radio as words on the screen inform us of the setting: October 5, 1957, Coalwood, West Virginia. Homer and his buddies, Roy Lee Cook (William Lee Scott) and Sherman O'Dell (Chad Lindberg), are talking about football tryouts. Football scholarships are the only way out of the town, and working in the mines, for these boys. "Why are the jocks the only ones who get to go to college," questions Homer. Roy Lee replies, "They're also the only ones who get the girls." Homer doesn't make it in football like his older brother, so he is destined for the mines, and to follow in his father's footsteps as mine foreman. Until he sees the dot of light streaking across the October sky. Then he wants to build a rocket. "I want to go into space," says Homer. After a disastrous attempt involving a primitive rocket and his mother's (Natalie Canerday) fence, Homer enlists the help of the nerdy Quentin Wilson (Chris Owen). Quentin asks Homer, "What do you want to know about rockets?" Homer quickly anwers, "Everything." His science teacher at Big Creek High School, Miss Frieda Riley (Laura Dern) greatly supports Homer, and the four boys work on building rockets in Homer's basement. His father, positive positive
{% endraw %} {% raw %}
learn.blurr_predict("This was a really good movie")
('positive', tensor(1), tensor([0.1387, 0.8613]))
{% endraw %} {% raw %}
learn.blurr_predict("Acting was so bad it was almost funny.")
('negative', tensor(0), tensor([0.9001, 0.0999]))
{% endraw %}

Inference

{% raw %}
learn.export(fname='seq_class_learn_export.pkl')
{% endraw %} {% raw %}
inf_learn = load_learner(fname='seq_class_learn_export.pkl')
inf_learn.blurr_predict("This movie should not be seen by anyone!!!!")
('negative', tensor(0), tensor([0.8242, 0.1758]))
{% endraw %}

Tests

The tests below to ensure the core training code above works for all pretrained sequence classification models available in huggingface. These tests are excluded from the CI workflow because of how long they would take to run and the amount of data that would be required to download.

Note: Feel free to modify the code below to test whatever pretrained classification models you are working with ... and if any of your pretrained sequence classification models fail, please submit a github issue (or a PR if you'd like to fix it yourself)

{% raw %}
try: del learn; torch.cuda.empty_cache()
except: pass
{% endraw %} {% raw %}
BLURR_MODEL_HELPER.get_models(task='SequenceClassification')
[transformers.modeling_albert.AlbertForSequenceClassification,
 transformers.modeling_auto.AutoModelForSequenceClassification,
 transformers.modeling_bart.BartForSequenceClassification,
 transformers.modeling_bert.BertForSequenceClassification,
 transformers.modeling_camembert.CamembertForSequenceClassification,
 transformers.modeling_distilbert.DistilBertForSequenceClassification,
 transformers.modeling_electra.ElectraForSequenceClassification,
 transformers.modeling_flaubert.FlaubertForSequenceClassification,
 transformers.modeling_longformer.LongformerForSequenceClassification,
 transformers.modeling_mobilebert.MobileBertForSequenceClassification,
 transformers.modeling_reformer.ReformerForSequenceClassification,
 transformers.modeling_roberta.RobertaForSequenceClassification,
 transformers.modeling_xlm.XLMForSequenceClassification,
 transformers.modeling_xlm_roberta.XLMRobertaForSequenceClassification,
 transformers.modeling_xlnet.XLNetForSequenceClassification]
{% endraw %} {% raw %}
pretrained_model_names = [
    'albert-base-v1',
    'facebook/bart-base',
    'bert-base-uncased',
    'camembert-base',
    'distilbert-base-uncased',
    'monologg/electra-small-finetuned-imdb',
    'flaubert/flaubert_small_cased', 
    'allenai/longformer-base-4096',
    'google/mobilebert-uncased',
    'roberta-base',
    'xlm-mlm-en-2048',
    'xlm-roberta-base',
    'xlnet-base-cased'
]
{% endraw %} {% raw %}
path = untar_data(URLs.IMDB_SAMPLE)

model_path = Path('models')
imdb_df = pd.read_csv(path/'texts.csv')
{% endraw %} {% raw %}
#hide_output
task = HF_TASKS_AUTO.SequenceClassification
bsz = 2
seq_sz = 128

test_results = []
for model_name in pretrained_model_names:
    error=None
    
    print(f'=== {model_name} ===\n')
    
    hf_arch, hf_config, hf_tokenizer, hf_model = BLURR_MODEL_HELPER.get_hf_objects(model_name, 
                                                                                   task=task, 
                                                                                   config_kwargs={'num_labels': 2})
    
    print(f'architecture:\t{hf_arch}\ntokenizer:\t{type(hf_tokenizer).__name__}\nmodel:\t\t{type(hf_model).__name__}\n')

    blocks = (HF_TextBlock(hf_arch=hf_arch, hf_tokenizer=hf_tokenizer, max_length=seq_sz, padding='max_length'), 
              CategoryBlock)

    dblock = DataBlock(blocks=blocks, 
                       get_x=ColReader('text'), 
                       get_y=ColReader('label'), 
                       splitter=ColSplitter(col='is_valid'))
    
    dls = dblock.dataloaders(imdb_df, bs=bsz)
    
    model = HF_BaseModelWrapper(hf_model)
    learn = Learner(dls, 
                    model,
                    opt_func=partial(Adam),
                    loss_func=CrossEntropyLossFlat(),
                    metrics=[accuracy],
                    cbs=[HF_BaseModelCallback],
                    splitter=hf_splitter)

    learn.create_opt()             # -> will create your layer groups based on your "splitter" function
    learn.freeze()
    
    b = dls.one_batch()
    
    try:
        print('*** TESTING DataLoaders ***')
        test_eq(len(b), bsz)
        test_eq(len(b[0]['input_ids']), bsz)
        test_eq(b[0]['input_ids'].shape, torch.Size([bsz, seq_sz]))
        test_eq(len(b[1]), bsz)

        print('*** TESTING One pass through the model ***')
        preds = learn.model(b[0])
        test_eq(len(preds[0]), bsz)
        test_eq(preds[0].shape, torch.Size([bsz, 2]))

        print('*** TESTING Training/Results ***')
        learn.fit_one_cycle(1, lr_max=1e-3)

        test_results.append((hf_arch, type(hf_tokenizer).__name__, type(hf_model).__name__, 'PASSED', ''))
        learn.show_results(learner=learn, max_n=2)
    except Exception as err:
        test_results.append((hf_arch, type(hf_tokenizer).__name__, type(hf_model).__name__, 'FAILED', err))
    finally:
        # cleanup
        del learn; torch.cuda.empty_cache()
{% endraw %} {% raw %}
arch tokenizer model result error
0 albert AlbertTokenizer AlbertForSequenceClassification PASSED
1 bart BartTokenizer BartForSequenceClassification PASSED
2 bert BertTokenizer BertForSequenceClassification PASSED
3 camembert CamembertTokenizer CamembertForSequenceClassification PASSED
4 distilbert DistilBertTokenizer DistilBertForSequenceClassification PASSED
5 electra ElectraTokenizer ElectraForSequenceClassification PASSED
6 flaubert FlaubertTokenizer FlaubertForSequenceClassification PASSED
7 longformer LongformerTokenizer LongformerForSequenceClassification PASSED
8 mobilebert MobileBertTokenizer MobileBertForSequenceClassification PASSED
9 roberta RobertaTokenizer RobertaForSequenceClassification PASSED
10 xlm XLMTokenizer XLMForSequenceClassification PASSED
11 xlm_roberta XLMRobertaTokenizer XLMRobertaForSequenceClassification PASSED
12 xlnet XLNetTokenizer XLNetForSequenceClassification PASSED
{% endraw %}

Example: Multi-label classification

Below demonstrates how to setup your blurr pipeline for a multi-label classification task

{% raw %}
raw_data = nlp.load_dataset('civil_comments', split='train[:1%]') 
len(raw_data)
Using custom data configuration default
18049
{% endraw %} {% raw %}
toxic_df = pd.DataFrame(raw_data, columns=list(raw_data.features.keys()))
toxic_df.head()
text toxicity severe_toxicity obscene threat insult identity_attack sexual_explicit
0 This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done! 0.000000 0.000000 0.0 0.0 0.00000 0.000000 0.0
1 Thank you!! This would make my life a lot less anxiety-inducing. Keep it up, and don't let anyone get in your way! 0.000000 0.000000 0.0 0.0 0.00000 0.000000 0.0
2 This is such an urgent design problem; kudos to you for taking it on. Very impressive! 0.000000 0.000000 0.0 0.0 0.00000 0.000000 0.0
3 Is this something I'll be able to install on my site? When will you be releasing it? 0.000000 0.000000 0.0 0.0 0.00000 0.000000 0.0
4 haha you guys are a bunch of losers. 0.893617 0.021277 0.0 0.0 0.87234 0.021277 0.0
{% endraw %} {% raw %}
lbl_cols = list(toxic_df.columns[2:]); lbl_cols
['severe_toxicity',
 'obscene',
 'threat',
 'insult',
 'identity_attack',
 'sexual_explicit']
{% endraw %} {% raw %}
toxic_df = toxic_df.round({col: 0 for col in lbl_cols})
toxic_df = toxic_df.convert_dtypes()

toxic_df.head()
text toxicity severe_toxicity obscene threat insult identity_attack sexual_explicit
0 This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done! 0.000000 0 0 0 0 0 0
1 Thank you!! This would make my life a lot less anxiety-inducing. Keep it up, and don't let anyone get in your way! 0.000000 0 0 0 0 0 0
2 This is such an urgent design problem; kudos to you for taking it on. Very impressive! 0.000000 0 0 0 0 0 0
3 Is this something I'll be able to install on my site? When will you be releasing it? 0.000000 0 0 0 0 0 0
4 haha you guys are a bunch of losers. 0.893617 0 0 0 1 0 0
{% endraw %} {% raw %}
task = HF_TASKS_AUTO.SequenceClassification

pretrained_model_name = "roberta-base" # "distilbert-base-uncased" "bert-base-uncased"
config = AutoConfig.from_pretrained(pretrained_model_name)
config.num_labels = len(lbl_cols)

hf_arch, hf_config, hf_tokenizer, hf_model = BLURR_MODEL_HELPER.get_hf_objects(pretrained_model_name, 
                                                                               task=task, 
                                                                               config=config)
{% endraw %}

Note how we have to configure the num_labels to the number of labels we are predicting. Given that our labels are already encoded, we use a MultiCategoryBlock with encoded=True and vocab equal to the columns with our 1's and 0's.

{% raw %}
blocks = (
    HF_TextBlock(hf_arch=hf_arch, hf_tokenizer=hf_tokenizer), 
    MultiCategoryBlock(encoded=True, vocab=lbl_cols)
)

dblock = DataBlock(blocks=blocks, 
                   get_x=ColReader('text'), get_y=ColReader(lbl_cols), 
                   splitter=RandomSplitter())
{% endraw %} {% raw %}
dls = dblock.dataloaders(toxic_df, bs=4)
{% endraw %} {% raw %}
b = dls.one_batch()
len(b), b[0]['input_ids'].shape, b[1].shape
(2, torch.Size([4, 404]), torch.Size([4, 6]))
{% endraw %} {% raw %}
dls.show_batch(dataloaders=dls, max_n=2)
text None
0 As usual WW plumbing the depths for deeper meaning... that is unless it involves an issue on which they disagree then it is ridicule 24/7. Clever creating the Bundyland series complete with cartoon banner. Set the tone for the level of journalism to expect... journalism? ... fatastisticism. \n\nI did notice you soft pedaling the ridicule of David Fry identifying him as troubled. My guess is that has more to do with sympathy for his pot smoking withdrawl rants than respect for his politics. Respect is never a factor with liberals as evidenced by your series of vapid caricatures. \n\nDid you happen to see the stories actual journalists did on Refuge mis-managment, fires, floods, and the millions of Carp that are harassing the birds away from the Bird Refuge? The stories of arbitrary miss-management that are driving unemployment ever higher in eastern Oregon. Curry County Sheriff turning in his badge in frustration for lack of resources dud to dwindling tax base engineered by arbitrary over reaching Federal Government policies. Or how about the thousands of miles of roads proposed to be removed from Oregon wild lands, cutting off public and fire access? What is the one issue the people of Oregon demand universally? Access to the wild lands? You are not reporting that it is being taken away. More closed roads. More illegal Federal Police, guns drawn, stops... as reported by the Sheriff of Grant County. \n\nI suspect the real problem was real people you couldn't care less about were articulate, were actual victims, and made rational arguments you could not respond to. The exact opposite of the great unbathed OWS movement, apart from the paid organizers, that held downtown Portland hostage and trashed 3 park blocks for 3 weeks. \n\nShame on you. You missed some really great stories and even greater people... people who don't drink $5 cups of coffee.
1 The lack of 'freedom of speech' to which I referred was the insistence by professors that liberal positons were the 'Truth' incarnate and that anything less than left wing progressivism was anathema and received lower grades on exams. As far as unions, are concerned, there is a field of study over at the Duck U Law School called 'Labor Law' which provides one a lengthy history of the benefits and the downsides of unions in this country. I have to admit that part of my disagreement with unions in Oregon is based on my mandatory membership in SEIU in order to be a State employee. There I saw the union realities of today in the public sector first hand; not the halcyon days of fights against child labor, egregious working conditions etc. Now, the public unions are just different types of Corporations whose purpose is to protect the incompetent and extract further tax dollars to support their liberal agendas. At least real Corporations have to produce something positive to sell.
{% endraw %} {% raw %}
model = HF_BaseModelWrapper(hf_model)

learn = Learner(dls, 
                model,
                opt_func=partial(Adam),
                loss_func=BCEWithLogitsLossFlat(),
                metrics=[partial(accuracy_multi, thresh=0.2)],
                cbs=[HF_BaseModelCallback],
                splitter=hf_splitter)

learn.loss_func.thresh = 0.2
learn.create_opt()             # -> will create your layer groups based on your "splitter" function
learn.freeze()
{% endraw %}

Since we're doing multi-label classification, we adjust our loss function to use binary cross-entropy and our metrics to use the multi-label friendly version of accuracy.

{% raw %}
preds = model(b[0])
preds[0].shape
torch.Size([4, 6])
{% endraw %} {% raw %}
learn.lr_find(suggestions=True)
SuggestedLRs(lr_min=0.05248074531555176, lr_steep=0.0014454397605732083)
{% endraw %} {% raw %}
learn.fit_one_cycle(3, lr_max=3e-3)
epoch train_loss valid_loss accuracy_multi time
0 0.036332 0.042089 0.992657 02:55
1 0.044398 0.039479 0.992657 02:57
2 0.033817 0.037818 0.992657 02:57
{% endraw %} {% raw %}
learn.show_results(learner=learn, max_n=2)
text None target
0 Predatory patrol towing isn't a big subject, and there is no advocacy group that is paying any attention to it, but the City of Portland has completely backed off of enforcing state law where the towing predators are operating on private property, and this is Commissioner Novick's failure. He's in charge of towing.\n\nThe City has allowed Retriever Towing to operate in open violation of ADA for years at their NW Quimby lot, and there is absolutely no provision in city ordinance that takes into account when they tow a mobility-disabled person's vehicle. Wheelchair or no wheelchair, the person has to get themselves to the tow yard (which does not have wheelchair access).\n\nLed by Senator Avel Gordly, the legislature enacted important new citizen protections against predatory towing effective January 1, 2008, but there is no effective way for Portlanders to learn what their rights are other than what the predatory towers themselves tell you. A description of the protections has never appeared on the City website. The version posted there has never been correct.\n\nIt is only in the past two years the the City began enforcing the signage requirements, and the City Towing Coordinator worked with the towers to pass the costs onto the citizens. $ 20 of every tow bill goes to the towers to "compensate" them for their signs, with no upper limit, and your $ 20 is likely to be paying for signs completely unrelated to the place your vehicle was towed from.\n\nPortland towers are allowed to operate in open violation of state law and the City's own statutes, providing towing services to property owners for free. That is specifically prohibited by state law and City ordinance.\n\nSenator Gordly's legislation was intended to protect people living in apartment complexes, generally low-income, minority populations, and they have no one standing up for them in City Hall. I hope you are that champion. []
1 To blame Rubio is to find a scapegoat. Trump is a product of many angry, mostly white Americans who now believe their low pay is the result of immigrants, the Chinese, the Mexican government, with a dash of corporate influence. And let's not forget Muslims, who perpetrated what the media dubbed "the greatest terrorist attack since 9/11 - 14 people killed. No one should rejoice over deaths, the largest attack in 15 years killing 14 shows little threat.\nBefore Trump, Republicans riled over immigration, foreign threats and government control. Spawning a child that ate their mother, the Tea Party came to see Republicans as part of establishment politics. Compromise = bad. This message was spread throughout the land via hate-talk radio and hundreds of right wing wacko web sites (we take Visa) that still spew lies, i.e. our Muslim President, hate for liberals and RINOS. The Heritage Foundation and CATO institute helped craft these arguments\nThen came Trump who spoke "lies we can believe in". []
{% endraw %} {% raw %}
learn.loss_func.thresh = 0.02
{% endraw %} {% raw %}
comment = """
Those damned affluent white people should only eat their own food, like cod cakes and boiled potatoes. 
No enchiladas for them!
"""
learn.blurr_predict(comment)
((#1) ['severe_toxicity'],
 tensor([False, False, False,  True, False, False]),
 tensor([2.0519e-06, 5.5858e-03, 1.1516e-04, 2.7981e-02, 1.8549e-03, 1.1074e-03]))
{% endraw %}

Cleanup