--- title: text.modeling.language_modeling keywords: fastai sidebar: home_sidebar summary: "This module contains custom models, custom splitters, etc... for both causal and MLM language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus." description: "This module contains custom models, custom splitters, etc... for both causal and MLM language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus." nb_path: "nbs/12_text-modeling-language-modeling.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
 
{% endraw %} {% raw %}
{% endraw %} {% raw %}
What we're running with at the time this documentation was generated:
torch: 1.10.1+cu111
fastai: 2.5.6
transformers: 4.16.2
{% endraw %}

Setup

For this example, we'll use the WIKITEXT_TINY dataset available from fastai to demonstrate how to configure BLURR code for language modeling

{% raw %}
wiki_path = untar_data(URLs.WIKITEXT_TINY)

train_df = pd.read_csv(wiki_path / "train.csv", header=None)
valid_df = pd.read_csv(wiki_path / "test.csv", header=None)

train_df["is_valid"] = False
valid_df["is_valid"] = True

df = pd.concat([train_df, valid_df])

print(len(df))
df.head()
662
0 is_valid
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z... False
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re... False
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit... False
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch... False
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers .... False
{% endraw %}

Mid-level API

{% raw %}

class LMMetricsCallback[source]

LMMetricsCallback(**kwargs) :: Callback

A fastai friendly metric implemented as a callback so that we can handle use cases where we don't want to count tokens marked to be ignored or else not count batches where there are no targs

{% endraw %} {% raw %}
{% endraw %}

In this section, we'll add helpful metrics for calculating accuracy and perplexity for both causal and masked language modeling tasks.

Examples

Causal Language Modeling

In causal language modeling, we are attempting to predict the next token given those before it.

{% raw %}
model_cls = AutoModelForCausalLM

pretrained_model_name = "gpt2"
hf_arch, hf_config, hf_tokenizer, hf_model = NLP.get_hf_objects(pretrained_model_name, model_cls=model_cls)

if hf_tokenizer.pad_token is None:
    hf_tokenizer.pad_token = "[PAD]"
Using pad_token, but it is not set yet.
{% endraw %} {% raw %}
preprocessor = LMPreprocessor(hf_tokenizer, chunk_size=128, text_attr=0)
proc_df = preprocessor.process_df(train_df, valid_df)
{% endraw %} {% raw %}
bbtfm = LMBatchTokenizeTransform(hf_arch, hf_config, hf_tokenizer, hf_model, lm_strategy_cls=CausalLMStrategy)
blocks = (TextBlock(batch_tokenize_tfm=bbtfm, input_return_type=CausalLMTextInput), noop)

dblock = DataBlock(blocks=blocks, get_x=ColReader("proc_0"), splitter=ColSplitter(col="is_valid"))
{% endraw %} {% raw %}
dls = dblock.dataloaders(proc_df, bs=2)
{% endraw %} {% raw %}
b = dls.one_batch()
b[0]["input_ids"].shape, b[0]["labels"].shape, b[1].shape
(torch.Size([2, 129]), torch.Size([2, 129]), torch.Size([2, 129]))
{% endraw %} {% raw %}
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=500)
text target
0 ₹ 40 million ( US $ 590 @,@ 000 ) was spent solely on VFX for Magadheera. \n \n = = = <unk> = = = \n \n During the film's shoot at Ramoji Film City in late November 2008, a 500 square feet ( 46 m2 ) film can, containing two or three scenes, was discovered missing from Rainbow lab. The filmmakers filed a case at <unk> police station. Security personnel and film unit members searched, but failed to recover the reels. Rajamouli's unit said it was not important if the scenes from �� 40 million ( US $ 590 @,@ 000 ) was spent solely on VFX for Magadheera. \n \n = = = <unk> = = = \n \n During the film's shoot at Ramoji Film City in late November 2008, a 500 square feet ( 46 m2 ) film can, containing two or three scenes, was discovered missing from Rainbow lab. The filmmakers filed a case at <unk> police station. Security personnel and film unit members searched, but failed to recover the reels. Rajamouli's unit said it was not important if the scenes from
1 \n <|endoftext|> Wins, L \n = Losses, T = \n <|endoftext|> \n = New York State Route 164 = \n \n New York State Route 164 ( NY 164 ) is a short state highway located entirely in the town of Patterson in northeastern Putnam County, New York, in the United States. It is a short, two @-@ lane back road that does not pass through any major populated areas and serves primarily as a connector between NY 311 and NY 22. NY 164 also allows for faster passage from Interstate 84 ( I @-@ 84 ) to the Putnam Lak \n Wins, L \n = Losses, T = \n \n = New York State Route 164 = \n \n New York State Route 164 ( NY 164 ) is a short state highway located entirely in the town of Patterson in northeastern Putnam County, New York, in the United States. It is a short, two @-@ lane back road that does not pass through any major populated areas and serves primarily as a connector between NY 311 and NY 22. NY 164 also allows for faster passage from Interstate 84 ( I @-@ 84 ) to the Putnam Lake area via NY
{% endraw %}
Training
{% raw %}
model = BaseModelWrapper(hf_model)
fit_cbs = [LMMetricsCallback()]

learn = Learner(
    dls,
    model,
    opt_func=partial(Adam),
    loss_func=PreCalculatedCrossEntropyLoss(),
    cbs=[BaseModelCallback],
    metrics=[perplexity],
    splitter=blurr_splitter,
).to_fp16()

learn.freeze()
{% endraw %} {% raw %}
learn.summary()
{% endraw %} {% raw %}
# preds = learn.model(b[0])
# len(preds),preds[0], preds[1].shape
{% endraw %} {% raw %}
print(len(learn.opt.param_groups))
5
{% endraw %} {% raw %}
learn.lr_find(suggest_funcs=[minimum, steep, valley, slide])
SuggestedLRs(minimum=0.0033113110810518267, steep=2.0892961401841603e-05, valley=0.0014454397605732083, slide=0.0020892962347716093)
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=3e-3, cbs=fit_cbs)
epoch train_loss valid_loss perplexity lm_accuracy time
0 3.501945 3.165918 23.710508 0.417112 24:51
{% endraw %}
Showing results

Below we'll add in additional functionality to more intuitively show the results of our model.

{% raw %}
{% endraw %} {% raw %}
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 . \n The German government has said that it does not consider Scientology a religion, but a " commercial enterprise with a history of taking advantage of vulnerable individuals and an extreme dislike of any criticism " whose " totalitarian structure a \n The German government has said that it does not consider Scientology a religion, but a " commercial enterprise with a history of taking advantage of vulnerable individuals and an extreme dislike of any criticism " whose " totalitarian structure an <\n < < was been that the will not want the to threat, but that " cult " " a religious of religious advantage of the individuals " families interest form of the form of. " < " " political " be a threat to the "s national system ". , the government g
1 security concerns — has ensured that Australian strategic policy has often been defined by relations with its allies. Regardless, a tendency towards strategic <unk> has also been evident, with Australians often reluctant to think about defence issue concerns — has ensured that Australian strategic policy has often been defined by relations with its allies. Regardless, a tendency towards strategic <unk> has also been evident, with Australians often reluctant to think about defence issues or to < and. including been that the troops planners is been been based by the between the allies. The of the strong to a isolationunk> has been been evident in with Australia increasingly preferring to engage that the policy. the considerunk> their. they n
{% endraw %}
Prediction
{% raw %}
learn.blurr_generate("Blurr is fun to work with because", max_length=50, do_sample=True, top_k=25)
[{'generated_texts': ' Blurr is fun to work with because of the high probability of a problem solving approach , a trait common to many of the early problems of physics . " The concept of " \' abstract \' problems " may be the first to describe what is meant'}]
{% endraw %}

Masked Language Modeling

In masked language modeling (MLM), we are attempting to predict the masked tokens. In Blurr, these are encapsulated by classes implementing the BaseLMStrategy base class.

For a list of some of the more common strategies, see table 3 of the Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer paper. When fine-tuning a MLM model. you'll want to make sure you use the same approach as the model authors should you be looking to reproduce their results ... but our approach here makes it easy to play with different strategies regardless.

In the example below, we'll tell Blurr we want to use the BERT-style masking strategy.

{% raw %}
model_cls = AutoModelForMaskedLM

pretrained_model_name = "distilroberta-base"
hf_arch, hf_config, hf_tokenizer, hf_model = NLP.get_hf_objects(pretrained_model_name, model_cls=model_cls)

if hf_tokenizer.pad_token is None:
    hf_tokenizer.pad_token = "[PAD]"
{% endraw %} {% raw %}
preprocessor = LMPreprocessor(hf_tokenizer, chunk_size=128, text_attr=0)
proc_df = preprocessor.process_df(train_df, valid_df)
{% endraw %} {% raw %}
bbtfm = LMBatchTokenizeTransform(hf_arch, hf_config, hf_tokenizer, hf_model, lm_strategy_cls=BertMLMStrategy)
blocks = (TextBlock(batch_tokenize_tfm=bbtfm, input_return_type=MLMTextInput), noop)

dblock = DataBlock(blocks=blocks, get_x=ColReader("proc_0"), splitter=ColSplitter(col="is_valid"))
{% endraw %} {% raw %}
dls = dblock.dataloaders(proc_df, bs=2)
{% endraw %} {% raw %}
b = dls.one_batch()
b[0]["input_ids"].shape, b[0]["labels"].shape, b[1].shape
(torch.Size([2, 130]), torch.Size([2, 130]), torch.Size([2, 130]))
{% endraw %} {% raw %}
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=250)
text target
0 <mask> � � u — <mask> to <mask> retire officers with more than 25 years [ of] service , as they thought them to be and ineffective , [ Oliv] most importantly , rivals for power [.] Most of <mask> older officers had more experience under the Vietnamese National Army <mask> the French colonial era , and some of the younger men saw them as too detached from the modern situation . The Young Turks had quite a <mask> of influence over Kh <mask> h , as Th i and K � � � had intervened milit arily to save <mask> from a <mask> attempt in September by <mask> als V [�] � n <mask> <mask> <mask> � � � � <mask> V <mask> � [ �] � � u — [ wanted] to [ forcibly] retire officers with more than 25 years [ of] service , as they thought them to be and ineffective , [ but] most importantly , rivals for power [.] Most of [ the] older officers had more experience under the Vietnamese National Army [ during] the French colonial era , and some of the younger men saw them as too detached from the modern situation . The Young Turks had quite a [ lot] of influence over Kh [án] h , as Th i and K � � � had intervened milit arily to save [ him] from a [ coup] attempt in September by [ Gener] als V [�] � n [ ] [ and] [ D] � � � � [ng] V [�] �
1 <mask> \n During the <mask> 1990 s , <mask> major <mask> were disbanded and a <mask> of equipment [ was] phased out or scrapped <mask> to a severe shortage of funds . The whole land forces structure <mask> re organized <mask> <mask> into territorial [ corps] , and from [ reg] iments into batt alions . In the mid @ - @ 1990 s , the situation of the <mask> forces was critical : [ FM] military budget <mask> three times lower than in 1989 ( million dollars ) , 50 % of the equipment was older than 30 years , and 60 % of the armoured vehicles and 85 % of <mask> missile units were non <mask> - @ operational <mask> Due <mask> lack of [ ] \n During the [ early] 1990 s , [ some] major [ units] were disbanded and a [ lot] of equipment [ was] phased out or scrapped [ due] to a severe shortage of funds . The whole land forces structure [ was] re organized [ from] [ armies] into territorial [ corps] , and from [ reg] iments into batt alions . In the mid @ - @ 1990 s , the situation of the [ land] forces was critical : [ the] military budget [ was] three times lower than in 1989 ( million dollars ) , 50 % of the equipment was older than 30 years , and 60 % of the armoured vehicles and 85 % of [ the] missile units were non [ @] - @ operational [.] Due [ to] lack of
{% endraw %}
Training
{% raw %}
model = BaseModelWrapper(hf_model)
fit_cbs = [LMMetricsCallback()]

learn = Learner(
    dls,
    model,
    opt_func=partial(Adam, decouple_wd=True),
    loss_func=PreCalculatedCrossEntropyLoss(),
    cbs=[BaseModelCallback],
    metrics=[perplexity],
    splitter=blurr_splitter,
).to_fp16()

learn.freeze()
{% endraw %} {% raw %}
learn.summary()
{% endraw %} {% raw %}
print(len(learn.opt.param_groups))
3
{% endraw %} {% raw %}
learn.lr_find(suggest_funcs=[minimum, steep, valley, slide])
SuggestedLRs(minimum=0.0005248074419796466, steep=1.3182567499825382e-06, valley=0.0003311311302240938, slide=0.0004786300996784121)
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=1e-4, cbs=fit_cbs)
epoch train_loss valid_loss perplexity lm_accuracy time
0 1.898339 1.804883 6.079262 0.642062 44:21
{% endraw %} {% raw %}
{% endraw %}
Showing results

Below we'll add in additional functionality to more intuitively show the results of our model.

{% raw %}
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 <mask> when the sett ler ' s presence led to competition over resources <mask> and to the occupation of the [ indigenous] inhabitants ' lands . European diseases dec imated Aboriginal populations , and the occupation <mask> <mask> of lands <mask> food resources sometimes <mask> to starvation . <mask> and large neither the British nor <mask> approached the conflict in an organised sense and conflict <mask> between groups of <mask> and individual tribes rather <mask> systematic warfare . At times <mask> however , the frontier wars did see the involvement of British soldiers and [ Roe] mounted police [ units] . Not all Aboriginal <mask> resisted white [ encro] achment on their lands , while many <mask> <mask> mounted police units and were involved in [ hostile] when the sett ler ' s presence led to competition over resources [,] and to the occupation of the [ indigenous] inhabitants ' lands . European diseases dec imated Aboriginal populations , and the occupation [ or] [ destruction] of lands [ and] food resources sometimes [ led] to starvation . [ By] and large neither the British nor [ the] approached the conflict in an organised sense and conflict [ occurred] between groups of [ settlers] and individual tribes rather [ than] systematic warfare . At times [,] however , the frontier wars did see the involvement of British soldiers and [ later] mounted police [ units] . Not all Aboriginal [ groups] resisted white [ encro] achment on their lands , while many [ served] [ in] mounted police units and were involved in [,] when the sett ler ' s presence led to competition over resources [,] and to the occupation of the [ indigenous] inhabitants ' lands . European diseases dec imated Aboriginal populations , and the occupation [ of] [ control] of lands [ and] food resources sometimes [ led] to starvation . [ By] and large neither the British nor [ the] approached the conflict in an organised sense and conflict [ was] between groups of [ settlers] and individual tribes rather [ than] systematic warfare . At times [,] however , the frontier wars did see the involvement of British soldiers and [ mounted] mounted police [ units] . Not all Aboriginal [ Australians] resisted white [ encro] achment on their lands , while many [ were] [ mounted] mounted police units and were involved in
1 , at the junction of the <mask> and the <mask> , which included [ a] heavy Japanese artillery bombardment . Australian patrols were resumed in late July and continued into August <mask> These attacks proved very costly , [Fil] amongst the Australian engineers that <mask> <mask> with rebuilding the bridges and roads that had been destroyed in <mask> flooding . \n [ Fighting] <mask> the northern <mask> continued during <mask> time also , and although preparations in the south for the final <mask> <mask> continued into August , combat operations on the <mask> ceased <mask> the <mask> [ came] to an end before these were completed . As a result , the final Australian operations on <mask> place , at the junction of the [ River] and the [ Road] , which included [ a] heavy Japanese artillery bombardment . Australian patrols were resumed in late July and continued into August [.] These attacks proved very costly , [ particularly] amongst the Australian engineers that [ were] [ tasked] with rebuilding the bridges and roads that had been destroyed in [ the] flooding . \n [ Fighting] [ in] the northern [ sector] continued during [ this] time also , and although preparations in the south for the final [ advance] [ towards] continued into August , combat operations on the [ island] ceased [ as] the [ war] [ came] to an end before these were completed . As a result , the final Australian operations on [ took] place , at the junction of the [ River] and the [ River] , which included [ a] heavy Japanese artillery bombardment . Australian patrols were resumed in late July and continued into August [.] These attacks proved very costly , [ particularly] amongst the Australian engineers that [ were] [ tasked] with rebuilding the bridges and roads that had been destroyed in [ the] flooding . \n [ Fighting] [ on] the northern [ beaches] continued during [ this] time also , and although preparations in the south for the final [ battles] [ on] continued into August , combat operations on the [ mainland] ceased [ and] the [ attacks] [ came] to an end before these were completed . As a result , the final Australian operations on [ took] place
{% endraw %}
Prediction

While Learner.blurr_generate will work well for causal LMs designed for text generation, it won't for MLM models designed to predict masked tokens. To accomodate the later, we add Learner.blurr_fill_mask ...

{% raw %}

Learner.blurr_fill_mask[source]

Learner.blurr_fill_mask(inp:Union[typing.List[int], str], n_preds:int=1, **kwargs)

For MLM models

Type Default Details
inp typing.Union[typing.List[int], str] Your input_ids or raw text string with a hf_tokenizer.mask_token
n_preds int 1 The number of predictions you want to return for the [MASK]ed token
kwargs No Content
{% endraw %} {% raw %}
{% endraw %} {% raw %}
learn.blurr_fill_mask(f"The best place on earth is {hf_tokenizer.mask_token}.", n_preds=5)
['The best place on earth is here.',
 'The best place on earth is America.',
 'The best place on earth is Antarctica.',
 'The best place on earth is mine.',
 'The best place on earth is there.']
{% endraw %}

High-level API

{% raw %}

class BlearnerForLM[source]

BlearnerForLM(dls:DataLoaders, hf_model:PreTrainedModel, base_model_cb:BaseModelCallback=BaseModelCallback, loss_func=None, opt_func=Adam, lr=0.001, splitter=trainable_params, cbs=None, metrics=None, path=None, model_dir='models', wd=None, wd_bn_bias=False, train_bn=True, moms=(0.95, 0.85, 0.95)) :: Blearner

Group together a model, some dls and a loss_func to handle training

{% endraw %} {% raw %}
{% endraw %}

We can use the BlearnerForLM for either Causal or Masked language models. With one line of code, we get our DataBlock, DataLoaders, and Blearner with sensible defaults and ready for training

Example

Causal language modeling

{% raw %}
learn = BlearnerForLM.from_data(df, "gpt2", text_attr=0, dl_kwargs={"bs": 2}).to_fp16()
Using pad_token, but it is not set yet.
{% endraw %} {% raw %}
learn.dls.show_batch(dataloaders=learn.dls, max_n=2, trunc_at=500)
text target
0 \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They A \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They Ar
1 \n = Ireland = \n \n Ireland ( / <unk> / ; Irish : <unk> [ <unk> ] ; Ulster @-@ Scots : <unk> [ <unk> ] ) is an island in the North Atlantic. It is separated from Great Britain to its east by the North Channel, the Irish Sea, and St George's Channel. Ireland is the second @-@ largest island of the British Isles, the third @-@ largest in Europe, and the twentieth @-@ largest on Earth. \n <unk>, Ireland is divided between the Republic of Ireland ( officially named Ireland ), which covers five @-@ <un \n = Ireland = \n \n Ireland ( / <unk> / ; Irish : <unk> [ <unk> ] ; Ulster @-@ Scots : <unk> [ <unk> ] ) is an island in the North Atlantic. It is separated from Great Britain to its east by the North Channel, the Irish Sea, and St George's Channel. Ireland is the second @-@ largest island of the British Isles, the third @-@ largest in Europe, and the twentieth @-@ largest on Earth. \n <unk>, Ireland is divided between the Republic of Ireland ( officially named Ireland ), which covers five @-@ <unk
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=3e-3, cbs=[BlearnerForLM.get_metrics_cb()])
epoch train_loss valid_loss perplexity lm_accuracy time
0 3.519682 3.112483 22.476774 0.427281 02:26
{% endraw %} {% raw %}
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 \n = Military history of Australia = \n \n The military history of Australia spans the nation's 220 @-@ year modern history, from the early Australian frontier wars between <unk> and Europeans to the ongoing conflicts in Iraq and Afghanistan in the ear \n = Military history of Australia = \n \n The military history of Australia spans the nation's 220 @-@ year modern history, from the early Australian frontier wars between <unk> and Europeans to the ongoing conflicts in Iraq and Afghanistan in the earl \n\n =\n = the\n\n\n =\n = Australian history of Australia is the period froms history years 1@ period period era. including the early days colonies to to the 18> Australia < < the early conflict between the and Afghanistan. the 1980 1980st century. The the
1 \n = Air Rhodesia Flight <unk> = \n \n Air Rhodesia Flight <unk> was a scheduled passenger flight that was shot down by the Zimbabwe People's Revolutionary Army ( <unk> ) on 3 September 1978, during the Rhodesian Bush War. The aircraft involved, a Vick \n = Air Rhodesia Flight <unk> = \n \n Air Rhodesia Flight <unk> was a scheduled passenger flight that was shot down by the Zimbabwe People's Revolutionary Army ( <unk> ) on 3 September 1978, during the Rhodesian Bush War. The aircraft involved, a Vicke \n\n =planes\n\n\n\n>\n Air\n =\n = Rhodesia Flight <unk> = a flight flight flight from was scheduled down by a Sovietan'ss Liberation Army (Zunk> ) on September August 1944. killing a liberationian civil War. The plane was in was civilian- Airickersount,'Air
{% endraw %} {% raw %}
learn.blurr_generate("Blurr is fun to work with because", max_length=50, do_sample=True, top_k=25)
[{'generated_texts': ' Blurr is fun to work with because you be able to find easy to work with the toolkit as well as your own own personal tools.\n\nIf you want to take the time to you need to know that the'}]
{% endraw %}

Masked language modeling

{% raw %}
learn = BlearnerForLM.from_data(df, "bert-base-cased", lm_strategy_cls=BertMLMStrategy, text_attr=0, dl_kwargs={"bs": 2}).to_fp16()
{% endraw %} {% raw %}
learn.dls.show_batch(dataloaders=learn.dls, max_n=2, trunc_at=250)
text target
0 = Bob [MASK] = Bob Dylan ( / < un ##k > / ; born Robert Allen [MASK] ##immer [MASK] , May 24 [##ott] 1941 ) is an American singer @ - @ songwriter , artist and writer . He has been influential in popular [MASK] and culture for more than five decades . Much of his most [MASK] work dates from the 1960s when his songs chronicle ##d social unrest [MASK] [Freiburg] Dylan [Chrysler] ##pu ##dia ##ted suggestions from journalists that he [MASK] a [MASK] [MASK] his generation . Nevertheless , early songs such [MASK] " [MASK] ##low ##in ' in the Wind " [MASK] [MASK] The Times They Are a [MASK] - @ < un ##k > ' " [MASK] anthem [IS] for the American civil rights [MASK] [anti] @ - [MASK] war movements [MASK] After he left his initial base in the American folk music revival , his six @ [MASK] @ minute single [MASK] Like a [MASK] [MASK] " altered the range of popular [MASK] in [MASK] . His mid @ - @ 1960s [MASK] , backed by rock musicians , reached [MASK] [MASK] end of the United States music charts while also attracting < un ##k > and criticism from others in [MASK] folk [MASK] [MASK] Dylan ' s lyrics [have] [MASK] various political , [MASK] , philosophical , and literary [MASK] . They def ##ied existing pop [MASK] conventions [MASK] appealed to the b ##urge ##oning counter [MASK] [tugging] Initially inspired by the performances = Bob [Dylan] = Bob Dylan ( / < un ##k > / ; born Robert Allen [Z] ##immer [##man] , May 24 [,] 1941 ) is an American singer @ - @ songwriter , artist and writer . He has been influential in popular [music] and culture for more than five decades . Much of his most [celebrated] work dates from the 1960s when his songs chronicle ##d social unrest [,] [although] Dylan [re] ##pu ##dia ##ted suggestions from journalists that he [was] a [spokesman] [for] his generation . Nevertheless , early songs such [as] " [B] ##low ##in ' in the Wind " [and] ["] The Times They Are a [@] - @ < un ##k > ' " [became] anthem [##s] for the American civil rights [and] [anti] @ - [@] war movements [.] After he left his initial base in the American folk music revival , his six @ [-] @ minute single ["] Like a [Rolling] [Stone] " altered the range of popular [music] in [1965] . His mid @ - @ 1960s [recordings] , backed by rock musicians , reached [the] [top] end of the United States music charts while also attracting < un ##k > and criticism from others in [the] folk [movement] [.] Dylan ' s lyrics [have] [incorporated] various political , [social] , philosophical , and literary [influences] . They def ##ied existing pop [music] conventions [and] appealed to the b ##urge ##oning counter [##culture] [.] Initially inspired by the performances
1 = Mortimer Wheeler = Sir Robert [MASK] Mortimer [MASK] CH , < un ##k > , MC [MASK] < un ##k > , < [##and] ##k > , F [Hopkins] , [MASK] un ##k > ( 10 September 1890 – [MASK] July 1976 ) was a British [MASK] and officer in the British Army . [MASK] [MASK] course [MASK] his career , he served as [MASK] of both the National Museum of Wales and London Museum , [Director] @ - @ General of the Archaeological Survey of India , and the founder and Honorary Director of the [MASK] of [MASK] [MASK] [MASK] , further writing [MASK] @ - @ [four] books on archaeological subjects . Born in Glasgow to a middle [@] - @ class family , Wheeler was [Divinity] largely in Yorkshire before re ##locating to [MASK] in [##SC] [Maurice] years . After studying Classics [MASK] [MASK] College London ( UC ##L ) [MASK] he [began] working professionally in archaeology [pendant] [MASK] in the Romano @ - @ British period . During World War I he volunteered for [MASK] in [MASK] [MASK] Artillery , [MASK] stationed on the [MASK] Front , where he rose to the [MASK] of major and was awarded the Military Cross . Returning to [MASK] , he obtained his doctorate from UC ##L before taking on [MASK] [MASK] at the National Museum of Wales [MASK] [MASK] as Keeper [MASK] Archaeology and then [MASK] Director , during which time he oversaw [MASK] [MASK] the Roman forts = Mortimer Wheeler = Sir Robert [Eric] Mortimer [Wheeler] CH , < un ##k > , MC [,] < un ##k > , < [un] ##k > , F [##RS] , [<] un ##k > ( 10 September 1890 – [22] July 1976 ) was a British [archaeologist] and officer in the British Army . [Over] [the] course [of] his career , he served as [Director] of both the National Museum of Wales and London Museum , [Director] @ - @ General of the Archaeological Survey of India , and the founder and Honorary Director of the [Institute] of [Archaeology] [in] [London] , further writing [twenty] @ - @ [four] books on archaeological subjects . Born in Glasgow to a middle [@] - @ class family , Wheeler was [raised] largely in Yorkshire before re ##locating to [London] in [his] [teenage] years . After studying Classics [at] [University] College London ( UC ##L ) [,] he [began] working professionally in archaeology [,] [specializing] in the Romano @ - @ British period . During World War I he volunteered for [service] in [the] [Royal] Artillery , [being] stationed on the [Western] Front , where he rose to the [rank] of major and was awarded the Military Cross . Returning to [Britain] , he obtained his doctorate from UC ##L before taking on [a] [position] at the National Museum of Wales [,] [first] as Keeper [of] Archaeology and then [as] Director , during which time he oversaw [excavation] [at] the Roman forts
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=6e-4, cbs=[BlearnerForLM.get_metrics_cb()])
epoch train_loss valid_loss perplexity lm_accuracy time
0 2.644017 2.414341 11.182396 0.575588 01:11
{% endraw %} {% raw %}
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 = Military [MASK] of Australia = The military history of Australia spans [MASK] nation ' [MASK] 220 @ - [MASK] year modern history [cliffs] from the [MASK] [MASK] frontier wars between < un ##k > and Europeans to the ongoing conflicts [MASK] Iraq and Afghanistan in the early 21st [MASK] . Although this history is short [when] compared to that of many [MASK] nations , Australia has been involved in numerous conflicts [MASK] wars , and [MASK] and military service have been significant influences on Australian society and national identity [MASK] including the An [MASK] [MASK] . The relationship [MASK] war and Australian society has also been shaped by [MASK] enduring [MASK] of Australian [MASK] culture and its unique [MASK] di ##lemma . As British < un ##k > , the Australian colonies participated in Britain ' s small wars of [supporter] 19th century , while later as a < un [MASK] > do ##mini ##on , and [MASK] an independent nation , Australia fought in the First World War and Second World War [MASK] [as] well as in the [wars] in [MASK] , Malaya , Borneo and Vietnam during the [MASK] War . In the [MASK] @ - @ Vietnam era Australian forces have been involved in numerous [MASK] peace ##keeping missions , through [the] United Nations and other agencies , including in the Sinai , Persian Gulf , < [un] ##k [MASK] [MASK] Somalia , East [MASK] and the Solomon Islands , [MASK] [MASK] recently they have also = Military [history] of Australia = The military history of Australia spans [the] nation ' [s] 220 @ - [@] year modern history [,] from the [early] [Australian] frontier wars between < un ##k > and Europeans to the ongoing conflicts [in] Iraq and Afghanistan in the early 21st [century] . Although this history is short [when] compared to that of many [other] nations , Australia has been involved in numerous conflicts [and] wars , and [war] and military service have been significant influences on Australian society and national identity [,] including the An [##zac] [spirit] . The relationship [between] war and Australian society has also been shaped by [the] enduring [themes] of Australian [strategic] culture and its unique [security] di ##lemma . As British < un ##k > , the Australian colonies participated in Britain ' s small wars of [the] 19th century , while later as a < un [##k] > do ##mini ##on , and [then] an independent nation , Australia fought in the First World War and Second World War [,] [as] well as in the [wars] in [Korea] , Malaya , Borneo and Vietnam during the [Cold] War . In the [Post] @ - @ Vietnam era Australian forces have been involved in numerous [international] peace ##keeping missions , through [the] United Nations and other agencies , including in the Sinai , Persian Gulf , < [un] ##k [>] [,] Somalia , East [Timor] and the Solomon Islands , [while] [more] recently they have also = Military [history] of Australia = The military history of Australia spans [the] nation ' [s] 220 @ - [@] year modern history [,] from the [pre] [and] frontier wars between < un ##k > and Europeans to the ongoing conflicts [in] Iraq and Afghanistan in the early 21st [century] . Although this history is short [when] compared to that of many [other] nations , Australia has been involved in numerous conflicts [and] wars , and [political] and military service have been significant influences on Australian society and national identity [,] including the An [##zac] [era] . The relationship [between] war and Australian society has also been shaped by [the] enduring [nature] of Australian [military] culture and its unique [political] di ##lemma . As British < un ##k > , the Australian colonies participated in Britain ' s small wars of [the] 19th century , while later as a < un [##k] > do ##mini ##on , and [as] an independent nation , Australia fought in the First World War and Second World War [,] [as] well as in the [wars] in [Afghanistan] , Malaya , Borneo and Vietnam during the [Vietnam] War . In the [220] @ - @ Vietnam era Australian forces have been involved in numerous [international] peace ##keeping missions , through [the] United Nations and other agencies , including in the Sinai , Persian Gulf , < [un] ##k [>] [,] Somalia , East [Timor] and the Solomon Islands , [and] [more] recently they have also
1 = [Air] [MASK] Flight < un ##k > = Air Rhodesia [Flight] < un ##k > was a scheduled passenger flight [PA] was [MASK] [MASK] by the Zimbabwe People ' s Revolutionary [Army] ( [MASK] [MASK] ##k > ) on 3 September 1978 , during the Rhodesia [MASK] Bush [MASK] . The aircraft involved , a Vickers [MASK] named the [<] un [MASK] > , was flying the last leg of Air Rhodesia [MASK] s regular [MASK] service from Victoria Falls to the [MASK] Salisbury , via the resort [MASK] of < un ##k > . Soon after Flight [MASK] un [##k] > [MASK] off , a group of < un ##k > guerrilla ##s scored a direct hit on its star ##board wing [with] a Soviet @ - @ made < un [MASK] > [Sounds] [MASK] @ [MASK] [MASK] [MASK] @ - @ air infrared < un ##k [>] missile , [may] [MASK] [MASK] aircraft and forcing an emergency landing . An attempted belly landing in a cotton field just west of < un ##k > was < un ##k > by [MASK] unseen ditch , which caused the plane to cart ##wheel and break [MASK] [MASK] [MASK] the 52 [MASK] and four crew , 38 [MASK] in this crash ; the insurgents then approached the wreckage , rounded up the 10 survivors they could see and massacre ##d them with automatic gunfire . Three passengers survived by [hiding] in the surrounding bush , while a further five lived = [Air] [Rhodesia] Flight < un ##k > = Air Rhodesia [Flight] < un ##k > was a scheduled passenger flight [that] was [shot] [down] by the Zimbabwe People ' s Revolutionary [Army] ( [<] [un] ##k > ) on 3 September 1978 , during the Rhodesia [##n] Bush [War] . The aircraft involved , a Vickers [Viscount] named the [<] un [##k] > , was flying the last leg of Air Rhodesia ['] s regular [scheduled] service from Victoria Falls to the [capital] Salisbury , via the resort [town] of < un ##k > . Soon after Flight [<] un [##k] > [took] off , a group of < un ##k > guerrilla ##s scored a direct hit on its star ##board wing [with] a Soviet @ - @ made < un [##k] > [2] [surface] @ [-] [@] [to] @ - @ air infrared < un ##k [>] missile , [critically] [damaging] [the] aircraft and forcing an emergency landing . An attempted belly landing in a cotton field just west of < un ##k > was < un ##k > by [an] unseen ditch , which caused the plane to cart ##wheel and break [up] [.] [Of] the 52 [passengers] and four crew , 38 [died] in this crash ; the insurgents then approached the wreckage , rounded up the 10 survivors they could see and massacre ##d them with automatic gunfire . Three passengers survived by [hiding] in the surrounding bush , while a further five lived = [Air] [Rhodesia] Flight < un ##k > = Air Rhodesia [Flight] < un ##k > was a scheduled passenger flight [that] was [operated] [operated] by the Zimbabwe People ' s Revolutionary [Army] ( [<] [un] ##k > ) on 3 September 1978 , during the Rhodesia [##n] Bush [War] . The aircraft involved , a Vickers [Viscount] named the [<] un [##k] > , was flying the last leg of Air Rhodesia ['] s regular [scheduled] service from Victoria Falls to the [capital] Salisbury , via the resort [town] of < un ##k > . Soon after Flight [<] un [##k] > [took] off , a group of < un ##k > guerrilla ##s scored a direct hit on its star ##board wing [with] a Soviet @ - @ made < un [##k] > [@] [of] @ [-] [-] [>] @ - @ air infrared < un ##k [>] missile , [which] [destroying] [the] aircraft and forcing an emergency landing . An attempted belly landing in a cotton field just west of < un ##k > was < un ##k > by [an] unseen ditch , which caused the plane to cart ##wheel and break [.] [.] [killed] the 52 [passengers] and four crew , 38 [injured] in this crash ; the insurgents then approached the wreckage , rounded up the 10 survivors they could see and massacre ##d them with automatic gunfire . Three passengers survived by [hiding] in the surrounding bush , while a further five lived
{% endraw %} {% raw %}
batch_tfm = first_blurr_tfm(learn.dls)
{% endraw %} {% raw %}
learn.blurr_fill_mask(f"The best place on earth is {batch_tfm.hf_tokenizer.mask_token}.", n_preds=5)
['The best place on earth is here.',
 'The best place on earth is heaven.',
 'The best place on earth is Earth.',
 'The best place on earth is paradise.',
 'The best place on earth is Egypt.']
{% endraw %}