--- title: text.modeling.question_answering keywords: fastai sidebar: home_sidebar summary: "This module contains custom models, loss functions, custom splitters, etc... for question answering tasks" description: "This module contains custom models, loss functions, custom splitters, etc... for question answering tasks" nb_path: "nbs/14_text-modeling-question-answering.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
 
{% endraw %} {% raw %}
{% endraw %} {% raw %}
What we're running with at the time this documentation was generated:
torch: 1.10.1+cu111
fastai: 2.5.6
transformers: 4.16.2
{% endraw %}

Setup

We'll use a subset of squad_v2 to demonstrate how to configure your blurr code for training extractive question answering models. See the data.question_answering module if any of this setting up of the squad_df below looks unfamiliar to you.

{% raw %}
raw_datasets = load_dataset("squad", split=["train[:1000]", "validation[:200]"])

raw_train_df = pd.DataFrame(raw_datasets[0])
raw_valid_df = pd.DataFrame(raw_datasets[1])

raw_train_df["is_valid"] = False
raw_valid_df["is_valid"] = True

squad_df = pd.concat([raw_train_df, raw_valid_df])

squad_df["ans_start_char_idx"] = squad_df.answers.apply(lambda v: v["answer_start"][0] if len(v["answer_start"]) > 0 else "0")
squad_df["answer_text"] = squad_df.answers.apply(lambda v: v["text"][0] if len(v["text"]) > 0 else "")
squad_df["ans_end_char_idx"] = squad_df["ans_start_char_idx"].astype(int) + squad_df["answer_text"].str.len()

print(len(squad_df))
Reusing dataset squad (/home/wgilliam/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
1200
{% endraw %} {% raw %}
model_cls = AutoModelForQuestionAnswering

pretrained_model_name = "bert-large-uncased-whole-word-masking-finetuned-squad"
hf_arch, hf_config, hf_tokenizer, hf_model = NLP.get_hf_objects(pretrained_model_name, model_cls=model_cls)

max_seq_len = 128
vocab = dict(enumerate(range(max_seq_len)))
{% endraw %} {% raw %}
preprocessor = QAPreprocessor(
    hf_tokenizer, id_attr="id", tok_kwargs={"return_overflowing_tokens": True, "max_length": max_seq_len, "stride": 64}
)

proc_df = preprocessor.process_df(squad_df)
proc_df.head(1)
id title context question answers is_valid ans_start_char_idx answer_text ans_end_char_idx proc_question proc_context ans_start_token_idx ans_end_token_idx is_answerable
0 5733be284776f41900661182 University_of_Notre_Dame Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct... To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]} False 515 Saint Bernadette Soubirous 541 To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputed 0 0 False
{% endraw %} {% raw %}
before_batch_tfm = QABatchTokenizeTransform(hf_arch, hf_config, hf_tokenizer, hf_model, max_length=max_seq_len)

blocks = (
    TextBlock(batch_tokenize_tfm=before_batch_tfm, input_return_type=QATextInput),
    CategoryBlock(vocab=vocab),
    CategoryBlock(vocab=vocab),
)

# since its preprocessed, we include an "text" key with the values of our question and context
def get_x(item):
    return {"text": (item.proc_question, item.proc_context), "id": item.id}


dblock = DataBlock(
    blocks=blocks,
    get_x=get_x,
    get_y=[ItemGetter("ans_start_token_idx"), ItemGetter("ans_end_token_idx")],
    splitter=ColSplitter(),
    n_inp=1,
)
{% endraw %} {% raw %}
dls = dblock.dataloaders(proc_df, bs=4)
{% endraw %} {% raw %}
len(dls.vocab), dls.vocab[0], dls.vocab[1]
(2,
 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127],
 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127])
{% endraw %} {% raw %}
dls.valid.show_batch(dataloaders=dls, max_n=2)
text found start/end answer
0 which nfl team represented the afc at super bowl 50? super bowl 50 was an american football game to determine the champion of the national football league ( nfl ) for the 2015 season. the american football conference ( afc ) champion denver broncos defeated the national football conference ( nfc ) champion carolina panthers 24 – 10 to earn their third super bowl title. the game was played on february 7, 2016, at levi's stadium in the san francisco bay area at santa clara, california. as this was the 50th super bowl, the league emphasized the " golden anniversary " with various gold - themed initiatives, as well as temporarily suspending the True (46, 48) denver broncos
1 which nfl team represented the afc at super bowl 50? earn their third super bowl title. the game was played on february 7, 2016, at levi's stadium in the san francisco bay area at santa clara, california. as this was the 50th super bowl, the league emphasized the " golden anniversary " with various gold - themed initiatives, as well as temporarily suspending the tradition of naming each super bowl game with roman numerals ( under which the game would have been known as " super bowl l " ), so that the logo could prominently feature the arabic numerals 50. False (0, 0)
{% endraw %}

Mid-level API

{% raw %}

class QAModelCallback[source]

QAModelCallback(after_create=None, before_fit=None, before_epoch=None, before_train=None, before_batch=None, after_pred=None, after_loss=None, before_backward=None, before_step=None, after_cancel_step=None, after_step=None, after_cancel_batch=None, after_batch=None, after_cancel_train=None, after_train=None, before_validate=None, after_cancel_validate=None, after_validate=None, after_cancel_epoch=None, after_epoch=None, after_cancel_fit=None, after_fit=None) :: BaseModelCallback

The prediction is a combination start/end logits

{% endraw %} {% raw %}
{% endraw %}

Here we create a question/answer specific subclass of BaseModelCallback in order to get all the start and end prediction.

{% raw %}

class QAMetricsCallback[source]

QAMetricsCallback(compute_metrics_func, validation_ds, qa_metrics=['exact_match', 'f1'], **kwargs) :: Callback

Basic class handling tweaks of the training loop by changing a Learner in various events

{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}

compute_qa_metrics[source]

compute_qa_metrics(results, dataset, hf_tokenizer, tok_kwargs, id_attr='id', n_best=20)

{% endraw %} {% raw %}

class PreCalculatedQALoss[source]

PreCalculatedQALoss(*args, axis=-1, **kwargs) :: PreCalculatedLoss

If you want to let your Hugging Face model calculate the loss for you, make sure you include the labels argument in your inputs and use PreCalculatedLoss as your loss function. Even though we don't really need a loss function per se, we have to provide a custom loss class/function for fastai to function properly (e.g. one with a decodes and activation methods). Why? Because these methods will get called in methods like show_results to get the actual predictions.

Note: The Hugging Face models will always calculate the loss for you if you pass a labels dictionary along with your other inputs (so only include it if that is what you intend to happen)

{% endraw %} {% raw %}
{% endraw %}

Hugging Face question answering models will calculate the loss for you when you include both the start_positions and end_positions in the inputs dictionary. This is done by the QABatchTokenizeTransform when include_labels = True (which is the default). This also requires fastai developers to set their Learner's loss function to the PreCalculatedQALoss for training to work properly.

Example

Notice below how I had to define the loss function after creating the Learner object. I'm not sure why, but the MultiTargetLoss above prohibits the learner from being exported if I do.

Training

{% raw %}
model = BaseModelWrapper(hf_model)
learn_cbs = [QAModelCallback]

validation_ds = proc_df[proc_df.is_valid == True].to_dict(orient="records")
fit_cbs = [QAMetricsCallback(compute_metrics_func=compute_qa_metrics, validation_ds=validation_ds)]

learn = Learner(dls, model, opt_func=partial(Adam), cbs=learn_cbs, splitter=blurr_splitter)

learn.loss_func = PreCalculatedQALoss()  # MultiTargetLoss()
learn.create_opt()  # -> will create your layer groups based on your "splitter" function
learn.freeze()
{% endraw %} {% raw %}
# learn.summary() # no glory here :(
{% endraw %} {% raw %}
print(len(learn.opt.param_groups))
3
{% endraw %} {% raw %}
learn.lr_find(suggest_funcs=[minimum, steep, valley, slide])
SuggestedLRs(minimum=0.0033113110810518267, steep=3.311311274956097e-06, valley=0.0014454397605732083, slide=0.0014454397605732083)
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=1e-3, cbs=fit_cbs)
epoch train_loss valid_loss exact_match f1 time
0 0.573754 0.898658 77.049180 84.941079 01:28
{% endraw %}

Showing results

And here we create a @typedispatched implementation of Learner.show_results for a more intuitive QA task

{% raw %}
{% endraw %} {% raw %}
learn.show_results(learner=learn, skip_special_tokens=True, max_n=4, trunc_at=500)
text found start/end answer pred start/end pred answer
0 which nfl team represented the afc at super bowl 50? super bowl 50 was an american football game to determine the champion of the national football league ( nfl ) for the 2015 season. the american football conference ( afc ) champion denver broncos defeated the national football conference ( nfc ) champion carolina panthers 24 – 10 to earn their third super bowl title. the game was played on february 7, 2016, at levi's stadium in the san francisco bay area at santa clara, california. as this was True (46, 48) denver broncos (46, 48) denver broncos
1 what day was the game played on? super bowl title. the game was played on february 7, 2016, at levi's stadium in the san francisco bay area at santa clara, california. as this was the 50th super bowl, the league emphasized the " golden anniversary " with various gold - themed initiatives, as well as temporarily suspending the tradition of naming each super bowl game with roman numerals ( under which the game would have been known as " super bowl l " ), so that the logo could prominently feature True (19, 23) february 7, 2016 (19, 21) february 7
2 who coached each super bowl 50 participant in their most recent super bowl appearance prior to super bowl 50? of only six teams to have acquired a 15 – 1 record, while the denver broncos became one of four teams to have made eight appearances in the super bowl. the broncos made their second super bowl appearance in three years, having reached super bowl xlviii, while the panthers made their second super bowl appearance in franchise history, their other appearance being super bowl xxxviii. coinci True (105, 107) john fox (105, 107) john fox
3 how much did it cost to build the stadium where super bowl 50 was played? on may 21, 2013, nfl owners at their spring meetings in boston voted and awarded the game to levi's stadium. the $ 1. 2 billion stadium opened in 2014. it is the first super bowl held in the san francisco bay area since super bowl xix in 1985, and the first in california since super bowl xxxvii took place in san diego in 2003. True (44, 49) $ 1. 2 billion (44, 49) $ 1. 2 billion
{% endraw %} {% raw %}
learn.unfreeze()
{% endraw %} {% raw %}
learn.fit_one_cycle(1, lr_max=slice(1e-9, 1e-7), cbs=fit_cbs)
epoch train_loss valid_loss exact_match f1 time
0 0.533515 0.899340 77.049180 84.870822 03:07
{% endraw %} {% raw %}
learn.recorder.plot_loss()
{% endraw %} {% raw %}
learn.show_results(learner=learn, max_n=2, trunc_at=100)
text found start/end answer pred start/end pred answer
0 which nfl team represented the afc at super bowl 50? super bowl 50 was an american football game to True (46, 48) denver broncos (46, 48) denver broncos
1 how many tackles did von miller accomlish by himself in the game? the broncos took an early lead in True (74, 75) five (74, 77) five solo tackles
{% endraw %}

Prediction

Note that there is a bug currently in fastai v2 (or with how I'm assembling everything) that currently prevents us from seeing the decoded predictions and probabilities for the "end" token.

{% raw %}

Learner.blurr_predict_answers[source]

Learner.blurr_predict_answers(question_contexts:Union[dict, typing.List[dict]], slow_word_ids_func:Optional[typing.Callable]=None)

Type Default Details
question_contexts typing.Union[dict, typing.List[dict]] The str (or list of strings) you want to get token classification predictions for
slow_word_ids_func typing.Optional[typing.Callable] None If using a slow tokenizer, users will need to prove a slow_word_ids_func that accepts a
tokenizzer, example index, and a batch encoding as arguments and in turn returnes the
equavlient of fast tokenizer's `word_ids``
{% endraw %} {% raw %}
{% endraw %} {% raw %}
context = "George Lucas created Star Wars in 1977. He directed and produced it."

learn.blurr_predict_answers({"question": "What did George Lucas make?", "context": context})
[{'answer': 'Star Wars', 'start': 21, 'end': 30, 'score': 0.9665559530258179}]
{% endraw %} {% raw %}
context = "George Lucas created Star Wars in 1977. He directed and produced it."

learn.blurr_predict_answers(
    [
        {"question": "What did George Lucas make?", "context": context},
        {"question": "What year did Star Wars come out?", "context": context},
        {"question": "What did George Lucas do?", "context": context},
        {"question": "Who plays Spock in the movie?", "context": context},
    ]
)
[{'answer': 'Star Wars', 'start': 21, 'end': 30, 'score': 0.9665559530258179},
 {'answer': '1977', 'start': 34, 'end': 38, 'score': 0.8944364786148071},
 {'answer': 'directed and produced it',
  'start': 43,
  'end': 67,
  'score': 0.28049948811531067},
 {'answer': 'George Lucas',
  'start': 0,
  'end': 12,
  'score': 0.18880312144756317}]
{% endraw %}

Inference

Note that I had to replace the loss function because of the above-mentioned issue to exporting the model with the MultiTargetLoss loss function. After getting our inference learner, we put it back and we're good to go!

{% raw %}
export_name = "q_and_a_learn_export"
{% endraw %} {% raw %}
learn.loss_func = CrossEntropyLossFlat()
learn.export(fname=f"{export_name}.pkl")
{% endraw %} {% raw %}
inf_learn = load_learner(fname=f"{export_name}.pkl")
inf_learn.loss_func = MultiTargetLoss()

context = "George Lucas created Star Wars in 1977. He directed and produced it."

inf_learn.blurr_predict_answers(
    [
        {"question": "What did George Lucas make?", "context": context},
        {"question": "What year did Star Wars come out?", "context": context},
        {"question": "What did George Lucas do?", "context": context},
        {"question": "Who plays Spock in the movie?", "context": context},
    ]
)
[{'answer': 'Star Wars', 'start': 21, 'end': 30, 'score': 0.9665559530258179},
 {'answer': '1977', 'start': 34, 'end': 38, 'score': 0.8944369554519653},
 {'answer': 'directed and produced it',
  'start': 43,
  'end': 67,
  'score': 0.2805001735687256},
 {'answer': 'George Lucas',
  'start': 0,
  'end': 12,
  'score': 0.18880519270896912}]
{% endraw %}

High-level API

{% raw %}

class BlearnerForQuestionAnswering[source]

BlearnerForQuestionAnswering(dls:DataLoaders, hf_model:PreTrainedModel, base_model_cb:BaseModelCallback=BaseModelCallback, loss_func=None, opt_func=Adam, lr=0.001, splitter=trainable_params, cbs=None, metrics=None, path=None, model_dir='models', wd=None, wd_bn_bias=False, train_bn=True, moms=(0.95, 0.85, 0.95)) :: Blearner

Group together a model, some dls and a loss_func to handle training

{% endraw %} {% raw %}
{% endraw %}

Example

BLearnerForQuestionAnswering requires a question, context (within which to find the answer to the question), and the start/end indices of where the answer lies in the tokenized context. Because those indices vary by tokenizer, we can pass a preprocess_func that will take our raw data, perform any preprocessing we want, and return it in a way that will work for extractive QA.

Preprocess data

{% raw %}
raw_datasets = load_dataset("squad", split=["train[:1000]", "validation[:200]"])

raw_train_df = pd.DataFrame(raw_datasets[0])
raw_valid_df = pd.DataFrame(raw_datasets[1])

raw_train_df["is_valid"] = False
raw_valid_df["is_valid"] = True

# concatenate into a single DataFrame
squad_df = pd.concat([raw_train_df, raw_valid_df])

# include the required start/end character indicies and full text of the answer
squad_df["ans_start_char_idx"] = squad_df.answers.apply(lambda v: v["answer_start"][0] if len(v["answer_start"]) > 0 else "0")
squad_df["answer_text"] = squad_df.answers.apply(lambda v: v["text"][0] if len(v["text"]) > 0 else "")
squad_df["ans_end_char_idx"] = squad_df["ans_start_char_idx"].astype(int) + squad_df["answer_text"].str.len()

# run our modified DataFrame thru the QAPreprocessor to get the start/end "token" indices we want to predict
preprocessor = QAPreprocessor(
    hf_tokenizer, id_attr="id", tok_kwargs={"return_overflowing_tokens": True, "max_length": max_seq_len, "stride": 64}
)

proc_df = preprocessor.process_df(squad_df)
proc_df.head(1)
Reusing dataset squad (/home/wgilliam/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
id title context question answers is_valid ans_start_char_idx answer_text ans_end_char_idx proc_question proc_context ans_start_token_idx ans_end_token_idx is_answerable
0 5733be284776f41900661182 University_of_Notre_Dame Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct... To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]} False 515 Saint Bernadette Soubirous 541 To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputed 0 0 False
{% endraw %}

Define your Blearner

{% raw %}
pretrained_model_name = "bert-large-uncased-whole-word-masking-finetuned-squad"

learn = BlearnerForQuestionAnswering.from_data(
    proc_df,
    pretrained_model_name,
    id_attr="id",
    question_attr="proc_question",
    context_attr="proc_context",
    max_seq_len=128,
    dl_kwargs={"bs": 4},
).to_fp16()

validation_ds = proc_df[proc_df.is_valid == True].to_dict(orient="records")
fit_cbs = [QAMetricsCallback(compute_metrics_func=compute_qa_metrics, validation_ds=validation_ds)]
{% endraw %} {% raw %}
learn.dls.show_batch(dataloaders=learn.dls, max_n=4, trunc_at=500)
text found start/end answer
0 with what institute did notre dame agree to an exchange program in the 1960s? ism. " thomas blantz, c. s. c., notre dame's vice president of student affairs, added that coeducation " opened up a whole other pool of very bright students. " two of the male residence halls were converted for the newly admitted female students that first year, while two others were converted for the next school year. in 1971 mary ann proctor became the first female undergraduate ; she transferred from st. mary's col False (0, 0)
1 who is beyonce's alter ego?... sasha fierce was released on november 18, 2008 in the united states. the album formally introduces beyonce's alter ego sasha fierce, conceived during the making of her 2003 single " crazy in love ", selling 482, 000 copies in its first week, debuting atop the billboard 200, and giving beyonce her third consecutive number - one album in the us. the album featured the number - one song " single ladies ( put a ring on it ) " and the top - five songs " if i were a boy True (13, 15) sasha fierce
2 what is the daily student paper at notre dame called? annually. the newspapers have varying publication interests, with the observer published daily and mainly reporting university and other news, and staffed by students from both notre dame and saint mary's college. unlike scholastic and the dome, the observer is an independent publication and does not have a faculty advisor or any editorial oversight from the university. in 1987, when some students believed that the observer began to show a co True (23, 25) the observer
3 with what institute did notre dame agree to an exchange program in the 1960s? features formerly considered advantageous and enviable are now seen as anachronistic and out of place.... in this environment of diversity, the integration of the sexes is a normal and expected aspect, replacing separatism. " thomas blantz, c. s. c., notre dame's vice president of student affairs, added that coeducation " opened up a whole other pool of very bright students. " two of the male residence halls were conve False (0, 0)
{% endraw %}

Train

{% raw %}
validation_ds = proc_df[proc_df.is_valid == True].to_dict(orient="records")
fit_cbs = [QAMetricsCallback(compute_metrics_func=compute_qa_metrics, validation_ds=validation_ds)]

learn.fit_one_cycle(3, lr_max=1e-3, cbs=fit_cbs)
epoch train_loss valid_loss exact_match f1 time
0 0.690718 0.919565 76.393443 84.056364 01:49
1 0.535996 0.990978 77.377049 82.723212 01:50
2 0.308149 0.990636 80.983607 85.467869 01:50
{% endraw %} {% raw %}
learn.show_results(learner=learn, skip_special_tokens=True, max_n=2, trunc_at=500)
text found start/end answer pred start/end pred answer
0 which nfl team represented the afc at super bowl 50? super bowl 50 was an american football game to determine the champion of the national football league ( nfl ) for the 2015 season. the american football conference ( afc ) champion denver broncos defeated the national football conference ( nfc ) champion carolina panthers 24 – 10 to earn their third super bowl title. the game was played on february 7, 2016, at levi's stadium in the san francisco bay area at santa clara, california. as this was True (46, 48) denver broncos (46, 48) denver broncos
1 what new orleans stadium was considered for super bowl 50? the league eventually narrowed the bids to three sites : new orleans'mercedes - benz superdome, miami's sun life stadium, and the san francisco bay area's levi's stadium. True (26, 31) mercedes - benz superdome (26, 31) mercedes - benz superdome
{% endraw %}

Inference

{% raw %}
learn = learn.to_fp32()
learn.loss_func = CrossEntropyLossFlat()
learn.export(fname=f"{export_name}.pkl")
{% endraw %} {% raw %}
inf_learn = load_learner(fname=f"{export_name}.pkl")
inf_learn.loss_func = MultiTargetLoss()

context = "George Lucas created Star Wars in 1977. He directed and produced it."

inf_learn.blurr_predict_answers(
    [
        {"question": "What did George Lucas make?", "context": context},
        {"question": "What year did Star Wars come out?", "context": context},
        {"question": "What did George Lucas do?", "context": context},
        {"question": "Who plays Spock in the movie?", "context": context},
    ]
)
[{'answer': 'Star Wars', 'start': 21, 'end': 30, 'score': 0.9861960411071777},
 {'answer': '1977', 'start': 34, 'end': 38, 'score': 0.7073510885238647},
 {'answer': 'directed and produced',
  'start': 43,
  'end': 64,
  'score': 0.4036509692668915},
 {'answer': None, 'start': 0, 'end': 0, 'score': 0.9132930040359497}]
{% endraw %}