--- title: data.seq2seq.language_modeling keywords: fastai sidebar: home_sidebar summary: "This module contains the bits required to use the fastai DataBlock API and/or mid-level data processing pipelines to organize your data for causal and masked language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus." description: "This module contains the bits required to use the fastai DataBlock API and/or mid-level data processing pipelines to organize your data for causal and masked language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus." nb_path: "nbs/01zb_data-seq2seq-language-modeling.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}
torch.cuda.set_device(1)
print(f'Using GPU #{torch.cuda.current_device()}: {torch.cuda.get_device_name()}')
Using GPU #1: GeForce GTX 1080 Ti
{% endraw %}

Causal LM

{% raw %}
wiki_path = untar_data(URLs.WIKITEXT_TINY)
wiki_path.ls()
(#2) [Path('/home/wgilliam/.fastai/data/wikitext-2/train.csv'),Path('/home/wgilliam/.fastai/data/wikitext-2/test.csv')]
{% endraw %} {% raw %}
train_df = pd.read_csv(wiki_path/'train.csv', header=None)
valid_df = pd.read_csv(wiki_path/'test.csv', header=None)

print(len(train_df), len(valid_df))
train_df.head()
615 47
0
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....
{% endraw %} {% raw %}
train_df['is_valid'] = False
valid_df['is_valid'] = True

df = pd.concat([train_df, valid_df])
df.head()
0 is_valid
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z... False
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re... False
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit... False
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch... False
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers .... False
{% endraw %} {% raw %}
task = HF_TASKS_AUTO.CausalLM

pretrained_model_name = "gpt2"
hf_arch, hf_config, hf_tokenizer, hf_model = BLURR_MODEL_HELPER.get_hf_objects(pretrained_model_name, task=task)

# some tokenizers like gpt and gpt2 do not have a pad token, so we add it here mainly for the purpose
# of setting the "labels" key appropriately (see below)
if (hf_tokenizer.pad_token is None): hf_tokenizer.pad_token = '[PAD]'

hf_tokenizer.pad_token, hf_tokenizer.pad_token_id
Using pad_token, but it is not set yet.
('[PAD]', 50256)
{% endraw %} {% raw %}

class HF_CausalLMBeforeBatchTransform[source]

HF_CausalLMBeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model, max_length=None, padding=True, truncation=True, is_split_into_words=False, ignore_token_id=-100, tok_kwargs={}, text_gen_kwargs={}, **kwargs) :: HF_BeforeBatchTransform

Handles everything you need to assemble a mini-batch of inputs and targets, as well as decode the dictionary produced as a byproduct of the tokenization process in the encodes method.

{% endraw %} {% raw %}
{% endraw %}

Our HF_CausalLMBeforeBatchTransform allows us to update the input's labels and our targets appropriately given a causal LM task.

The labels argument allows you to forgo calculating the loss yourself by letting huggingface return it for you should you choose to do that. Padding tokens are set to -100 by default (e.g., CrossEntropyLossFlat().ignore_index) and prevent cross entropy loss from considering token prediction for tokens it should ... i.e., the padding tokens. For more information on the meaning of this argument, see the huggingface glossary entry for "Labels"

{% raw %}
before_batch_tfm = HF_CausalLMBeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model)
blocks = (HF_Seq2SeqBlock(before_batch_tfm=before_batch_tfm), noop)

dblock = DataBlock(blocks=blocks, get_x=ColReader(0), splitter=ColSplitter(col='is_valid'))
{% endraw %} {% raw %}
dls = dblock.dataloaders(df, bs=4)
{% endraw %} {% raw %}
b = dls.one_batch()
{% endraw %} {% raw %}
b[0]['input_ids'].shape, b[0]['labels'].shape, b[1].shape
(torch.Size([4, 1024]), torch.Size([4, 1024]), torch.Size([4, 1024]))
{% endraw %} {% raw %}
explode_types(b)
{tuple: [dict, torch.Tensor]}
{% endraw %} {% raw %}
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=500)
text target
0 \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They Are a @-@ <unk>'" became anthems for the American civil rights and anti @-@ war movements. After he left his initial base in the American folk music revival, his six @-@ minute single " Like a Rolling Stone " altered the range of popular music in 1965. His mid @-@ 1960s recordings, backed by rock musicians, reached the top end of the United States music charts while also attracting <unk> and criticism from others in the folk movement. \n Dylan's lyrics have incorporated various political, social, philosophical, and literary influences. They defied existing pop music conventions and appealed to the burgeoning counterculture. Initially inspired by the performances of Little Richard and the songwriting of Woody Guthrie, Robert Johnson, and Hank Williams, Dylan has amplified and <unk> musical genres. His recording career, spanning more than 50 years, has explored the traditions in American song — from folk, blues, and country to gospel, rock and roll, and rockabilly to English, Scottish, and Irish folk music, embracing even jazz and the Great American Songbook. Dylan performs with guitar, keyboards, and harmonica. Backed by a changing lineup of musicians, he has toured steadily since the late 1980s on what has been dubbed the Never Ending Tour. His accomplishments as a recording artist and performer have been central to his career, but songwriting is considered his greatest contribution. \n Since 1994, Dylan has published six books of drawings and paintings, and his work has been exhibited in major art galleries. As a musician, Dylan has sold more than 100 million records, making him one of the best @-@ selling artists of all time. He has also received numerous awards including eleven Grammy Awards, a Golden Globe Award, and an Academy Award. Dylan has been inducted into the Rock and Roll Hall of Fame, Minnesota Music Hall of Fame, Nashville Songwriters Hall of Fame, and Songwriters Hall of Fame. The Pulitzer Prize jury in 2008 awarded him a special citation for " his profound impact on popular music and American culture, marked by lyrical compositions of extraordinary poetic power. " In May 2012, Dylan received the Presidential Medal of Freedom from President Barack Obama. \n \n = = Life and career = = \n \n \n = = = Origins and musical beginnings = = = \n \n Bob Dylan was born Robert Allen Zimmerman ( Hebrew name <unk> <unk> <unk> <unk> [ Shabtai <unk> ben <unk> ] ) in St Mary's Hospital on May 24, 1941, in Duluth, Minnesota, and raised in Hibbing, Minnesota, on the <unk> Range west of Lake Superior. He has a younger brother, David. Dylan's paternal grandparents, <unk> and Anna Zimmerman, emigrated from Odessa, in the Russian Empire ( now Ukraine ), to the United States following the anti @-@ Semitic <unk> of 1905. His maternal grandparents, Ben and Florence Stone, were Lithuanian Jews who arrived in the United States in 1902. In his autobiography, Chronicles : Volume One, Dylan wrote that his paternal grandmother's maiden name was <unk> and her family originated from <unk> district of <unk> Province in northeastern Turkey. \n Dylan's parents, <unk> Zimmerman and <unk> " Beatty " Stone, were part of a small but close @-@ knit Jewish community. They lived in Duluth until Robert was six, when his father had <unk> and the family returned to his mother's hometown, Hibbing, where they lived for the rest of Robert's childhood. In his early years he listened to the radio — first to blues and country stations from Shreveport, Louisiana and later, when he was a teenager, to rock and roll. He formed several bands while attending Hibbing High School. In the Golden <unk>, he performed covers of songs by Little Richard and Elvis Presley. Their performance of Danny & the Juniors'" Rock and Roll Is Here to Stay " at their high school talent show was so loud that the principal cut the microphone. In 1959, his high school <unk> carried the caption " Robert Zimmerman : to join'Little Richard '. " The same year, as <unk> <unk> [ sic ], he performed two dates with Bobby <unk>, playing piano and clapping. \n Zimmerman moved to Minneapolis in September 1959 and \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They Are a @-@ <unk>'" became anthems for the American civil rights and anti @-@ war movements. After he left his initial base in the American folk music revival, his six @-@ minute single " Like a Rolling Stone " altered the range of popular music in 1965. His mid @-@ 1960s recordings, backed by rock musicians, reached the top end of the United States music charts while also attracting <unk> and criticism from others in the folk movement. \n Dylan's lyrics have incorporated various political, social, philosophical, and literary influences. They defied existing pop music conventions and appealed to the burgeoning counterculture. Initially inspired by the performances of Little Richard and the songwriting of Woody Guthrie, Robert Johnson, and Hank Williams, Dylan has amplified and <unk> musical genres. His recording career, spanning more than 50 years, has explored the traditions in American song — from folk, blues, and country to gospel, rock and roll, and rockabilly to English, Scottish, and Irish folk music, embracing even jazz and the Great American Songbook. Dylan performs with guitar, keyboards, and harmonica. Backed by a changing lineup of musicians, he has toured steadily since the late 1980s on what has been dubbed the Never Ending Tour. His accomplishments as a recording artist and performer have been central to his career, but songwriting is considered his greatest contribution. \n Since 1994, Dylan has published six books of drawings and paintings, and his work has been exhibited in major art galleries. As a musician, Dylan has sold more than 100 million records, making him one of the best @-@ selling artists of all time. He has also received numerous awards including eleven Grammy Awards, a Golden Globe Award, and an Academy Award. Dylan has been inducted into the Rock and Roll Hall of Fame, Minnesota Music Hall of Fame, Nashville Songwriters Hall of Fame, and Songwriters Hall of Fame. The Pulitzer Prize jury in 2008 awarded him a special citation for " his profound impact on popular music and American culture, marked by lyrical compositions of extraordinary poetic power. " In May 2012, Dylan received the Presidential Medal of Freedom from President Barack Obama. \n \n = = Life and career = = \n \n \n = = = Origins and musical beginnings = = = \n \n Bob Dylan was born Robert Allen Zimmerman ( Hebrew name <unk> <unk> <unk> <unk> [ Shabtai <unk> ben <unk> ] ) in St Mary's Hospital on May 24, 1941, in Duluth, Minnesota, and raised in Hibbing, Minnesota, on the <unk> Range west of Lake Superior. He has a younger brother, David. Dylan's paternal grandparents, <unk> and Anna Zimmerman, emigrated from Odessa, in the Russian Empire ( now Ukraine ), to the United States following the anti @-@ Semitic <unk> of 1905. His maternal grandparents, Ben and Florence Stone, were Lithuanian Jews who arrived in the United States in 1902. In his autobiography, Chronicles : Volume One, Dylan wrote that his paternal grandmother's maiden name was <unk> and her family originated from <unk> district of <unk> Province in northeastern Turkey. \n Dylan's parents, <unk> Zimmerman and <unk> " Beatty " Stone, were part of a small but close @-@ knit Jewish community. They lived in Duluth until Robert was six, when his father had <unk> and the family returned to his mother's hometown, Hibbing, where they lived for the rest of Robert's childhood. In his early years he listened to the radio — first to blues and country stations from Shreveport, Louisiana and later, when he was a teenager, to rock and roll. He formed several bands while attending Hibbing High School. In the Golden <unk>, he performed covers of songs by Little Richard and Elvis Presley. Their performance of Danny & the Juniors'" Rock and Roll Is Here to Stay " at their high school talent show was so loud that the principal cut the microphone. In 1959, his high school <unk> carried the caption " Robert Zimmerman : to join'Little Richard '. " The same year, as <unk> <unk> [ sic ], he performed two dates with Bobby <unk>, playing piano and clapping. \n Zimmerman moved to Minneapolis in September 1959 and
1 \n = Guitar Hero = \n \n The Guitar Hero series ( sometimes referred to as the Hero series ) is a series of music rhythm games first published in 2005 by RedOctane and Harmonix, and distributed by Activision, in which players use a guitar @-@ shaped game controller to simulate playing lead, bass guitar, and rhythm guitar across numerous rock music songs. Players match notes that scroll on @-@ screen to colored fret buttons on the controller, strumming the controller in time to the music in order to score points, and keep the virtual audience excited. The games attempt to mimic many features of playing a real guitar, including the use of fast @-@ fingering hammer @-@ <unk> and pull @-@ offs and the use of the whammy bar to alter the pitch of notes. Most games support single player modes, typically a Career mode to play through all the songs in the game, and both competitive and cooperative multiplayer modes. With the introduction of Guitar Hero World Tour in 2008, the game includes support for a four @-@ player band including vocals and drums. The series initially used mostly cover versions of songs created by <unk> Sound, but most recent titles feature soundtracks that are fully master recordings, and in some cases, special re @-@ recordings, of the songs. Later titles in the series feature support for downloadable content in the form of new songs. \n In 2005, RedOctane, a company specializing in the manufacture of unique game controllers, was inspired to create Guitar Hero based on RedOctane's experience creating hardware for Konami's Guitar Freaks arcade game. They enlisted Harmonix, who previously developed several music video games, for development assistance. The first game in the series was made on a budget of $ 1 million. The series became extremely successful, leading to the acquisition of RedOctane by Activision in 2007. Harmonix was acquired by MTV Games and went on to create the Rock Band series of music games in the same vein as Guitar Hero. Activision brought Neversoft ( primarily known for their Tony Hawk series of <unk> games ) on board for future development duties. Additional companies, such as Budcat Creations and Vicarious Visions have assisted in the adaptation of the games for other systems. \n The series currently has eight major releases ( six Guitar Hero games, two DJ Hero games and Band Hero ) and five expansions on gaming consoles. There are spin @-@ offs for Windows and Macintosh systems, mobile phones, the Nintendo DS ( the Guitar Hero : On Tour series ), and an arcade game. The Guitar Hero franchise was a primary brand during the emergence of the popularity of rhythm games as a cultural phenomenon in North America. Such games have been utilized as a learning and development tool for medical purposes. The first game in the series was considered by several journalists to be one of the most influential video games of the first decade of the 21st century. The series has sold more than 25 million units worldwide, earning US $ 2 billion at retail, claimed by Activision to be the 3rd largest game franchise after the Mario and <unk> NFL franchises ; the third main title of the series, Guitar Hero III : Legends of Rock is also claimed by Activision to be the first single video game title to exceed $ 1 billion in sales. \n Despite early success, the series, along with the overall rhythm game genre, suffered from poor sales starting in 2009. Activision had stated in early 2011 that the series was on hiatus for 2011, while a seventh main title in the series was under development ; this title was later cancelled due to the poor quality of the emerging product. Activision later shut down sales of the series'downloadable content, although users who purchased material from it previously may still play what they bought. \n In 2015, Activision announced the first new title to the series in 5 years, Guitar Hero Live, released in October 2015. The title is considered a reboot of the series, with development being performed by FreeStyleGames, who had developed the DJ Hero games previously. \n \n = = History = = \n \n \n = = = Origins and development at Harmonix ( 2005 @-@ 2006 ) = = = \n \n Guitar Hero was created from a partnership between RedOctane, then their own company that produced specialized video game controllers, and Harmonix, a music video game development company who had previously produced <unk>, Amplitude and Karaoke Revolution. RedOctane was seeking to bring in a Guitar Freaks @-@ like game, highly popular in Japan at the time, into Western markets, and approached Harmonix about helping them to develop a music game involving a guitar controller. Both companies agreed to it, and went on to produce Guitar Hero in 2005. The title was highly successful, leading to the development of its successful sequel Guitar Hero II in 2006. While the original controllers for the first Guitar Hero game were designed by Ryan Lesser, Rob Kay, Greg <unk> and Alex <unk> of Harmon \n = Guitar Hero = \n \n The Guitar Hero series ( sometimes referred to as the Hero series ) is a series of music rhythm games first published in 2005 by RedOctane and Harmonix, and distributed by Activision, in which players use a guitar @-@ shaped game controller to simulate playing lead, bass guitar, and rhythm guitar across numerous rock music songs. Players match notes that scroll on @-@ screen to colored fret buttons on the controller, strumming the controller in time to the music in order to score points, and keep the virtual audience excited. The games attempt to mimic many features of playing a real guitar, including the use of fast @-@ fingering hammer @-@ <unk> and pull @-@ offs and the use of the whammy bar to alter the pitch of notes. Most games support single player modes, typically a Career mode to play through all the songs in the game, and both competitive and cooperative multiplayer modes. With the introduction of Guitar Hero World Tour in 2008, the game includes support for a four @-@ player band including vocals and drums. The series initially used mostly cover versions of songs created by <unk> Sound, but most recent titles feature soundtracks that are fully master recordings, and in some cases, special re @-@ recordings, of the songs. Later titles in the series feature support for downloadable content in the form of new songs. \n In 2005, RedOctane, a company specializing in the manufacture of unique game controllers, was inspired to create Guitar Hero based on RedOctane's experience creating hardware for Konami's Guitar Freaks arcade game. They enlisted Harmonix, who previously developed several music video games, for development assistance. The first game in the series was made on a budget of $ 1 million. The series became extremely successful, leading to the acquisition of RedOctane by Activision in 2007. Harmonix was acquired by MTV Games and went on to create the Rock Band series of music games in the same vein as Guitar Hero. Activision brought Neversoft ( primarily known for their Tony Hawk series of <unk> games ) on board for future development duties. Additional companies, such as Budcat Creations and Vicarious Visions have assisted in the adaptation of the games for other systems. \n The series currently has eight major releases ( six Guitar Hero games, two DJ Hero games and Band Hero ) and five expansions on gaming consoles. There are spin @-@ offs for Windows and Macintosh systems, mobile phones, the Nintendo DS ( the Guitar Hero : On Tour series ), and an arcade game. The Guitar Hero franchise was a primary brand during the emergence of the popularity of rhythm games as a cultural phenomenon in North America. Such games have been utilized as a learning and development tool for medical purposes. The first game in the series was considered by several journalists to be one of the most influential video games of the first decade of the 21st century. The series has sold more than 25 million units worldwide, earning US $ 2 billion at retail, claimed by Activision to be the 3rd largest game franchise after the Mario and <unk> NFL franchises ; the third main title of the series, Guitar Hero III : Legends of Rock is also claimed by Activision to be the first single video game title to exceed $ 1 billion in sales. \n Despite early success, the series, along with the overall rhythm game genre, suffered from poor sales starting in 2009. Activision had stated in early 2011 that the series was on hiatus for 2011, while a seventh main title in the series was under development ; this title was later cancelled due to the poor quality of the emerging product. Activision later shut down sales of the series'downloadable content, although users who purchased material from it previously may still play what they bought. \n In 2015, Activision announced the first new title to the series in 5 years, Guitar Hero Live, released in October 2015. The title is considered a reboot of the series, with development being performed by FreeStyleGames, who had developed the DJ Hero games previously. \n \n = = History = = \n \n \n = = = Origins and development at Harmonix ( 2005 @-@ 2006 ) = = = \n \n Guitar Hero was created from a partnership between RedOctane, then their own company that produced specialized video game controllers, and Harmonix, a music video game development company who had previously produced <unk>, Amplitude and Karaoke Revolution. RedOctane was seeking to bring in a Guitar Freaks @-@ like game, highly popular in Japan at the time, into Western markets, and approached Harmonix about helping them to develop a music game involving a guitar controller. Both companies agreed to it, and went on to produce Guitar Hero in 2005. The title was highly successful, leading to the development of its successful sequel Guitar Hero II in 2006. While the original controllers for the first Guitar Hero game were designed by Ryan Lesser, Rob Kay, Greg <unk> and Alex <unk> of Harmon
{% endraw %}

Masked LM

Coming Soon

Cleanup