transformers-Causal lanuage modeling

本文主要是介绍transformers-Causal lanuage modeling，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

https://huggingface.co/docs/transformers/main/en/tasks/language_modelinghttps://huggingface.co/docs/transformers/main/en/tasks/language_modelingcausal lanuage model常用于文本生成。预测token系列中的下一个toekn，并且model只能关注左侧的token，模型看不到右侧的token。

from datasets import load_dataseteli5 = load_dataset("eli5", split="train_asks[:5000]")
eli5 = eli5.train_test_split(test_size=0.2)
eli5["train"][0]
{'answers': {'a_id': ['c3d1aib', 'c3d4lya'],'score': [6, 3],'text': ["The velocity needed to remain in orbit is equal to the square root of Newton's constant times the mass of earth divided by the distance from the center of the earth. I don't know the altitude of that specific mission, but they're usually around 300 km. That means he's going 7-8 km/s.\n\nIn space there are no other forces acting on either the shuttle or the guy, so they stay in the same position relative to each other. If he were to become unable to return to the ship, he would presumably run out of oxygen, or slowly fall into the atmosphere and burn up.","Hope you don't mind me asking another question, but why aren't there any stars visible in this photo?"]},'answers_urls': {'url': []},'document': '','q_id': 'nyxfp','selftext': '_URL_0_\n\nThis was on the front page earlier and I have a few questions about it. Is it possible to calculate how fast the astronaut would be orbiting the earth? Also how does he stay close to the shuttle so that he can return safely, i.e is he orbiting at the same speed and can therefore stay next to it? And finally if his propulsion system failed, would he eventually re-enter the atmosphere and presumably die?','selftext_urls': {'url': ['http://apod.nasa.gov/apod/image/1201/freeflyer_nasa_3000.jpg']},'subreddit': 'askscience','title': 'Few questions about this space walk photograph.','title_urls': {'url': []}}

1.Preprocess

from transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("distilgpt2")

eli5 = eli5.flatten()
eli5["train"][0]
{'answers.a_id': ['c3d1aib', 'c3d4lya'],'answers.score': [6, 3],'answers.text': ["The velocity needed to remain in orbit is equal to the square root of Newton's constant times the mass of earth divided by the distance from the center of the earth. I don't know the altitude of that specific mission, but they're usually around 300 km. That means he's going 7-8 km/s.\n\nIn space there are no other forces acting on either the shuttle or the guy, so they stay in the same position relative to each other. If he were to become unable to return to the ship, he would presumably run out of oxygen, or slowly fall into the atmosphere and burn up.","Hope you don't mind me asking another question, but why aren't there any stars visible in this photo?"],'answers_urls.url': [],'document': '','q_id': 'nyxfp','selftext': '_URL_0_\n\nThis was on the front page earlier and I have a few questions about it. Is it possible to calculate how fast the astronaut would be orbiting the earth? Also how does he stay close to the shuttle so that he can return safely, i.e is he orbiting at the same speed and can therefore stay next to it? And finally if his propulsion system failed, would he eventually re-enter the atmosphere and presumably die?','selftext_urls.url': ['http://apod.nasa.gov/apod/image/1201/freeflyer_nasa_3000.jpg'],'subreddit': 'askscience','title': 'Few questions about this space walk photograph.','title_urls.url': []}

def preprocess_function(examples):return tokenizer([" ".join(x) for x in examples["answers.text"]])tokenized_eli5 = eli5.map(preprocess_function,batched=True,num_proc=4,remove_columns=eli5["train"].column_names,
)
block_size = 128def group_texts(examples):# Concatenate all texts.concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}total_length = len(concatenated_examples[list(examples.keys())[0]])# We drop the small remainder, we could add padding if the model supported it instead of this drop, you can# customize this part to your needs.if total_length >= block_size:total_length = (total_length // block_size) * block_size# Split by chunks of block_size.result = {k: [t[i : i + block_size] for i in range(0, total_length, block_size)]for k, t in concatenated_examples.items()}result["labels"] = result["input_ids"].copy()return resultlm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)from transformers import DataCollatorForLanguageModelingtokenizer.pad_token = tokenizer.eos_token
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

2.Train

from transformers import AutoModelForCausalLM, TrainingArguments, Trainermodel = AutoModelForCausalLM.from_pretrained("distilgpt2")training_args = TrainingArguments(output_dir="my_awesome_eli5_clm-model",evaluation_strategy="epoch",learning_rate=2e-5,weight_decay=0.01,push_to_hub=True,
)trainer = Trainer(model=model,args=training_args,train_dataset=lm_dataset["train"],eval_dataset=lm_dataset["test"],data_collator=data_collator,
)trainer.train()

3.推理

from transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("my_awesome_eli5_clm-model")
inputs = tokenizer(prompt, return_tensors="pt").input_idstokenizer.batch_decode(outputs, skip_special_tokens=True)

这篇关于transformers-Causal lanuage modeling的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---