# Custom GPT2 Model won't load after training **URL:** \ **Category:** Intermediate\ **Created:** [September 15, 2021, 12:20pm UTC](https://discuss.huggingface.co/t/custom-gpt2-model-wont-load-after-training/10017 "2021-09-15T12:20:52Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![mingaflo](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@mingaflo](https://discuss.huggingface.co/u/mingaflo)\ **Post date:** [September 15, 2021, 12:20pm UTC](https://discuss.huggingface.co/t/custom-gpt2-model-wont-load-after-training/10017/1 "2021-09-15T12:20:52Z")
## Environment info - `transformers` version: 4.10.2 - Platform: Linux-5.11.0-34-generic-x86\_64-with-glibc2.29 - Python version: 3.8.10 - PyTorch version (GPU?): 1.8.1+cu102 (True) - Tensorflow version (GPU?): 2.4.1 (False) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: - Using distributed or parallel set-up in script?: ## Information Model I am using GPT2PretrainedModel. The problem arises when using: - the official example scripts: (give details below) - [x] my own modified scripts: (give details below) The tasks I am working on is: - an official GLUE/SQUaD task: (give the name) - [x] my own task or dataset: (give details below) ## The Problem I was able to train my customly build model but I am not able to load it with the `from_pretrained()` function. BTW I don’t save the model manually if that is important. The saving is done by the Huggingface-Trainer. The Error message: ```auto model = CustomGPTModel.from_pretrained("results/checkpoint-19065", config=config) File "/home/flo/PycharmProjects/EET2/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1325, in from_pretrained model = cls(config, *model_args, **model_kwargs) TypeError: __init__ () missing 1 required positional argument: 'config' ``` I load the model like this: ```python config = AutoConfig.from_pretrained("results/checkpoint-19065") model = CustomGPTModel.from_pretrained("dbmdz/german-gpt2", config=config) # custom = CustomGPTModel(model=model, config=config) training_args = TrainingArguments( output_dir='./results', # output directory per_device_train_batch_size=1, # batch size per device during training per_device_eval_batch_size=1, # batch size for evaluation logging_dir='./logs/event/', # directory for storing logs ) trainer = Trainer( model=model, # the instantiated 🤗 Transformers model to be trained # model=custom, # the instantiated 🤗 Transformers model to be trained args=training_args, # training arguments, defined above compute_metrics=compute_everything, ) trainer.predict(test_dataset=test_dataset) ``` As you can tell from the commented code, I tried a lot of different approaches to no avail. Other approaches I tried: ```auto config = AutoConfig.from_pretrained("results/checkpoint-19065") model = CustomGPTModel.from_pretrained("results/checkpoint-19065", config=config) # or config = AutoConfig.from_pretrained("results/checkpoint-19065") model = CustomGPTModel.from_pretrained("results/checkpoint-19065") ``` Anyway the question is _How do I load my custom model?_ I think it is because of the way I initialize the CustomGPTModel (see below). ## The Task / More Information on what I am Doing I am training the “dbmdz/german-gpt2” on a multilabel-classification task. For this I had to create my own model by subclassing the GPT2PretrainedModel. This is what the model looks like: ```python class CustomGPTModel(GPT2PreTrainedModel): def __init__ (self, model, config): super(CustomGPTModel, self). __init__ (config) self.num_labels = config.num_labels self.init_weights() ### Architecture: self.transformer = model self.linear1 = nn.Linear(config.n_embd, 256) self.score = nn.Linear(256, self.num_labels, bias=False) self.dropout = nn.Dropout(p=0.2) self.sig = nn.Sigmoid() self.relu = nn.ReLU() # Model parallel self.model_parallel = False self.device_map = None def forward(self, input_ids=None, past_key_values=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None, ): return_dict = return_dict if return_dict is not None else self.config.use_return_dict transformer_outputs = self.transformer( input_ids, past_key_values=past_key_values, attention_mask=attention_mask, token_type_ids=token_type_ids, position_ids=position_ids, head_mask=head_mask, inputs_embeds=inputs_embeds, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) hidden_states = transformer_outputs[0] # call model hdn_2 = self.linear1(hidden_states) # first linear logits = self.score(self.dropout(self.relu(hdn_2))) # apply activation/dropout and final layer if input_ids is not None: batch_size, sequence_length = input_ids.shape[:2] else: batch_size, sequence_length = inputs_embeds.shape[:2] assert ( self.config.pad_token_id is not None or batch_size == 1 ), "Cannot handle batch sizes > 1 if no padding token is defined." if self.config.pad_token_id is None: sequence_lengths = -1 else: if input_ids is not None: sequence_lengths = torch.ne(input_ids, self.config.pad_token_id).sum(-1) - 1 pooled_logits = logits[range(batch_size), sequence_lengths] loss = None if labels is not None: loss_fct = BCEWithLogitsLoss() loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1, self.num_labels)) return (loss, pooled_logits) else: return logits ``` Here I initialize the model for training: ```python training_args = TrainingArguments( output_dir='./results', # output directory num_train_epochs=10, # total number of training epochs per_device_train_batch_size=1, # batch size per device during training per_device_eval_batch_size=1, # batch size for evaluation warmup_steps=500, # number of warmup steps for learning rate scheduler weight_decay=0.01, # strength of weight decay logging_dir='./logs/event/', # directory for storing logs logging_steps=1000, load_best_model_at_end=True, evaluation_strategy="epoch", # Evaluation is done (and logged) every eval_steps save_strategy="epoch", # logging_first_step = True, do_eval=True, ) trainer = Trainer( model=custom_gpt2, # the instantiated 🤗 Transformers model to be trained args=training_args, # training arguments, defined above train_dataset=train_dataset, # training dataset eval_dataset=val_dataset, # evaluation dataset compute_metrics=compute_everything, callbacks=[EarlyStoppingCallback(early_stopping_patience=3)], ) trainer.train() ``` ## Expected behavior The model should get loaded as expected. I tried to fix it for two days now so I thought creating an issue is the last straw. Hopefully someone can explain what I am doing wrong 😅 If someone needs more information please tell me! ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [September 15, 2021, 1:27pm UTC](https://discuss.huggingface.co/t/custom-gpt2-model-wont-load-after-training/10017/2 "2021-09-15T13:27:25Z")
You should either: - use the regular `torch.load` to load the weights of your models - or make sure your custom model class subclasses `PreTrainedModel` and is initalized with a single config, like all Transformers models if you want to use `from_pretrained`.