# Trainer crashes during predict and with compute\_metrics **URL:** \ **Category:** Beginners\ **Created:** [November 19, 2020, 8:16pm UTC](https://discuss.huggingface.co/t/trainer-crashes-during-predict-and-with-compute-metrics/2136 "2020-11-19T20:16:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![shivin7](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@shivin7](https://discuss.huggingface.co/u/shivin7)\ **Post date:** [November 19, 2020, 8:16pm UTC](https://discuss.huggingface.co/t/trainer-crashes-during-predict-and-with-compute-metrics/2136/1 "2020-11-19T20:16:10Z")
Hi, I’m trying to train a `XLNetForSequenceClassification` model using `Trainer` to classify sentences into 3 categories. It works fine for the training and eval datasets during `trainer.train()` (loss reduces as expected), but if I try to use `compute_metrics` argument in my trainer or I try to obtain the predictions on the same eval dataset using `trainer.predict()`, it crashes with the following error : ```auto /usr/local/lib/python3.6/dist-packages/transformers/trainer.py in predict(self, test_dataset) 1353 test_dataloader = self.get_test_dataloader(test_dataset) 1354 -> 1355 return self.prediction_loop(test_dataloader, description="Prediction") 1356 1357 def prediction_loop( /usr/local/lib/python3.6/dist-packages/transformers/trainer.py in prediction_loop(self, dataloader, description, prediction_loss_only) 1442 eval_losses_gatherer.add_arrays(self._gather_and_numpify(losses_host, "eval_losses")) 1443 if not prediction_loss_only: -> 1444 preds_gatherer.add_arrays(self._gather_and_numpify(preds_host, "eval_preds")) 1445 labels_gatherer.add_arrays(self._gather_and_numpify(labels_host, "eval_label_ids")) 1446 /usr/local/lib/python3.6/dist-packages/transformers/trainer_pt_utils.py in add_arrays(self, arrays) 328 # If we get new arrays that are too big too fit, we expand the shape fo the storage 329 self._storage = nested_expand_like(self._storage, arrays_shape[1], padding_index=self.padding_index) --> 330 slice_len = self._nested_set_tensors(self._storage, arrays) 331 for i in range(self.world_size): 332 self._offsets[i] += slice_len /usr/local/lib/python3.6/dist-packages/transformers/trainer_pt_utils.py in _nested_set_tensors(self, storage, arrays) 335 if isinstance(arrays, (list, tuple)): 336 for x, y in zip(storage, arrays): --> 337 slice_len = self._nested_set_tensors(x, y) 338 return slice_len 339 assert ( /usr/local/lib/python3.6/dist-packages/transformers/trainer_pt_utils.py in _nested_set_tensors(self, storage, arrays) 335 if isinstance(arrays, (list, tuple)): 336 for x, y in zip(storage, arrays): --> 337 slice_len = self._nested_set_tensors(x, y) 338 return slice_len 339 assert ( /usr/local/lib/python3.6/dist-packages/transformers/trainer_pt_utils.py in _nested_set_tensors(self, storage, arrays) 347 else: 348 storage[self._offsets[i] : self._offsets[i] + slice_len, : arrays.shape[1]] = arrays[ --> 349 i * slice_len : (i + 1) * slice_len 350 ] 351 return slice_len ValueError: could not broadcast input array from shape (4565,16,768) into shape (916,16,768) ``` Here 916 is the size of the eval dataset and 16 is the batch\_size, and my guess is that 4565 is the longest concatenated feature list? My code is as follows : ```auto class XLNetDataset(data.Dataset): def __init__ (self, dfObject): self.dfObject = dfObject # Pandas dataframe def __len__ (self): return self.dfObject.shape[0] def __getitem__ (self, idx): if torch.is_tensor(idx): idx = idx.tolist() dfRows = self.dfObject.iloc[idx] dfSentences = dfRows['sentence'] dfLabels = dfRows['p_typ'] return dfSentences, dfLabels def XLNetCollatFunc(data): sents = [elem[0] for elem in data] labels = [elem[1] for elem in data] encoded_result = xlTokenizer(sents, padding=True, truncation=True, max_length=128, return_tensors='pt', return_attention_mask=True) output = {'input_ids': encoded_result['input_ids'], 'attention_mask': encoded_result['attention_mask'], 'token_type_ids': encoded_result['token_type_ids'], 'labels': torch.tensor(labels)} return output trainDataset = XLNetDataset(trainData) # trainData is pandas DF containing train sentences testDataset = XLNetDataset(testData) # testData is pandas DF containing test sentences xlTokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') xlNetModel = XLNetForSequenceClassification.from_pretrained('xlnet-base-cased', num_labels=3) for param in xlNetModel.base_model.parameters(): param.requires_grad = False trainArgs = TrainingArguments( num_train_epochs = 1, evaluation_strategy = 'epoch', per_device_train_batch_size = 16, per_device_eval_batch_size = 16 ) trainer = Trainer( model = xlNetModel, args = trainArgs, train_dataset = trainDataset, eval_dataset = testDataset, data_collator = XLNetCollatFunc ) trainer.train() trainer.predict(testDataset) ``` I’m guessing the problem is somewhere with my custom data collator (I’m still a little unsure of the exact data format the data collator or trainer is expected to receive), but I can’t understand how it is able to produce training and evaluation loss during `trainer.train()` and not during the `predict()` call. I’m using the latest API version (3.5) ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 19, 2020, 9:03pm UTC](https://discuss.huggingface.co/t/trainer-crashes-during-predict-and-with-compute-metrics/2136/2 "2020-11-19T21:03:19Z")
There have been several issues around this. This should be solved in the latest release (pre-release to be more specific). You should be able to install it with ```auto pip install --upgrade --pre transformers ``` ---
**Author:** ![shivin7](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@shivin7](https://discuss.huggingface.co/u/shivin7)\ **Post date:** [November 19, 2020, 9:23pm UTC](https://discuss.huggingface.co/t/trainer-crashes-during-predict-and-with-compute-metrics/2136/3 "2020-11-19T21:23:28Z")
Yes, that totally works. Thank you for your help! I browsed the topics here but didn’t see if there was any issue already opened, my bad! Thanks again! ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 19, 2020, 9:39pm UTC](https://discuss.huggingface.co/t/trainer-crashes-during-predict-and-with-compute-metrics/2136/4 "2020-11-19T21:39:27Z")
The issues were on GitHub, not the forum, that’s why you didn’t find them 🙂 . Glad this is solved for you now! ---
**Author:** ![sb1](https://avatars.discourse-cdn.com/v4/letter/s/0ea827/32.png) [@sb1](https://discuss.huggingface.co/u/sb1)\ **Post date:** [April 13, 2021, 3:04am UTC](https://discuss.huggingface.co/t/trainer-crashes-during-predict-and-with-compute-metrics/2136/5 "2021-04-13T03:04:54Z")
What was the fix for this? Currently I’m getting: could not broadcast input array from shape (757) into shape (204) in trainer\_utils.py \_nested\_set\_tensors(self, storage, arrays) 344 for i in range(self.world\_size): 345 if len(arrays.shape) == 1: → 346 storage[self.\_offsets[i] : self.\_offsets[i] + slice\_len] = arrays[i \* slice\_len : (i + 1) \* slice\_len] 347 else: 348 storage[self.\_offsets[i] : self.\_offsets[i] + slice\_len, : arrays.shape[1]] = arrays[ unfortunately I have to use the transformers version 3.5 and the respective trainer version. what was the workaround? Is it because of dynamic padding?(padding through the data collator?)