# Using huggingface transformers trainer method for hugging face datasets

**URL:** <https://discuss.huggingface.co/t/using-huggingface-transformers-trainer-method-for-hugging-face-datasets/25070>\
**Category:** 🤗Datasets\
**Created:** [October 27, 2022, 2:30pm UTC](https://discuss.huggingface.co/t/using-huggingface-transformers-trainer-method-for-hugging-face-datasets/25070 "2022-10-27T14:30:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sp4912](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sp4912](https://discuss.huggingface.co/u/sp4912)\
**Post date:** [October 27, 2022, 2:30pm UTC](https://discuss.huggingface.co/t/using-huggingface-transformers-trainer-method-for-hugging-face-datasets/25070/1 "2022-10-27T14:30:30Z")

</div>

I am trying to train a transformer(Salesforce codet5-small) using the huggingface trainer method and on a hugging face Dataset (namely, “eth\_py150\_open”). However, I’m encountering a number of issues.

Here is the relevant code snippet:

```auto
import torch
import transformers
from datasets import load_dataset_builder
from datasets import load_dataset

corpus=load_dataset("eth_py150_open", split='train')

training_args = transformers.TrainingArguments( #general training arguments
    per_device_train_batch_size = 8,
    warmup_steps = 0,
    weight_decay = 0.01,
    learning_rate = 1e-4,
    num_train_epochs = 12,
    output_dir = './runs/run2/output/',
    logging_dir = './runs/run2/logging/',
    logging_steps = 50,
    save_steps= 10000,
    remove_unused_columns=False,
)

model = transformers.T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-small').cuda()

trainer = transformers.Trainer(
    model = model,
   args = training_args,
    train_dataset = corpus,
)

```

However, when running trainer.train(), I get the following error:

```auto
*****Running training*****
  Num examples = 74749
  Num Epochs = 12
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 112128
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-28-3435b262f1ae> in <module>
----> 1 trainer.train()

3 frames
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in _prepare_inputs(self, inputs)
   2414 if len(inputs) == 0:
   2415 raise ValueError(
-> 2416 "The batch received was empty, your model won't be able to train on it. Double-check that your "
   2417 f"training dataset contains keys expected by the model: {','.join(self._signature_columns)}."
   2418 )

TypeError: can only join an iterable

```

I have tried converting corpus to a torch Dataset object using the following code I found on a related GitHub intended for this purpose:

```auto
from torch.utils.data import Dataset
 
class HFDataset(Dataset):
    def __init__ (self, dset):
        self.dset = dset

    def __getitem__ (self, idx):
        return self.dset[idx]

    def __len__ (self):
        return len(self.dset)

train_ds = HFDataset(corpus)

```

However, I run into the same “can only join an iterable” issue. I’d really appreciate any help in resolving this!

---

<div class="post-metadata">

**Author:** ![wonderfullymade](https://avatars.discourse-cdn.com/v4/letter/w/ce7236/32.png) [@wonderfullymade](https://discuss.huggingface.co/u/wonderfullymade)\
**Post date:** [April 15, 2024, 11:58am UTC](https://discuss.huggingface.co/t/using-huggingface-transformers-trainer-method-for-hugging-face-datasets/25070/2 "2024-04-15T11:58:28Z")

</div>

Hi, how were you able to fix this error?