# How does one create a pytoch data loader using an interleaved hugging face dataset? **URL:** \ **Category:** Beginners\ **Created:** [August 10, 2023, 6:22pm UTC](https://discuss.huggingface.co/t/how-does-one-create-a-pytoch-data-loader-using-an-interleaved-hugging-face-dataset/50320 "2023-08-10T18:22:55Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 10, 2023, 6:22pm UTC](https://discuss.huggingface.co/t/how-does-one-create-a-pytoch-data-loader-using-an-interleaved-hugging-face-dataset/50320/1 "2023-08-10T18:22:55Z")
When I interleave data sets, get a tokenized batch, feed the batch to the pytorch data loader, I get errors: ```auto # -*- coding: utf-8 -*- """issues with dataloader and custom data sets Automatically generated by Colaboratory. Original file is located at https://colab.research.google.com/drive/1sbs95as_66mtK9VK_vbaE9gLE-Tjof1- """ !pip install datasets !pip install pytorch !pip install transformers token = None batch_size = 10 from datasets import load_dataset import torch from transformers import GPT2Tokenizer, GPT2LMHeadModel tokenizer = GPT2Tokenizer.from_pretrained("gpt2") if tokenizer.pad_token_id is None: tokenizer.pad_token = tokenizer.eos_token probe_network = GPT2LMHeadModel.from_pretrained("gpt2") device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu") probe_network = probe_network.to(device) # -- Get batch from dataset from datasets import load_dataset # path, name = 'brando/debug1_af', 'debug1_af' path, name = 'brando/debug0_af', 'debug0_af' remove_columns = [] dataset = load_dataset(path, name, streaming=True, split="train", token=token).with_format("torch") print(f'{dataset=}') batch = dataset.take(batch_size) # print(f'{next(iter(batch))=}') # - Prepare functions to tokenize batch def preprocess(examples): # gets the raw text batch according to the specific names in table in data set & tokenize return tokenizer(examples["link"], padding="max_length", max_length=128, truncation=True, return_tensors="pt") def map(batch): # apply preprocess to batch to all examples in batch represented as a dataset return batch.map(preprocess, batched=True, remove_columns=remove_columns) tokenized_batch = batch.map(preprocess, batched=True, remove_columns=remove_columns) tokenized_batch = map(batch) # print(f'{next(iter(tokenized_batch))=}') from torch.utils.data import Dataset, DataLoader, SequentialSampler dataset = tokenized_batch print(f'{type(dataset)=}') print(f'{dataset. __class__ =}') print(f'{isinstance(dataset, Dataset)=}') # for i, d in enumerate(dataset): # assert isinstance(d, dict) # # dd = dataset[i] # # assert isinstance(dd, dict) loader_opts = {} classifier_opts = {} # data_loader = DataLoader(dataset, shuffle=False, batch_size=loader_opts.get('batch_size', 1), # num_workers=loader_opts.get('num_workers', 0), drop_last=False, sampler=SequentialSampler(range(512)) ) data_loader = DataLoader(dataset, shuffle=False, batch_size=loader_opts.get('batch_size', 1), num_workers=loader_opts.get('num_workers', 0), drop_last=False, sampler=None) print(f'{iter(data_loader)=}') print(f'{next(iter(data_loader))=}') print('Done\a') ``` with error: ```auto --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map) 126 try: --> 127 return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem}) 128 except TypeError: 9 frames TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found During handling of the above exception, another exception occurred: TypeError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map) 148 return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] 149 --> 150 raise TypeError(default_collate_err_msg_format.format(elem_type)) 151 152 TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found ``` why? And why doesn’t the single data set c4 and wiki-text give this error? Only interleaved data sets? Ideally I don’t want to write my own collate\_function. - colab: [Google Colab](https://colab.research.google.com/drive/1sbs95as_66mtK9VK_vbaE9gLE-Tjof1-?usp=sharing) - related: [python - How does one create a pytorch data loader with a custom hugging face data set without having errors? - Stack Overflow](https://stackoverflow.com/questions/76872115/how-does-one-create-a-pytorch-data-loader-with-a-custom-hugging-face-data-set-wi) - cross: [python - How does one create a pytoch data loader using an interleaved hugging face dataset? - Stack Overflow](https://stackoverflow.com/questions/76878387/how-does-one-create-a-pytoch-data-loader-using-an-interleaved-hugging-face-datas) - related: [HuggingFace dataset: each element in list of batch should be of equal size - #4 by brando](https://discuss.huggingface.co/t/huggingface-dataset-each-element-in-list-of-batch-should-be-of-equal-size/10697/4) ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 10, 2023, 6:41pm UTC](https://discuss.huggingface.co/t/how-does-one-create-a-pytoch-data-loader-using-an-interleaved-hugging-face-dataset/50320/2 "2023-08-10T18:41:51Z")
similar error: [How does one create a pytorch data loader with a custom hugging face data set without having errors? - #2 by brando](https://discuss.huggingface.co/t/how-does-one-create-a-pytorch-data-loader-with-a-custom-hugging-face-data-set-without-having-errors/50204/2) ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 10, 2023, 7:23pm UTC](https://discuss.huggingface.co/t/how-does-one-create-a-pytoch-data-loader-using-an-interleaved-hugging-face-dataset/50320/3 "2023-08-10T19:23:01Z")
> [@HuggingFace dataset: each element in list of batch should be of equal size](https://discuss.huggingface.co/t/huggingface-dataset-each-element-in-list-of-batch-should-be-of-equal-size/10697): > > I’m trying to use HuggingFace’s tokenizers and datasets with a PyTorch dataloader, like so: dataset = load\_dataset( 'wikitext', 'wikitext-2-raw-v1', split='train[:5%]', # take only first 5% of the dataset cache\_dir=cache\_dir) tokenized\_dataset = dataset.map( lambda e: self.tokenizer(e['text'], padding=True, ma… ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [August 18, 2023, 2:49pm UTC](https://discuss.huggingface.co/t/how-does-one-create-a-pytoch-data-loader-using-an-interleaved-hugging-face-dataset/50320/4 "2023-08-18T14:49:33Z")
Make sure the datastes you’re interleaving have the same columns, otherwise the resulting dataset might contain `None` for missing data. Feel free to rename or remove columns if needed.