# Reproducibility problem while using GPT-2

**URL:** <https://discuss.huggingface.co/t/reproducibility-problem-while-using-gpt-2/9235>\
**Category:** 🤗Transformers\
**Created:** [August 14, 2021, 12:16pm UTC](https://discuss.huggingface.co/t/reproducibility-problem-while-using-gpt-2/9235 "2021-08-14T12:16:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshayaravinnakshan](https://avatars.discourse-cdn.com/v4/letter/a/c77e96/32.png) [@akshayaravinnakshan](https://discuss.huggingface.co/u/akshayaravinnakshan)\
**Post date:** [August 14, 2021, 12:16pm UTC](https://discuss.huggingface.co/t/reproducibility-problem-while-using-gpt-2/9235/1 "2021-08-14T12:16:54Z")

</div>

Hi folks,  
I am working on a classification model using GPT2 with GPT2ForSequenceClassification. But I am facing a problem with reproducibility.  
def seed\_worker(worker\_id):

```
worker_seed = torch.initial_seed() % 2**32

numpy.random.seed(worker_seed)

random.seed(worker_seed)

```

### To get reproducability I have used torch.manual\_seed() to seed the RNG for all devices (both CPU and CUDA)

torch.manual\_seed(110)

### To get reproducability remove dataloader#s randomness

g = torch.Generator()

g.manual\_seed(110)

> # Move pytorch dataset into dataloader.
> 
> train\_dataloader = DataLoader(dataset, batch\_size=batch\_size, shuffle=True ,collate\_fn=gpt2\_classificaiton\_collator,num\_workers=0,worker\_init\_fn=seed\_worker,generator=g)
> 
> # Create pytorch dataset.
> 
> valid\_dataset = MovieReviewsDataset(train\_dataset,train=False, use\_tokenizer=tokenizer)
> 
> # Move pytorch dataset into dataloader.
> 
> valid\_dataloader = DataLoader(valid\_dataset, batch\_size=batch\_size, shuffle=True, collate\_fn=gpt2\_classificaiton\_collator,num\_workers=0,worker\_init\_fn=seed\_worker,generator=g)

I also did **set\_seed(110)**. But my code always provides me a new accuracy score for each training cycle. please help