# Why do I get UnboundLocalError: local variable 'batch\_idx' referenced before assignment when using interleaved data sets with Hugging Face (HF)?

**URL:** <https://discuss.huggingface.co/t/why-do-i-get-unboundlocalerror-local-variable-batch-idx-referenced-before-assignment-when-using-interleaved-data-sets-with-hugging-face-hf/69573>\
**Category:** Beginners\
**Created:** [January 18, 2024, 3:59am UTC](https://discuss.huggingface.co/t/why-do-i-get-unboundlocalerror-local-variable-batch-idx-referenced-before-assignment-when-using-interleaved-data-sets-with-hugging-face-hf/69573 "2024-01-18T03:59:52Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [January 18, 2024, 11:31pm UTC](https://discuss.huggingface.co/t/why-do-i-get-unboundlocalerror-local-variable-batch-idx-referenced-before-assignment-when-using-interleaved-data-sets-with-hugging-face-hf/69573/3 "2024-01-18T23:31:18Z")

</div>

The issue happened because llama2 does not have a max sequence length set. So it defaults to max int `1000000000000000019884624838656 `. You can set it manually if you google the max seq len for your model e.g., for llama2-7b:

```python
    # - Get tokenized train data set
    # Note: Setting `batched=True` in the `dataset.map` function of Hugging Face's datasets library processes the data in batches rather than one item at a time, significantly speeding up the tokenization and preprocessing steps.
    tokenized_train_datasets = raw_train_datasets.map(tokenize_function, batched=True, remove_columns=remove_columns)
    # block_size: int = tokenizer.model_max_length
    block_size: int = 4096

```

That should remove the issue in the code because

```python
    total_length = (total_length // block_size) * block_size

```

will not be zero when trying to concatenate all 1000 texts in the batch when forming a batch all of which have a length of block size (assumign block size is max seq len or some value you chose).

ref: [[Tokenizers]What this max\_length number?](https://discuss.huggingface.co/t/tokenizers-what-this-max-length-number/28484)

---

_[View the full topic](https://discuss.huggingface.co/t/why-do-i-get-unboundlocalerror-local-variable-batch-idx-referenced-before-assignment-when-using-interleaved-data-sets-with-hugging-face-hf/69573)._