# Static Quantization with Own dataset

**URL:** <https://discuss.huggingface.co/t/static-quantization-with-own-dataset/41675>\
**Category:** 🤗Optimum\
**Created:** [May 31, 2023, 4:15pm UTC](https://discuss.huggingface.co/t/static-quantization-with-own-dataset/41675 "2023-05-31T16:15:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![aroger](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@aroger](https://discuss.huggingface.co/u/aroger)\
**Post date:** [May 31, 2023, 4:15pm UTC](https://discuss.huggingface.co/t/static-quantization-with-own-dataset/41675/1 "2023-05-31T16:15:37Z")

</div>

Hi! I want to apply static quantization on my model using my own dataset for calibration.  
There is no special explanation on how to use get\_calibration\_dataset with “external dataset”. As it’s not the same than with Hugging Face dataset, i don’t get if i need to convert my dataset to a special format or specify any configuration. I’ve tried with the path to my csv, even tried in json.

update : it seems to be mandatory to use a dataset from datasets. As load\_dataset is called in the function and args seem to be for HG dataset (no data\_files argument available)

Will appreciate any info

thanks!

---

<div class="post-metadata">

**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\
**Post date:** [June 1, 2023, 1:40pm UTC](https://discuss.huggingface.co/t/static-quantization-with-own-dataset/41675/2 "2023-06-01T13:40:49Z")

</div>

hi @aroger, would this help: [Load tabular data](https://huggingface.co/docs/datasets/v2.12.0/en/tabular_load#csv-files)?

Then you could follow the examples for calibration, e.g. [optimum/run\_glue.py at main · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/blob/main/examples/onnxruntime/quantization/text-classification/run_glue.py)

---

<div class="post-metadata">

**Author:** ![aroger](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@aroger](https://discuss.huggingface.co/u/aroger)\
**Post date:** [June 1, 2023, 1:47pm UTC](https://discuss.huggingface.co/t/static-quantization-with-own-dataset/41675/3 "2023-06-01T13:47:41Z")

</div>

hi @fxmarty,  
The problem is that in the source code of quantizer.get\_calibration\_dataset() load\_dataset is set like :

```auto
calib_dataset = load_dataset(
            dataset_name,
            name=dataset_config_name,
            split=dataset_split,
            use_auth_token=use_auth_token,
        )

```

as you can see it’s not possible to use what is explained in [Load tabular data](https://huggingface.co/docs/datasets/v2.12.0/en/tabular_load#csv-files).

In the example, they use glue which is easier but i want to calibrate my quantization on my own dataset as it matches more my requirements and my use case.

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](https://discuss.huggingface.co/u/regisss)\
**Post date:** [July 1, 2023, 9:59am UTC](https://discuss.huggingface.co/t/static-quantization-with-own-dataset/41675/4 "2023-07-01T09:59:11Z")

</div>

Hi @aroger, so if `dataset_name="csv"` and a way to provide the `data_files` argument in `get_calibration_dataset` is provided that should work. Would you like to open a PR to enable this?