# Directly load models from a remote storage like S3

**URL:** <https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862>\
**Category:** Amazon SageMaker\
**Created:** [September 9, 2021, 6:29pm UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862 "2021-09-09T18:29:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![leifan](https://avatars.discourse-cdn.com/v4/letter/l/7cd45c/32.png) [@leifan](https://discuss.huggingface.co/u/leifan)\
**Post date:** [September 9, 2021, 6:29pm UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862/1 "2021-09-09T18:29:06Z")

</div>

Hi,  
Instead of download the transformers model to the local file, could we directly read and write models from S3?  
I have tested that we can read csv and txt files directly from S3, but not for models. Is there any solution?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 10, 2021, 6:22am UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862/2 "2021-09-10T06:22:04Z")

</div>

Hey @leifan,

Is your question related to inference or training?  
If it is related to the question you could download a model from s3 when starting the training job the same way as you would do with data.

```python
huggingface_estimator.fit({
    'train': 's3://<my-bucket>/<prefix>/train', # containing train files
    'test': 's3://<my-bucket>/<prefix>/test', # containing test files
    'model': 's3://<another-bucket>/<prefix>/model', # containing model files (config.json, pytroch_model.bin, etc.)
})

```

SageMaker will then download when starting the training job all of these files into your container.  
The Path of the files can either be accessed from the env var `SM_CHANNEL_XXXX`, e.g. `SM_CHANNEL_TRAIN`, `SM_CHANNEL_MODEL` or directly from, e.g. `/opt/ml/input/train`

And then you can load your model in your training script with

```python
AutoModelForXXX.from_pretrained(os.environ.get('SM_CHANNEL_MODEL',None))

```

---

<div class="post-metadata">

**Author:** ![leifan](https://avatars.discourse-cdn.com/v4/letter/l/7cd45c/32.png) [@leifan](https://discuss.huggingface.co/u/leifan)\
**Post date:** [September 10, 2021, 4:31pm UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862/3 "2021-09-10T16:31:40Z")

</div>

Hi, @philschmid,

Thank you so much for your reply.  
My question is related to the training process. I know huggingface has really nice functions for model deployment on SageMaker.

Let me clarify my use-case.

Currently I’m training transformer models (Huggingface) on SageMaker (AWS). I have to copy the model files from S3 buckets to SageMaker and copy the trained models back to S3 after training. This takes a lot of time especially when I have a lot of hyper-tuning experiments and the models are large. For example, if I have 10 trails and in each trail, I save 10 checkpoints. The total size for this single model is 10\times10\times4GB = 400GB (Each model is 3 to 5 GB).

As you can see, it takes a long time to transfer model files back and forth. So I’m considering if Pytorch allow reading and writing models directly from S3, so that we can skip the step of storing the files locally. This will greatly reduce the overall time it costs.

I heard from the tech support of my company that Pytorch does not support loading models from remote storage. It only allows loading models locally. I wish to confirm that and if it is true, could we build a new feature for remote storage and loading?

Thanks.

Best,

Lei

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 13, 2021, 7:46am UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862/4 "2021-09-13T07:46:25Z")

</div>

Yes, there are at least three options on how to improve this.

**Option 1: Use EFS/FSx instead of S3**

> Amazon SageMaker supports using Amazon Elastic File System (EFS) and FSx for Lustre as data sources to use during training.

[https://sagemaker.readthedocs.io/en/stable/overview.html?highlight=efs#use-file-systems-as-training-inputs](https://sagemaker.readthedocs.io/en/stable/overview.html?highlight=efs#use-file-systems-as-training-inputs)

That way you can continuously save your checkpoints and log files to the filesystem as than uploading it at the end to s3.

**Option 2: Use S3 Checkpointing for uploads**

> After you enable checkpointing, SageMaker saves checkpoints to Amazon S3 and syncs your training job with the checkpoint S3 bucket.

> **[Use Checkpoints in Amazon SageMaker - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html#model-checkpoints-enable)**
>
> Use checkpoints in Amazon SageMaker to save the state of models. Training can be resumed from the saved checkpoints.

When checkpointing is enabled sagemaker automatically asynchronously uploads every artifact written to `checkpoint_local_path` during Training.

**Option 3: Use the Hugging Face Hub**

You can use `push_to_hub` method to save your artifacts asynchronously to the hugging face hub into a repository.

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [November 18, 2022, 3:13pm UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862/5 "2022-11-18T15:13:29Z")

</div>

**Is there an example of how to use the checkpointing with Huggingface estimator?**

Would it look something like:

```auto
huggingface_estimator = HuggingFace(
        entry_point='train.py',
        source_dir='./scripts',
        instance_type='ml.p3.2xlarge',
        checkpoint_s3_uri="s3://mybucket/",
        instance_count=1,
        role=role,
        transformers_version='4.4',
        pytorch_version='1.6',
        py_version='py36',
        hyperparameters = hyperparameters
)

```

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [November 18, 2022, 4:20pm UTC](https://discuss.huggingface.co/t/directly-load-models-from-a-remote-storage-like-s3/9862/7 "2022-11-18T16:20:02Z")

</div>

Yes that’s right. The HF Estimator inherits from the Estimator class, so the usage would be the same.