# Combining encoder from one model and a decoder for another for image reconstruction

**URL:** <https://discuss.huggingface.co/t/combining-encoder-from-one-model-and-a-decoder-for-another-for-image-reconstruction/27836>\
**Category:** Beginners\
**Created:** [December 15, 2022, 3:01am UTC](https://discuss.huggingface.co/t/combining-encoder-from-one-model-and-a-decoder-for-another-for-image-reconstruction/27836 "2022-12-15T03:01:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![spencerh](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@spencerh](https://discuss.huggingface.co/u/spencerh)\
**Post date:** [December 15, 2022, 3:01am UTC](https://discuss.huggingface.co/t/combining-encoder-from-one-model-and-a-decoder-for-another-for-image-reconstruction/27836/1 "2022-12-15T03:01:49Z")

</div>

Hi! I’m learning the _way of the land_ around Hugging Face code, specifically reading the code behind LayoutLMV3 and MAE.

If my objective is to do a pre-training of masked image reconstruction, much like what ViT-MAE decoder does, but with the encoder from another model like LayoutLMv3, can I modify:

```nohighlight
class ViTMAEForPreTraining(ViTMAEPreTrainedModel):
    def __init__ (self, config):
        super(). __init__ (config)
        self.config = config
        
        // change the following original code from
        // self.vit = ViTMAEModel(config)
        // to instead 
        self.encoder = Encoder(config) 
        // where Encoder is derived from 
        // class LayoutLMv3Model(LayoutLMv3PreTrainedModel) for example
        self.decoder = ViTMAEDecoder(config, num_patches=self.vit.embeddings.num_patches)

        # Initialize weights and apply final processing
        self.post_init()

```

My initial hunch is that because LayoutLMv3 returns a BaseModelOutput while the encoder block of MAE returns a VitMAEModelOutput which includes ids\_restore, it won’t work. Does this mean we will also have to create a new ModelOutput class?

If it is not as simple as this, then do I need to create a new Embedding class for VitMAE that takes in encoding like LayoutLMV3? Any help would be greatly appreciated, thanks!