# Blip2 with a new LLM

**URL:** <https://discuss.huggingface.co/t/blip2-with-a-new-llm/50855>\
**Category:** Intermediate\
**Created:** [August 15, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/blip2-with-a-new-llm/50855 "2023-08-15T18:21:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![inkasaras](https://avatars.discourse-cdn.com/v4/letter/i/4491bb/32.png) [@inkasaras](https://discuss.huggingface.co/u/inkasaras)\
**Post date:** [August 15, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/blip2-with-a-new-llm/50855/1 "2023-08-15T18:21:26Z")

</div>

Hey,

I would like to add a new LLM to a Blip2 model. Different from the already pre-trained ones, like Vicuma, OPT or FlanT5.

For that, I’m loading the Blip2 model one piece at a time. So I’m loading the Vision model first then the Q Former, and finally, I would like to load the LLM. I followed the instructions [here](https://huggingface.co/docs/transformers/main/model_doc/blip-2#transformers.Blip2VisionModel), but now I’m at a loss I don’t know how to connect these 3 parts or what inputs go into each other. If any one has any idea it would be much appreciated.

Code to load the model from hugging face

```auto
from transformers import Blip2VisionConfig, Blip2VisionModel

# Initializing a Blip2VisionConfig with Salesforce/blip2-opt-2.7b style configuration
vit_configuration = Blip2VisionConfig()

# Initializing a Blip2VisionModel (with random weights) 
vit_model = Blip2VisionModel(vit_configuration)

# Accessing the model configuration
vit_configuration_f = vit_model.config

```

Code for loading the Q Former

```auto
from transformers import Blip2QFormerConfig, Blip2QFormerModel

# Initializing a BLIP-2 Salesforce/blip2-opt-2.7b style configuration
q_configuration = Blip2QFormerConfig()

# Initializing a model (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
q_model = Blip2QFormerModel(q_configuration)
# Accessing the model configuration
q_configuration_f = q_model.config

```