# Git-base-vatex: input pixel\_value dimension mismatch (blocking issue)

**URL:** <https://discuss.huggingface.co/t/git-base-vatex-input-pixel-value-dimension-mismatch-blocking-issue/55549>\
**Category:** 🤗Transformers\
**Created:** [September 19, 2023, 2:31am UTC](https://discuss.huggingface.co/t/git-base-vatex-input-pixel-value-dimension-mismatch-blocking-issue/55549 "2023-09-19T02:31:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![skar02](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@skar02](https://discuss.huggingface.co/u/skar02)\
**Post date:** [September 19, 2023, 2:31am UTC](https://discuss.huggingface.co/t/git-base-vatex-input-pixel-value-dimension-mismatch-blocking-issue/55549/1 "2023-09-19T02:31:56Z")

</div>

> <https://github.com/huggingface/transformers/issues/26230>
>
> \### System Info
> 
> \- \`transformers\` version: 4.30.2
> \- Platform: Linux-4.19.0-25…-cloud-amd64-x86\_64-with-debian-10.13
> \- Python version: 3.7.12
> \- Huggingface\_hub version: 0.15.1
> \- Safetensors version: 0.3.1 but is ignored because of PyTorch version too old.
> \- PyTorch version (GPU?): 1.9.0+cu111 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using GPU in script?: Yes 
> \- Using distributed or parallel set-up in script?: No 
> 
> \### Who can help?
> 
> @NielsRogge
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> \`\`\`
> from torch.utils.data import Dataset
> import av
> import numpy as np
> import torch 
> from PIL import Image
> from huggingface\_hub import hf\_hub\_download
> from transformers import AutoProcessor, AutoModelForCausalLM
> from generativeimage2text.make\_dataset import create\_video\_captions
> from transformers import AutoProcessor, AutoModelForCausalLM, TrainingArguments, Trainer
> from typing import Union, List
> import json
> import glob
> import os
> import math
> from datasets import load\_dataset
> import shutil 
> from tqdm import tqdm
> from moviepy.editor import VideoFileClip
> from evaluate import load
> processor = AutoProcessor.from\_pretrained("microsoft/git-base-vatex")
> model = AutoModelForCausalLM.from\_pretrained("microsoft/git-base-vatex")
> np.random.seed(45)
> 
> 
> def read\_video\_pyav(container, indices):
> '''
> Decode the video with PyAV decoder.
> Args:
> container (\`av.container.input.InputContainer\`): PyAV container.
> indices (\`List\[int\]\`): List of frame indices to decode.
> Returns:
> result (np.ndarray): np array of decoded frames of shape (num\_frames, height, width, 3).
> '''
> frames = \[\]
> container.seek(0)
> start\_index = indices\[0\]
> end\_index = indices\[-1\]
> for i, frame in enumerate(container.decode(video=0)):
> if i \> end\_index:
> break
> if i \>= start\_index and i in indices:
> frames.append(frame)
> return np.stack(\[x.to\_ndarray(format="rgb24") for x in frames\])
> 
> 
> def sample\_frame\_indices(clip\_len, seg\_len):
> '''
> Sample a given number of frame indices from the video.
> 
> Args:
> clip\_len (\`int\`): Total number of frames to sample.
> seg\_len (\`int\`): Maximum allowed index of sample's last frame.
> 
> Returns:
> indices (\`List\[int\]\`): List of sampled frame indices
> '''
> frame\_sample\_rate = (seg\_len / clip\_len) - 1
> converted\_len = int(clip\_len \* frame\_sample\_rate)
> end\_idx = np.random.randint(converted\_len, seg\_len)
> start\_idx = end\_idx - converted\_len
> indices = np.linspace(start\_idx, end\_idx, num=clip\_len)
> indices = np.clip(indices, start\_idx, end\_idx - 1).astype(np.int64)
> return indices
> 
> class VideoCaptioningDataset(Dataset):
> def \_\_init\_\_(self, videos, captions, processor, num\_frames):
> self.videos = videos 
> self.captions = captions
> self.processor = processor
> self.num\_frames = num\_frames 
> self.cache = {} # to store processed samples
>     
> def \_\_len\_\_(self):
> return len(self.videos)
>     
> def \_\_getitem\_\_(self, idx):
> if idx in self.cache:
> return self.cache\[idx\]
> 
> video\_file = list(self.videos)\[idx\]
> caption = self.captions\[idx\]
>         
> container = av.open(video\_file)
>         
> indices = sample\_frame\_indices(
> clip\_len=self.num\_frames, seg\_len=container.streams.video\[0\].frames
> )
> frames = read\_video\_pyav(container, indices)
>         
> # process the pixel values and caption with the processor
> pixel\_values = self.processor(images=list(frames), return\_tensors="pt").pixel\_values
> # pixel\_values = pixel\_values.squeeze(0)
> inputs = self.processor(text=caption, return\_tensors="pt", padding="max\_length", max\_length=50)
> sample = {
> "pixel\_values": pixel\_values,
> "input\_ids": inputs\["input\_ids"\],
> "attention\_mask": inputs\["attention\_mask"\],
> "labels": inputs\["input\_ids"\],
> }
> return sample
> from sklearn.model\_selection import train\_test\_split
> videos = \['/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_03\_segment\_0.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_0.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_1.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_2.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_3.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_4.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_5.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_6.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_7.mp4', 
> '/home/name/GenerativeImage2Text/generativeimage2text/output\_videos/clip\_07\_segment\_8.mp4'\]
> captions = \['hi', 'hi', 'hi', 'hi', 'hi', 'hi', 'hi', 'hi', 'hi', 'hi'\] # for demo -- the real data is PHI but I can confirm that the video files exist and is in the same format so that isn't the issue. 
> 
> dataset = VideoCaptioningDataset(videos, captions, processor, 6)
> train\_dataset, val\_dataset = train\_test\_split(dataset, test\_size=0.1)
> training\_args = TrainingArguments(
> output\_dir=f"video\_finetune\_1",  
> learning\_rate=5e-5,
> num\_train\_epochs=50, 
> fp16=True,
> per\_device\_train\_batch\_size=4,
> per\_device\_eval\_batch\_size=4,
> gradient\_accumulation\_steps=16,
> save\_total\_limit=3,
> evaluation\_strategy="steps",
> eval\_steps=50,
> save\_strategy="steps",
> save\_steps=50,
> logging\_steps=50,
> remove\_unused\_columns=False,
> push\_to\_hub=False,
> label\_names=\["labels"\],
> load\_best\_model\_at\_end=True
> )
>     
> def compute\_metrics(eval\_pred):
> logits, labels = eval\_pred
> predicted = logits.argmax(-1)
> decoded\_labels = processor.batch\_decode(labels, skip\_special\_tokens=True)
> decoded\_predictions = processor.batch\_decode(predicted, skip\_special\_tokens=True)
> wer\_score = wer.compute(predictions=decoded\_predictions, references=decoded\_labels)
> bleu\_score = bleu.compute(predictions=decoded\_predictions, references=decoded\_labels)
> 
> return {"wer\_score": wer\_score, "bleu\_score": bleu\_score}
> 
> wer = load("wer")
> bleu = load("bleu")
> trainer = Trainer(
> model=model,
> args=training\_args,
> train\_dataset=train\_dataset,
> eval\_dataset=val\_dataset,
> compute\_metrics=compute\_metrics,
> )
> 
> val = train\_dataset\[0\]\["pixel\_values"\].ndim
> print(f"the dim is {val}")
> trainer.train()   
> \`\`\`
> 
> At this point when I do the print the dimension is 5 (as expected). But when I print the dimension of \`\`\`pixel\_values\`\`\` in the first line in \`\`\` forward \`\`\` in file \`\`\` modeling\_git.py" \`\`\` the dimension is 6. Because of this I get error 
> \`\`\` raise ValueError("pixel\_values must be of rank 4 or 5") ValueError: pixel\_values must be of rank 4 or 5 \`\`\`
> 
> This is the full stack trace for reference: 
> 
> \`\`\`
> File "generativeimage2text/video\_finetune.py", line 231, in \<module\>
> trainer.train()
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/transformers/trainer.py", line 1649, in train
> ignore\_keys\_for\_eval=ignore\_keys\_for\_eval,
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/transformers/trainer.py", line 1938, in \_inner\_training\_loop
> tr\_loss\_step = self.training\_step(model, inputs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/transformers/trainer.py", line 2759, in training\_step
> loss = self.compute\_loss(model, inputs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/transformers/trainer.py", line 2784, in compute\_loss
> outputs = model(\*\*inputs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in \_call\_impl
> return forward\_call(\*input, \*\*kwargs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/accelerate/utils/operations.py", line 553, in forward
> return model\_forward(\*args, \*\*kwargs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/accelerate/utils/operations.py", line 541, in \_\_call\_\_
> return convert\_to\_fp32(self.model\_forward(\*args, \*\*kwargs))
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/torch/cuda/amp/autocast\_mode.py", line 141, in decorate\_autocast
> return func(\*args, \*\*kwargs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/transformers/models/git/modeling\_git.py", line 1507, in forward
> return\_dict=return\_dict,
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in \_call\_impl
> return forward\_call(\*input, \*\*kwargs)
> File "/home/name/GenerativeImage2Text/git2/lib/python3.7/site-packages/transformers/models/git/modeling\_git.py", line 1250, in forward
> raise ValueError("pixel\_values must be of rank 4 or 5")
> ValueError: pixel\_values must be of rank 4 or 5
> \`\`\`
> 
> \### Expected behavior
> 
> Ideally the dimension of \`\`\`pixel\_values\`\`\` inside \`\`\`forward\`\`\` would also be 5 and the finetuning of git-base-vatex on video would work This is a blocking issue and any help would be really appreciated!