I would like to finme tune a transformer on images 512x512. I was looking at the huggingpics notebook as a starting point but failed to modify it to train on images with size different from 224x224. I tried adding the “size” to ViTFeatureExtractor.from_pretrained (unknown keyword) and to ViTForImag…

Hi, Yes you can achieve that, by passing interpolate_pos_encoding=True to the forward of the model ( docs ). Explanation: so the Vision Transformer (ViT) was pre-trained on images of resolution 224x224. For instance, google/vit-base-patch16-224 · Hugging Face used images of size 224x224 and a patch …

Thank you so much! now its working! IMHO, as this is definitely not trivial and different to other models, that perhaps a small section on finetuning should be added? as well as feature extraction (it seems to be different, much smaller vectors to, say efficientnet b7 that has 2580 or something and…

Hi, Yes definitely, let me open a PR today to add this! And yes ViT is very much like BERT :wink: it outputs a vector of size 768 for each “patch” (which can be seen as each “word”), whereas a model like ResNet outputs a “feature map” of shape (batch_size, num_channels, height, width).

Umm i am still doing something wrong, I feel, as I can train really large batches (over 128) which shouldn’t be possible even with 24gb of gpu memory and results are about the same as with the 224 I’ll detail what I do: This is the model i use: def get_model(size): …

Hi, I’m findind hard to achieve to set the interporlate_pos_encoding = True by redefining the model’s forward method. Could you make a brief step by step case on how to do so in order to train the model, and not just make a forward pass? I thought it was just about modifying the model.config whic…

I fine tune the model by using the Trainer build class from transformers, not directly by calling the forward method, so I’m not finding the way to set the interpolate_pos_encoding to true in that case.

ValueError: mean must have 4 elements if it is an iterable, got 3 my program is showing above error how to solve it

Same error to me while trying to train an image captioning model (labels size 32, pixel values size [3, 224, 224]) ValueError: mean must have 4 elements if it is an iterable, got 3

@Adhil1123 I got the solution for ValueError: mean must have 4 elements if it is an iterable, got 3 My images were in CMYK mode, so they had 4 channels intead of 3. I had to convert them to RGB with img.convert('RGB') to run the code successfuly .

[image] jorgeili: I’m findind hard to achieve to set the interporlate_pos_encoding = True by redefining the model’s forward method. Hi one can achieve this as explained here: Fine-tuning ViT with more patches/higher resolution - #4 by mohotmoz

Fine tuning image transformer on higher resolution

Beginners

nielsr May 1, 2024, 4:09pm 11

Hi one can achieve this as explained here: Fine-tuning ViT with more patches/higher resolution - #4 by mohotmoz

Topic		Replies	Views
Fine-tuning ViT with more patches/higher resolution Intermediate	3	3961	December 26, 2022
Changing resolution of transformer models for training 🤗Transformers	0	692	September 2, 2022
Pyramid Vision Transformer: Issue with input image size larger than 224 px 🤗Transformers	0	1711	September 15, 2023
How to use Transformers ViTs with different resolutions like in timm? 🤗Transformers	0	127	November 14, 2024
Vision Transformer embeddings interpolation 🤗Transformers	0	411	July 6, 2022

Fine tuning image transformer on higher resolution

Related topics