# VideoMAE Pretrain Batch Masking **URL:** \ **Category:** Models\ **Created:** [August 26, 2022, 6:06pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176 "2022-08-26T18:06:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![aljipa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aljipa/32/5815_2.png) [@aljipa](https://discuss.huggingface.co/u/aljipa)\ **Post date:** [August 26, 2022, 6:06pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/1 "2022-08-26T18:06:15Z")
I have a question about whether it’s possible to use multiple masks in a single micro-batch of size \>1 for VideoMAE pre-training. In the [current code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/videomae/modeling_videomae.py#L146) it seems that when inverse mask ([B, P], where P number of patches) is applied to batched embeddings ([B, P, D], where P number of patches, D number of features) we will lose the batch dimension ([P, D], all batch entries’ visible patches will be clamped together) and we will get this error when the number of patches for batch entries differ and we cannot reshape: `{RuntimeError}shape '[B, -1, D]' is invalid for input of size X` Or are we supposed to use the same mask for the micro-batch to make sure dimensions are the same? Thank you! ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [September 26, 2022, 10:21am UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/2 "2022-09-26T10:21:03Z")
Thanks for reporting this, I will take a look. ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [September 26, 2022, 10:59am UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/3 "2022-09-26T10:59:54Z")
I’ve taken a look, this runs fine for me: ```auto from transformers import VideoMAEFeatureExtractor, VideoMAEForPreTraining import torch model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") # assume the batch size is 2 num_frames = 16 pixel_values = torch.randn(2,num_frames,3,224,224) num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame bool_masked_pos_1 = torch.randint(0, 2, (1, seq_length)).bool() bool_masked_pos_2 = torch.randint(0, 2, (1, seq_length)).bool() bool_masked_pos = torch.cat([bool_masked_pos_1, bool_masked_pos_2]) outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) loss = outputs.loss ``` ---
**Author:** ![anunez](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anunez](https://discuss.huggingface.co/u/anunez)\ **Post date:** [September 26, 2022, 12:52pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/4 "2022-09-26T12:52:22Z")
Thank you for the code! I executed it on my environment and it still says: `RuntimeError: shape '[2, -1, 768]' is invalid for input of size 1209600` Pytorch version 1.9.0 and transformers version 4.22.1. ---
**Author:** ![aljipa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aljipa/32/5815_2.png) [@aljipa](https://discuss.huggingface.co/u/aljipa)\ **Post date:** [September 26, 2022, 1:13pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/5 "2022-09-26T13:13:56Z")
Hi, as far as I understand there’s a chance that these two masks could end up being of the same masked/unmasked ratio. Which is why it may or may not work based on chance. ---
**Author:** ![aljipa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aljipa/32/5815_2.png) [@aljipa](https://discuss.huggingface.co/u/aljipa)\ **Post date:** [September 26, 2022, 1:15pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/6 "2022-09-26T13:15:54Z")
I was able to make this work for batches using this snippet: ```python # e.g. mask_ratio = 0.9 bool_masked_pos = np.ones(seq_length) mask_num = math.ceil(seq_length * mask_ratio) mask = np.random.choice(seq_length, mask_num, replace=False) bool_masked_pos[mask] = 0 ``` Basically making sure we mask the same number of pixels across all images. ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [September 26, 2022, 1:37pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/7 "2022-09-26T13:37:50Z")
Oh yes that’s indeed required. The mask ratio needs to be the same for each video in a batch. Note that the VideoMAE authors trained the model like that. The code snippet is also used in the [original implementation](https://github.com/MCG-NJU/VideoMAE/blob/18f667ca7f8fa3835955328ecc40a1046d81ecae/modeling_pretrain.py#L90). ---
**Author:** ![anunez](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anunez](https://discuss.huggingface.co/u/anunez)\ **Post date:** [September 26, 2022, 3:32pm UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/8 "2022-09-26T15:32:52Z")
Thank you for the solution @aljipa! I guess `bool_masked_pos` needs to be casted to boolean and stack “batch size” `bool_masked_pos` elements, i.e., following your code snippet: ```auto batch_size = 2 bool_masked_pos = np.ones(seq_length) mask_num = math.ceil(seq_length * mask_ratio) mask = np.random.choice(seq_length, mask_num, replace=False) bool_masked_pos[mask] = 0 # Torch and bool cast, extra dimension added for concatenation bool_masked_pos = torch.as_tensor(bool_masked_pos).bool().unsqueeze(0) bool_masked_pos = torch.cat([bool_masked_pos for _ in range(batch_size)]) ``` So that for all the elements within the batch, the mask is the same as mentioned by @nielsr. ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [September 27, 2022, 6:40am UTC](https://discuss.huggingface.co/t/videomae-pretrain-batch-masking/22176/9 "2022-09-27T06:40:31Z")
> [@anunez](#): > > So that for all the elements within the batch, the mask is the same as mentioned by @nielsr. No that’s not necessary, I said that the mask ratio needs to be the same (like 0.9), not necessarily the mask itself. The boolean mask can differ between examples in a batch.