# Implementing Triplet loss in Vit

**URL:** <https://discuss.huggingface.co/t/implementing-triplet-loss-in-vit/157754>\
**Category:** 🤗Transformers\
**Created:** [June 2, 2025, 8:34pm UTC](https://discuss.huggingface.co/t/implementing-triplet-loss-in-vit/157754 "2025-06-02T20:34:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AnanthZeke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ananthzeke/32/48678_2.png) [@AnanthZeke](https://discuss.huggingface.co/u/AnanthZeke)\
**Post date:** [June 2, 2025, 8:34pm UTC](https://discuss.huggingface.co/t/implementing-triplet-loss-in-vit/157754/1 "2025-06-02T20:34:23Z")

</div>

I’m currently working on training a ViT-MSN (Masked Siamese Network) model using triplet loss for a self-supervised or metric learning task. I’m using Hugging Face’s Trainer class for training, and I would like to integrate a miner (like those from pytorch-metric-learning) to improve the selection of positive and negative samples during training.

Here are the key challenges I’m facing:

1. How to integrate triplet loss into a Hugging Face Trainer setup?  
I understand that I might need to subclass the Trainer or override the compute\_loss method, but I’m not entirely sure how to best pass the anchor, positive, and negative embeddings properly when working with a ViT-MSN model.

2. How to use a miner within the Hugging Face Trainer class?  
I’d like to use a miner like BatchHardMiner or TripletMarginMiner from pytorch-metric-learning. What’s the best practice for integrating such miners in the Hugging Face training loop, especially when customizing loss computation?

If anyone has done something similar or has suggestions/examples on how to:

structure the input data,

modify the Trainer to work with triplet loss and miners,

and properly align it with ViT-MSN architecture,

your input would be greatly appreciated!

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 3, 2025, 12:18am UTC](https://discuss.huggingface.co/t/implementing-triplet-loss-in-vit/157754/2 "2025-06-03T00:18:20Z")

</div>

I think overriding the function is the most reliable way.

> [@Specify Loss for Trainer / TrainingArguments](https://discuss.huggingface.co/t/specify-loss-for-trainer-trainingarguments/10481):
>
> I’d like to fine-tune for a regression task rather than a classification task. How do I change the default loss in either TrainingArguments or Trainer()?

[https://stackoverflow.com/questions/66302371/how-to-specify-the-loss-function-when-finetuning-a-model-using-the-huggingface-t](https://stackoverflow.com/questions/66302371/how-to-specify-the-loss-function-when-finetuning-a-model-using-the-huggingface-t)

> [@Trainer.evaluate() doesn't return evaluation loss](https://discuss.huggingface.co/t/trainer-evaluate-doesnt-return-evaluation-loss/140841):
>
> I am using Seq2SeqTrainer to lora finetune madlad 3b model using my own dataset. When I run trainer.evaluate(), the evaluation loss is not printing. Code: output\_dir = "madlad\_run\_5" training\_args = Seq2SeqTrainingArguments( output\_dir, per\_device\_train\_batch\_size=32, # per\_device\_eval\_batch\_size=2, num\_train\_epochs=1, # max\_steps=1000, # gradient\_accumulation\_steps=8, bf16=True, torch\_compile=True, # gradient\_checkpointing=True, # torch\_empty\_cache\_st…