# Eval with trainer not running with PEFT LoRA model

**URL:** <https://discuss.huggingface.co/t/eval-with-trainer-not-running-with-peft-lora-model/53286>\
**Category:** 🤗Transformers\
**Created:** [September 2, 2023, 8:16pm UTC](https://discuss.huggingface.co/t/eval-with-trainer-not-running-with-peft-lora-model/53286 "2023-09-02T20:16:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![oran-sh](https://avatars.discourse-cdn.com/v4/letter/o/a4c791/32.png) [@oran-sh](https://discuss.huggingface.co/u/oran-sh)\
**Post date:** [September 2, 2023, 8:16pm UTC](https://discuss.huggingface.co/t/eval-with-trainer-not-running-with-peft-lora-model/53286/1 "2023-09-02T20:16:15Z")

</div>

Hi there,

I have a training script that when running with defined model and trainer, the evaluation loop works fine.  
When wrapping the model with PEFT LoRA config evaluation doesn’t run. The training is completed properly but the evaluation loop every X steps just doesn’t seem to happen.  
When I dug in a bit I saw it happened because for some reason in PEFT mode the trainer didn’t provide labels to the evaluation loop so it was skipped.  
Here’s the package versions and script:

```auto
torch==2.0.1
transformers==4.32.1
peft==0.5.0

```

Original code:

```auto
class CLIPForSceneClassification(CLIPModel):

    def forward(self,
        pixel_values,
        labels,
        text_inputs):

        s = pixel_values.shape # B,S,3,H,W
        pixel_values = pixel_values.view([s[0]*s[1],s[2],s[3],s[4]])
        outputs = self.vision_model(pixel_values=pixel_values)
        outputs = outputs[1].view([s[0],s[1],-1])
        image_embeds = torch.mean(outputs,dim=1)

        text_outputs = self.text_model(**text_inputs)
        text_embeds = text_outputs[1]

        image_embeds = self.visual_projection(image_embeds)
        text_embeds = self.text_projection(text_embeds)

        # normalized features
        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
        text_embeds = text_embeds / text_embeds.norm(p=2, dim=-1, keepdim=True)

        # cosine similarity as logits
        logit_scale = self.logit_scale.exp()
        logits = torch.matmul(image_embeds, text_embeds.t()) * logit_scale
        # logits_per_text = logits_per_image.t()

        loss = None
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.n_classes), labels.view(-1))

        return loss, logits

def compute_metrics(pred):
        labels = pred.label_ids
        preds = pred.predictions
        preds = preds.argmax(-1)
        precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average="weighted")
        acc = accuracy_score(labels, preds)
        return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}

model = CLIPForSceneClassification.from_pretrained(args.model_name)

training_args = TrainingArguments(
    output_dir=args.model_dir,
    num_train_epochs=args.epochs,
    per_device_train_batch_size=args.train_batch_size,
    per_device_eval_batch_size=args.eval_batch_size,
    warmup_steps=args.warmup_steps,
    evaluation_strategy="steps",
    eval_steps=50,
    logging_dir=f"{args.output_data_dir}/logs",
    learning_rate=float(args.learning_rate),
    # label_names=["real", "animated"],
    report_to="none",
    use_mps_device=torch.backends.mps.is_available(),
    logging_steps=10,
    save_strategy="steps",
    save_steps=50,
    dataloader_num_workers=2,
    remove_unused_columns=False,
)

# create Trainer instance
trainer = Trainer(
    model=lora_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)

```

Here’s what I modified with PEFT:

```auto
config = LoraConfig(
    r=4,
    lora_alpha=4,
    lora_dropout=0.0,
    target_modules=["visual_projection", "text_projection"],
    # modules_to_save=["seq.4"],
)
lora_model = get_peft_model(model, config)
lora_model.print_trainable_parameters()

...

# create Trainer instance
trainer = Trainer(
    model=lora_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)

```

Would be happy for help here. Thanks!!!

---

<div class="post-metadata">

**Author:** ![suzyahyah](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@suzyahyah](https://discuss.huggingface.co/u/suzyahyah)\
**Post date:** [September 10, 2023, 4:55am UTC](https://discuss.huggingface.co/t/eval-with-trainer-not-running-with-peft-lora-model/53286/2 "2023-09-10T04:55:02Z")

</div>

They haven’t made PEFT compatible with the [`find_labels`](https://github.com/huggingface/transformers/blob/95b374952dc27d8511541d6f5a4e22c9ec11fb24/src/transformers/utils/generic.py#L482) function yet.

The easiest fix should be to add this argument in training args.

```auto
from transformers import TrainingArguments
training_args = TrainingArguments(...,label_names=['labels'])

```