# Compute Perplexity using compute\_metrics in SFTTrainer

**URL:** <https://discuss.huggingface.co/t/compute-perplexity-using-compute-metrics-in-sfttrainer/74974>\
**Category:** Intermediate\
**Created:** [February 27, 2024, 11:34am UTC](https://discuss.huggingface.co/t/compute-perplexity-using-compute-metrics-in-sfttrainer/74974 "2024-02-27T11:34:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shenoyajith](https://avatars.discourse-cdn.com/v4/letter/s/47e85d/32.png) [@shenoyajith](https://discuss.huggingface.co/u/shenoyajith)\
**Post date:** [February 27, 2024, 11:34am UTC](https://discuss.huggingface.co/t/compute-perplexity-using-compute-metrics-in-sfttrainer/74974/1 "2024-02-27T11:34:34Z")

</div>

How can I compute perplexity as a metric when using the SFTTrainer and log at end of each epoch, by using that in compute\_metrics argument. I intend to pick the best checkpoint with least perplexity.

Here is the dimension of logits and labels that go into the compute\_metrics function (50, 256, 50272) (total\_records,seq\_len\_vocab\_size).  
and labels (50, 256).

How can I compute perplexity using a compute metrics function for a CasualLM task?

> dataset = load\_dataset(“imdb”)  
> trainer = SFTTrainer(  
> “facebook/opt-350m”,  
> train\_dataset=dataset[“train”].select(range(50)),  
> eval\_dataset = dataset[“test”].select(range(50)),  
> dataset\_text\_field=“text”,  
> max\_seq\_length=256,  
> compute\_metrics = #logic goes here,  
> args = training\_args,  
> )  
> trainer.train()

---

<div class="post-metadata">

**Author:** ![Pragyan-02](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@Pragyan-02](https://discuss.huggingface.co/u/Pragyan-02)\
**Post date:** [January 22, 2025, 7:16am UTC](https://discuss.huggingface.co/t/compute-perplexity-using-compute-metrics-in-sfttrainer/74974/2 "2025-01-22T07:16:40Z")

</div>

Hi, I wrote this function to log perplexity at each logging step.

def compute\_metrics(pred: EvalPrediction):  
predictions = torch.tensor(pred[0])

```
targets = torch.tensor(pred[1])

perplexity = Perplexity(ignore_index=-100)

perplexity_score = perplexity(predictions, targets)

return {"Perplexity": perplexity_score}

```

In trainer called like this:

```
trainer = SFTTrainer(
    **trainer_args,
    train_dataset=train_data,
    eval_dataset=valid_data,
    peft_config=peft_config ,
    dataset_text_field="text",
    max_seq_length=config.block_size,
    compute_metrics = utils.compute_metrics)

```