# Using evaluate.evaluator on a PEFT model

**URL:** <https://discuss.huggingface.co/t/using-evaluate-evaluator-on-a-peft-model/76864>\
**Category:** Beginners\
**Created:** [March 11, 2024, 10:00am UTC](https://discuss.huggingface.co/t/using-evaluate-evaluator-on-a-peft-model/76864 "2024-03-11T10:00:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![countermode](https://avatars.discourse-cdn.com/v4/letter/c/5e9695/32.png) [@countermode](https://discuss.huggingface.co/u/countermode)\
**Post date:** [March 11, 2024, 10:00am UTC](https://discuss.huggingface.co/t/using-evaluate-evaluator-on-a-peft-model/76864/1 "2024-03-11T10:00:08Z")

</div>

For an exercise I am experimenting with PEFT on GPT-2. One task is to evaluate the performance. For this I chose evaluate.evaluator, which worked fine on plain GPT-2. However, when trying to evaluate the fine-tuned model like this

```auto
task_evaluator = evaluator('text-classification')
eval_results = task_evaluator.compute(
    model_or_pipeline=lora_model,
    tokenizer=tokenizer,
    data=ds_test,
    input_column=dataset_textfield_name,
    metric=evaluate.combine(['accuracy', 'f1']),
    label_mapping=label2id
)

```

I got the following error:

> AttributeError Traceback (most recent call last)  
> File ~/.local/lib/python3.10/site-packages/peft/peft\_model.py:529, in PeftModel. **getattr** (self, name)  
> 528 try:  
> → 529 return super(). **getattr** (name) # defer to nn.Module’s logic  
> 530 except AttributeError:
> 
> File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module. **getattr** (self, name)  
> 1613 return modules[name]  
> → 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(  
> 1615 type(self). **name** , name))
> 
> AttributeError: ‘PeftModelForSequenceClassification’ object has no attribute ‘task’
> 
> During handling of the above exception, another exception occurred:
> 
> AttributeError Traceback (most recent call last)  
> File ~/.local/lib/python3.10/site-packages/peft/tuners/lora/model.py:273, in LoraModel. **getattr** (self, name)  
> 272 try:  
> → 273 return super(). **getattr** (name) # defer to nn.Module’s logic  
> 274 except AttributeError:
> 
> File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module. **getattr** (self, name)  
> 1613 return modules[name]  
> → 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(  
> 1615 type(self). **name** , name))
> 
> AttributeError: ‘LoraModel’ object has no attribute ‘task’
> 
> During handling of the above exception, another exception occurred:
> 
> AttributeError Traceback (most recent call last)  
> Cell In[19], line 1  
> ----\> 1 eval\_results = task\_evaluator.compute(  
> 2 model\_or\_pipeline=lora\_model,  
> 3 tokenizer=tokenizer,  
> 4 data=ds\_test,  
> 5 input\_column=dataset\_textfield\_name,  
> 6 metric=evaluate.combine([‘accuracy’, ‘f1’]),  
> 7 label\_mapping=label2id  
> 8 )  
> 9 print(eval\_results)
> 
> File ~/.local/lib/python3.10/site-packages/evaluate/evaluator/text\_classification.py:134, in TextClassificationEvaluator.compute(self, model\_or\_pipeline, data, subset, split, metric, tokenizer, feature\_extractor, strategy, confidence\_level, n\_resamples, device, random\_state, input\_column, second\_input\_column, label\_column, label\_mapping)  
> 130 data = self.load\_data(data=data, subset=subset, split=split)  
> 131 metric\_inputs, pipe\_inputs = self.prepare\_data(  
> 132 data=data, input\_column=input\_column, second\_input\_column=second\_input\_column, label\_column=label\_column  
> 133 )  
> → 134 pipe = self.prepare\_pipeline(  
> 135 model\_or\_pipeline=model\_or\_pipeline,  
> 136 tokenizer=tokenizer,  
> 137 feature\_extractor=feature\_extractor,  
> 138 device=device,  
> 139 )  
> 140 metric = self.prepare\_metric(metric)  
> 142 # Compute predictions
> 
> File ~/.local/lib/python3.10/site-packages/evaluate/evaluator/base.py:474, in Evaluator.prepare\_pipeline(self, model\_or\_pipeline, tokenizer, feature\_extractor, device)  
> 472 if tokenizer is not None and feature\_extractor is not None:  
> 473 logger.warning(“Ignoring the value of the preprocessor argument (`tokenizer` or `feature_extractor`).”)  
> → 474 if (pipe.task != self.task) and not (self.task == “translation” and pipe.task.startswith(“translation”)):  
> 475 raise ValueError(  
> 476 f"Incompatible `model_or_pipeline`. Please specify `model_or_pipeline` compatible with the `{self.task}` task."  
> 477 )  
> 478 return pipe
> 
> File ~/.local/lib/python3.10/site-packages/peft/peft\_model.py:531, in PeftModel. **getattr** (self, name)  
> 529 return super(). **getattr** (name) # defer to nn.Module’s logic  
> 530 except AttributeError:  
> → 531 return getattr(self.base\_model, name)
> 
> File ~/.local/lib/python3.10/site-packages/peft/tuners/lora/model.py:275, in LoraModel. **getattr** (self, name)  
> 273 return super(). **getattr** (name) # defer to nn.Module’s logic  
> 274 except AttributeError:  
> → 275 return getattr(self.model, name)
> 
> File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module. **getattr** (self, name)  
> 1612 if name in modules:  
> 1613 return modules[name]  
> → 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(  
> 1615 type(self). **name** , name))
> 
> AttributeError: ‘GPT2ForSequenceClassification’ object has no attribute ‘task’

What should I do to use evaluate.evaluator for fine-tuned models?

---

<div class="post-metadata">

**Author:** ![ivohanke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ivohanke/32/25671_2.png) [@ivohanke](https://discuss.huggingface.co/u/ivohanke)\
**Post date:** [April 10, 2024, 4:47pm UTC](https://discuss.huggingface.co/t/using-evaluate-evaluator-on-a-peft-model/76864/2 "2024-04-10T16:47:15Z")

</div>

@countermode Could you find the error? Have the same here…