# Repeating eval-F1 scores with seed + data randomization

**URL:** <https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377>\
**Category:** Intermediate\
**Created:** [July 1, 2025, 7:31pm UTC](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377 "2025-07-01T19:31:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkabir](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@rkabir](https://discuss.huggingface.co/u/rkabir)\
**Post date:** [July 1, 2025, 7:31pm UTC](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377/1 "2025-07-01T19:31:53Z")

</div>

I was finetuning Roberta on a multi-label classification problem 10 times and keeping track of each evaluation F1 score. However, even when randomizing the selection of training and testing data, as well as seed numbers (i.e. transformers.set\_seed, numpy, tf, tf.keras, random), **I find that towards the end the evaluation F1 scores repeat**. **I have an example of a run through below**.

Any advice? The number of sentences used to train is 250, and the number used to evaluate is 250. The number used to test at the end is 200. All of these are randomly selected from a source of 5000+ sentences.

1. {‘eval\_loss’: 0.5038333535194397, ‘eval\_f1’: 0.8596491228070176, ‘eval\_recall’: 0.8966666666666667, ‘eval\_accuracy’: 0.885, ‘eval\_runtime’: 0.1429, ‘eval\_samples\_per\_second’: 1399.396, ‘eval\_steps\_per\_second’: 13.994}

2. {‘eval\_loss’: 0.4131013751029968, ‘eval\_f1’: 0.8973174175330509, ‘eval\_recall’: 0.9133333333333333, ‘eval\_accuracy’: 0.92, ‘eval\_runtime’: 0.1412, ‘eval\_samples\_per\_second’: 1416.786, ‘eval\_steps\_per\_second’: 14.168}

3. {‘eval\_loss’: 0.4108392000198364, ‘eval\_f1’: 0.90541946467417, ‘eval\_recall’: 0.9299999999999999, ‘eval\_accuracy’: 0.925, ‘eval\_runtime’: 0.1377, ‘eval\_samples\_per\_second’: 1452.709, ‘eval\_steps\_per\_second’: 14.527}

4. {‘eval\_loss’: 0.7171087861061096, ‘eval\_f1’: 0.8828281582436557, ‘eval\_recall’: 0.9166666666666666, ‘eval\_accuracy’: 0.905, ‘eval\_runtime’: 0.1368, ‘eval\_samples\_per\_second’: 1462.467, ‘eval\_steps\_per\_second’: 14.625}

5. {‘eval\_loss’: 1.1088610887527466, ‘eval\_f1’: 0.8746081504702194, ‘eval\_recall’: 0.9, ‘eval\_accuracy’: 0.9, ‘eval\_runtime’: 0.1388, ‘eval\_samples\_per\_second’: 1440.98, ‘eval\_steps\_per\_second’: 14.41}

6. {‘eval\_loss’: 1.3679903745651245, ‘eval\_f1’: 0.8815424420960754, ‘eval\_recall’: 0.91, ‘eval\_accuracy’: 0.905, ‘eval\_runtime’: 0.1375, ‘eval\_samples\_per\_second’: 1454.361, ‘eval\_steps\_per\_second’: 14.544}

7. {‘eval\_loss’: 1.5757907629013062, ‘eval\_f1’: 0.8815424420960754, ‘eval\_recall’: 0.91, ‘eval\_accuracy’: 0.905, ‘eval\_runtime’: 0.1393, ‘eval\_samples\_per\_second’: 1436.076, ‘eval\_steps\_per\_second’: 14.361}

8. {‘eval\_loss’: 1.7049527168273926, ‘eval\_f1’: 0.8815424420960754, ‘eval\_recall’: 0.91, ‘eval\_accuracy’: 0.905, ‘eval\_runtime’: 0.1389, ‘eval\_samples\_per\_second’: 1439.714, ‘eval\_steps\_per\_second’: 14.397}

9. {‘eval\_loss’: 1.7694827318191528, ‘eval\_f1’: 0.8815424420960754, ‘eval\_recall’: 0.91, ‘eval\_accuracy’: 0.905, ‘eval\_runtime’: 0.1383, ‘eval\_samples\_per\_second’: 1446.456, ‘eval\_steps\_per\_second’: 14.465}

10. {‘eval\_loss’: 1.80936861038208, ‘eval\_f1’: 0.8815424420960754, ‘eval\_recall’: 0.91, ‘eval\_accuracy’: 0.905, ‘eval\_runtime’: 0.1048, ‘eval\_samples\_per\_second’: 1909.088, ‘eval\_steps\_per\_second’: 19.091}

---

<div class="post-metadata">

**Author:** ![rkabir](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@rkabir](https://discuss.huggingface.co/u/rkabir)\
**Post date:** [July 1, 2025, 8:26pm UTC](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377/2 "2025-07-01T20:26:58Z")

</div>

If it is any help, here is my custom train function (the relevant part at least):

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/c/a/ca7dc45ec3586dc84ecfaa890210bb342a7e3894.png)

and here is a place where I do the seed randomizations:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/e/de6bf6fd5d21abd3f8e17dbab6a5c15abc2e725f.png)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 1, 2025, 9:00pm UTC](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377/3 "2025-07-01T21:00:16Z")

</div>

I wonder if the model output has reached the ideal value…

> [@Same metrics after every epoch](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851):
>
> This is a duplicate of a question I’ve posted on Stack Overflow that until now has gotten no answers (link to [SO post](https://stackoverflow.com/questions/78492018/hugging-face-trainer-object-returning-constant-values-for-metrics-every-epoch)). Basically, I’m using the Trainer class to fine tune a model on a text classification task and the metrics I’m tracking are taking on the same values after every epoch - changing LR and batch size does not seem to change anything. This is the code I’m currently using: def compute\_metrics(eval\_preds): logits, labels = eval\_preds preds = np.argmax(logits, axis=-1) …

> **[F1 - a Hugging Face Space by evaluate-metric](https://huggingface.co/spaces/evaluate-metric/f1)**
>
> The F1 score is the harmonic mean of the precision and recall. It can be computed with the equation: F1 = 2 \* (precision \* recall) / (precision + recall)

---

<div class="post-metadata">

**Author:** ![rkabir](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@rkabir](https://discuss.huggingface.co/u/rkabir)\
**Post date:** [July 2, 2025, 4:37pm UTC](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377/4 "2025-07-02T16:37:47Z")

</div>

> [@Repeating eval-F1 scores with seed + data randomization](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377/3):
>
> I wonder if the model output has reached the ideal value…

I don’t think so, given that in the third finetuning instance it reached an eval\_f1 of 0.9054, higher than 0.88154.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 2, 2025, 4:52pm UTC](https://discuss.huggingface.co/t/repeating-eval-f1-scores-with-seed-data-randomization/161377/5 "2025-07-02T16:52:46Z")

</div>

Hmm… For now by ChatGPT:

* * *

## Summary

Your repeated F1 scores happen because with only **250 evaluation samples** , the macro-F1 metric can take on only a few discrete values (“quantization”), and if your train/test splits or model converges similarly, you’ll see the **exact same** F1 repeated. To fix this:

1. **Explicitly randomize splits each run** by supplying a fresh `seed` to both `.shuffle()` and `.train_test_split()` in the Hugging Face Datasets API.
2. **Use k-fold (or repeated) cross-validation** to average F1 over multiple splits, smoothing out quantization.
3. **Increase your evaluation set size** or report complementary metrics (e.g., micro-F1, precision, recall) for finer granularity.

* * *

## 1. Truly Random Splits

By default, `dataset.train_test_split()` shuffles using an internal RNG that may not change across runs unless you pass `seed` yourself. Likewise, `dataset.shuffle()` needs its own `seed`. Example:

```python
import random

seed = random.randint(0, 2**32 - 1)
ds = ds.shuffle(seed=seed)
train_test = ds.train_test_split(test_size=250, shuffle=True, seed=seed)
train_ds, eval_ds = train_test["train"], train_test["test"]

```

* * *

## 2. k-Fold Cross-Validation

Single splits on small data lead to coarse F1 jumps (~1/250≈0.004). Instead, use scikit-learn’s CV to average across folds:

```python
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=None)
scores = cross_val_score(
    estimator=my_trainer, # a wrapper around your Hugging Face Trainer
    X=X_data,
    y=y_labels,
    cv=cv,
    scoring="f1_macro",
)
print(f"Macro-F1: {scores.mean():.4f} ± {scores.std():.4f}")

```

This yields a distribution of F1s and a more reliable mean ± std .

* * *

## 3. Increase Eval Size & Add Metrics

- **Larger hold-out** (e.g., 1 000 samples) makes F1 steps smaller (1/1000=0.001), reducing repetition.
- **Complementary metrics** like micro-F1, precision, recall, or AUC often vary smoothly and reveal differences that macro-F1 alone can miss ([arxiv.org](https://arxiv.org/abs/1707.09861 "Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging")).

By ensuring truly random splits, averaging over multiple folds, and broadening your evaluation metrics or set size, you’ll eliminate the “stuck” F1 behavior and obtain meaningful performance variability.