comet-bio-mqm-n6jrlwtt

A domain-adapted version of wmt22-comet-da finetuned on the Amazon Bio-MQM biomedical translation evaluation dataset.

Training duration: 2 epochs on the Amazon Bio-MQM dataset (PyTorch-Lightning epoch index 0–1; checkpoint epoch=1-step=20, global step 20, validation Kendall τ = 0.366).

Training details

Parameter	Value
Base model	xlm-roberta-large
Finetuning data	Amazon Bio-MQM (dev splits)
Training epochs	2 (Lightning epoch index 0–1)
Checkpoint	epoch=1-step=20, global step 20
Validation Kendall τ	0.366
Language pairs	de↔en, es↔en, fr↔en, ru↔en, zh↔en
Loss	mse
Encoder LR	5e-07
Head LR	1e-05
Batch size	8
Run	comet-bio-mqm-n6jrlwtt

Usage

from comet import download_model, load_from_checkpoint

model_path = download_model("AdleBenSalem/comet-bio-mqm-n6jrlwtt")
model = load_from_checkpoint(model_path)

data = [{
    "src": "The patient was administered 500 mg of amoxicillin.",
    "mt":  "Der Patient erhielt 500 mg Amoxicillin.",
    "ref": "Dem Patienten wurden 500 mg Amoxicillin verabreicht.",
}]
output = model.predict(data, batch_size=8, gpus=1)
print(output.scores)

Citation

If you use this model, please cite the original COMET paper and the Bio-MQM dataset:

@inproceedings{rei-etal-2020-comet,
  title     = {COMET: A Neural Framework for MT Evaluation},
  author    = {Rei, Ricardo and Stewart, Craig and Farinha, Ana C and Lavie, Alon},
  booktitle = {Proceedings of EMNLP 2020},
}

@inproceedings{bio-mqm-2024,
  title     = {Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains},
  booktitle = {Proceedings of ACL 2024},
}