seanfarrell/icd11_synonym_model

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'other specified febrile seizures',
    'Other specified febrile seizures',
    'traumatic rupture of stomach',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.1484],
#         [1.0000, 1.0000, 0.1484],
#         [0.1484, 0.1484, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 10,000,000 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 3 tokens mean: 10.79 tokens max: 39 tokens	min: 3 tokens mean: 10.78 tokens max: 39 tokens

• Samples:

  sentence_0	sentence_1
  Amyloidosis	amyloidosis
  Other complications of spinal or epidural anaesthesia during labour or delivery	other complications of spinal or epidural anaesthesia during labour or delivery
  loeys-dietz syndrome type 1	rupture of dissecting aneurysm of aorta

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0256	500	0.7794
0.0512	1000	0.5638
0.0768	1500	0.4108
0.1024	2000	0.3219
0.1280	2500	0.2583
0.1536	3000	0.2133
0.1792	3500	0.1789
0.2048	4000	0.1547
0.2304	4500	0.137
0.2560	5000	0.1212
0.2816	5500	0.1101
0.3072	6000	0.1016
0.3328	6500	0.0949
0.3584	7000	0.088
0.3840	7500	0.0837
0.4096	8000	0.0803
0.4352	8500	0.0766
0.4608	9000	0.0733
0.4864	9500	0.0695
0.5120	10000	0.0685
0.5376	10500	0.0676
0.5632	11000	0.0654
0.5888	11500	0.0633
0.6144	12000	0.0613
0.6400	12500	0.0603
0.6656	13000	0.0593
0.6912	13500	0.0579
0.7168	14000	0.0572
0.7424	14500	0.0557
0.7680	15000	0.0562
0.7936	15500	0.0544
0.8192	16000	0.0548
0.8448	16500	0.0533
0.8704	17000	0.053
0.8960	17500	0.0525
0.9216	18000	0.0518
0.9472	18500	0.0499
0.9728	19000	0.0495
0.9984	19500	0.0493
1.0240	20000	0.0496
1.0496	20500	0.0484
1.0752	21000	0.0481
1.1008	21500	0.0487
1.1264	22000	0.0482
1.1520	22500	0.0469
1.1776	23000	0.047
1.2032	23500	0.0471
1.2288	24000	0.047
1.2544	24500	0.0451
1.2800	25000	0.0471
1.3055	25500	0.0454
1.3311	26000	0.0455
1.3567	26500	0.0453
1.3823	27000	0.0436
1.4079	27500	0.0436
1.4335	28000	0.0438
1.4591	28500	0.0439
1.4847	29000	0.0435
1.5103	29500	0.0437
1.5359	30000	0.0439
1.5615	30500	0.0426
1.5871	31000	0.0437
1.6127	31500	0.043
1.6383	32000	0.0425
1.6639	32500	0.0425
1.6895	33000	0.042
1.7151	33500	0.0421
1.7407	34000	0.0422
1.7663	34500	0.0416
1.7919	35000	0.0426
1.8175	35500	0.0418
1.8431	36000	0.0414
1.8687	36500	0.041
1.8943	37000	0.042
1.9199	37500	0.0414
1.9455	38000	0.0404
1.9711	38500	0.0409
1.9967	39000	0.0403
2.0223	39500	0.0401
2.0479	40000	0.0407
2.0735	40500	0.0403
2.0991	41000	0.0405
2.1247	41500	0.04
2.1503	42000	0.0403
2.1759	42500	0.0398
2.2015	43000	0.0412
2.2271	43500	0.04
2.2527	44000	0.0396
2.2783	44500	0.039
2.3039	45000	0.0396
2.3295	45500	0.0389
2.3551	46000	0.0398
2.3807	46500	0.0401
2.4063	47000	0.0389
2.4319	47500	0.0398
2.4575	48000	0.039
2.4831	48500	0.0386
2.5087	49000	0.0391
2.5343	49500	0.04
2.5599	50000	0.0383
2.5855	50500	0.039
2.6111	51000	0.0386
2.6367	51500	0.0375
2.6623	52000	0.0388
2.6879	52500	0.0386
2.7135	53000	0.0385
2.7391	53500	0.0393
2.7647	54000	0.0382
2.7903	54500	0.0384
2.8159	55000	0.0386
2.8415	55500	0.0384
2.8671	56000	0.0378
2.8927	56500	0.0386
2.9183	57000	0.0384
2.9439	57500	0.0375
2.9695	58000	0.0399
2.9951	58500	0.0385

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.1.2
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}