Update Hugging Face org references: llm-semantic-router → vllm-sr

f60baa4 verified 22 days ago

17 kB

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:606
  - loss:MultipleNegativesRankingLoss
base_model: vllm-sr/mmbert-embed-32k-2d-matryoshka
widget:
  - source_sentence: What might be contained in Item 8 of a financial document?
    sentences:
      - >-
        What amount was authorized for future share repurchases by the company
        as of October 31, 2023?


        Answer: $1.0 billion
      - |-
        What information is contained in Item 8 of a financial document?

        Answer: Item 8 contains the Financial Statements and Supplementary Data.
      - |-
        What might be contained in Item 8 of a financial document?

        Answer: Financial Statements and Supplementary Data
  - source_sentence: >-
      In IBM's 2023 Annual Report, which pages detail the Financial Statements
      and Supplementary Data?
    sentences:
      - >-
        How are the pages of IBM's financial statements and supplementary data
        described in terms of location in the 2023 Annual Report?


        Answer: They are incorporated by reference.
      - >-
        In IBM's 2023 Annual Report, which pages detail the Financial Statements
        and Supplementary Data?


        Answer: Pages 44 through 121
      - |-
        What does Item 8 of the financial document pertain to?

        Answer: It pertains to Financial Statements and Supplementary Data.
  - source_sentence: How much free cash flow did Delta Air Lines generate in 2023?
    sentences:
      - |-
        What is the title of Item 8 in the document?

        Answer: Financial Statements and Supplementary Data
      - >-
        How does Free Cash Flow differ from net cash provided by operating
        activities?


        Answer: Free Cash Flow is calculated by subtracting purchases of
        property and equipment from net cash provided by operating activities.
      - |-
        How much free cash flow did Delta Air Lines generate in 2023?

        Answer: $2,003 million
  - source_sentence: How much did the AUS for equity increase from 2022 to 2023?
    sentences:
      - >-
        What caused the effective tax rate for 2022 to be impacted?


        Answer: The effective tax rate for 2022 was impacted by net discrete tax
        benefits of $130, primarily related to excess tax benefits from stock
        compensation.
      - >-
        How did foreign exchange rate fluctuations impact the company's
        consolidated revenues and income before income taxes for the fiscal
        years ended May 31, 2023, 2022, and 2021?


        Answer: Foreign exchange rate fluctuations had a negative impact on the
        company's consolidated revenues by approximately $2,859 million in 2023,
        $295 million in 2022, and positively by $893 million in 2021. Similarly,
        for income before income taxes, the impact was a detriment of
        approximately $824 million in 2023, $87 million in 2022, and a benefit
        of $260 million in 2021.
      - |-
        How much did the AUS for equity increase from 2022 to 2023?

        Answer: $95 billion
  - source_sentence: What is included in Item 8 of the document?
    sentences:
      - >-
        How has Google Search evolved since its inception?


        Answer: Google Search has evolved from a basic web page search tool into
        a dynamic, multimodal experience, expanding into mobile web search,
        introducing Universal Search, and new search modalities like voice and
        visual search.
      - >-
        What is the content of Item 8 in the document?


        Answer: Item 8 of the document includes Financial Statements and
        Supplementary Data.
      - |-
        What is included in Item 8 of the document?

        Answer: Financial Statements and Supplementary Data
pipeline_tag: sentence-similarity
library_name: sentence-transformers

SentenceTransformer based on vllm-sr/mmbert-embed-32k-2d-matryoshka

This is a sentence-transformers model finetuned from vllm-sr/mmbert-embed-32k-2d-matryoshka. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: vllm-sr/mmbert-embed-32k-2d-matryoshka
Maximum Sequence Length: 32768 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What is included in Item 8 of the document?',
    'What is included in Item 8 of the document?\n\nAnswer: Financial Statements and Supplementary Data',
    'What is the content of Item 8 in the document?\n\nAnswer: Item 8 of the document includes Financial Statements and Supplementary Data.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0078, 0.9297, 0.5508],
#         [0.9297, 0.9922, 0.6680],
#         [0.5508, 0.6680, 1.0000]], dtype=torch.bfloat16)

Training Details

Training Dataset

Unnamed Dataset

Size: 606 training samples
Columns: sentence_0, sentence_1, and sentence_2

Approximate statistics based on the first 606 samples:

	sentence_0	sentence_1	sentence_2
type	string	string	string
details	min: 10 tokens mean: 22.46 tokens max: 55 tokens	min: 18 tokens mean: 52.15 tokens max: 229 tokens	min: 18 tokens mean: 49.13 tokens max: 120 tokens

Samples:

sentence_0	sentence_1	sentence_2
`What led to the increase in operating income margin for the Dollar Tree segment in 2022?`	`What led to the increase in operating income margin for the Dollar Tree segment in 2022? Answer: The increase in operating income margin for the Dollar Tree segment in 2022 was primarily due to the gross profit margin increase and a decrease in the selling, general and administrative expense rate.`	`What was the increase in the gross profit margin for the fiscal year 2022 compared to the previous year? Answer: The increase in the gross profit margin for the fiscal year 2022 compared to the previous year was 2.1%.`
`How much net cash was provided by operating activities in 2022?`	`How much net cash was provided by operating activities in 2022? Answer: $4.5 billion`	`What were the main components contributing to the net cash provided by operating activities in 2023? Answer: Operating income, depreciation and amortization, and various adjustments in operating assets and liabilities were the main contributors.`
`How was the stock-based compensation expense of $254 million accounted for in the company's financial statements?`	`How was the stock-based compensation expense of $254 million accounted for in the company's financial statements? Answer: The stock-based compensation expense of $254 million was recorded in the line item selling, general, and administrative expenses in the company's consolidated statement of income.`	`How much did the cost of revenue increase in stock-based compensation expense from fiscal year 2021 to 2023? Answer: $2 million`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

num_train_epochs: 2
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Framework Versions

Python: 3.12.10
Sentence Transformers: 5.2.2
Transformers: 5.0.0
PyTorch: 2.7.0+cu128
Accelerate: 1.12.0
Datasets: 4.5.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}