# Different embeddings when using sentence transformers and transformers.js **URL:** \ **Category:** Beginners\ **Created:** [February 15, 2024, 3:46pm UTC](https://discuss.huggingface.co/t/different-embeddings-when-using-sentence-transformers-and-transformers-js/73345 "2024-02-15T15:46:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![danielruss](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@danielruss](https://discuss.huggingface.co/u/danielruss)\ **Post date:** [February 15, 2024, 3:46pm UTC](https://discuss.huggingface.co/t/different-embeddings-when-using-sentence-transformers-and-transformers-js/73345/1 "2024-02-15T15:46:51Z")
Hello, I am building a multilabel classifier that uses the embeddings from sentence-transformers/all-MiniLM-L6-v2 as input. After training a model that produces good enough results, I would like to run this in the brower using transformersjs and the Xenova/all-MiniLM-L6-v2 model. However, I am getting different embeddings for the same text. Here is my python code: ```auto model_name = "sentence-transformers/all-MiniLM-L6-v2" mdl = SentenceTransformer(model_name) raw_inputs = [ "I've been waiting for a HuggingFace course my whole life.", "I hate this so much!", ] se = mdl.encode(raw_inputs) # the first 4 dimensions... # [[-0.0635541 0.00168205 0.08878317 0.01061784] # [-0.0278877 0.02493023 0.01891949 0.03274209]] ``` The js code: ```auto import { pipeline } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@latest'; let extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2') let result = await extractor( [ "I've been waiting for a HuggingFace course my whole life.", "I hate this so much!", ], { pooling: 'mean', normalize: true }); console.log( result.data.slice(0,4) ) console.log( result.data.slice(384,388) ) // [-0.0713, 0.0169, 0.0940, 0.00842] // [-0.0041, 0.0070, 0.0365 0.0422] ``` I would like to reproduce the sentence transformer embeddings. If this is not possible, I just need to have the same embeddings between python and javascript, and I will try to retrain. My specific questions are: 1. Am I doing this correctly? 2. If so, can I get the javascript embeddings to match? Thank you ---
**Author:** ![Stefan-LTB](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@Stefan-LTB](https://discuss.huggingface.co/u/Stefan-LTB)\ **Post date:** [April 19, 2024, 9:34am UTC](https://discuss.huggingface.co/t/different-embeddings-when-using-sentence-transformers-and-transformers-js/73345/2 "2024-04-19T09:34:13Z")
Hi, I recognized the same with sentence-transformers and transformers. Why is this the case and do I make a mistake or is it ok, as long as you use the same way for all embeddings you create? My goal is building a local RAG application ```auto from sentence_transformers import SentenceTransformer from transformers import AutoTokenizer, AutoModel import torch import pandas as pd # sentence-transformer version model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") sentences = [ "This framework generates embeddings for each input sentence", "Sentences are passed as a list of strings.", "The quick brown fox jumps over the lazy dog.", ] embeddings = model.encode(sentences) df = pd.DataFrame(embeddings, index=sentences) print("sentence-transformer version") print(df) # --------------------------------------------------------------- # transformer version # Mean Pooling - Take attention mask into account for correct averaging def mean_pooling(model_output, attention_mask): token_embeddings = model_output[0] # First element of model_output contains all token embeddings input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float() sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1) sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9) return sum_embeddings / sum_mask # Load AutoModel from huggingface model repository tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") # Tokenize sentences encoded_input = tokenizer( sentences, padding=True, truncation=True, max_length=128, return_tensors="pt" ) # Compute token embeddings with torch.no_grad(): model_output = model(**encoded_input) # Perform pooling. In this case, mean pooling sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"]) df2 = pd.DataFrame(sentence_embeddings.numpy(), index=sentences) print("transformer version") print(df2) ``` ---
**Author:** ![danielruss](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@danielruss](https://discuss.huggingface.co/u/danielruss)\ **Post date:** [April 19, 2024, 8:17pm UTC](https://discuss.huggingface.co/t/different-embeddings-when-using-sentence-transformers-and-transformers-js/73345/3 "2024-04-19T20:17:28Z")
@Stefan-LTB I was able to get the embeddings to match. I believe it was caused because transformersjs preferentially uses quantized models. So by adding `pipeline('feature-extraction','Xenova/all-MiniLM-L6-v2',{quantized:false})` in my js code. The values looked the same. I am not sure why your transformers and sentence\_transformers are not providing the same results. I’m pretty sure I looked at this case also. You may want to double check your masking. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [January 29, 2025, 10:32pm UTC](https://discuss.huggingface.co/t/different-embeddings-when-using-sentence-transformers-and-transformers-js/73345/4 "2025-01-29T22:32:12Z")
Closed on January 29, 2025, 10:32pm UTC This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.