# Unable to push embeddings onto GPU

**URL:** <https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574>\
**Category:** Beginners\
**Created:** [June 1, 2025, 7:06pm UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574 "2025-06-01T19:06:07Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![User515](https://avatars.discourse-cdn.com/v4/letter/u/6bbea6/32.png) [@User515](https://discuss.huggingface.co/u/User515)\
**Post date:** [June 1, 2025, 7:06pm UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/1 "2025-06-01T19:06:07Z")

</div>

Ubuntu 24.04, RTX 4060, i5-13400F, 16 GB DDR5 RAM

I’m new to WASM and Llama but I’m trying to set up a local RAG API server. What I’d like to eventually do is repeatedly check a text file (that is growing in size) and let me ask it questions about it.

My issue right now is that the Instruct model will use GPU when I set it up as a chat server alone, but the embedding model doesn’t use GPU when I instantiate a server with Instruct and Embedding. I set up the server using the following command…

```auto
wasmedge --dir .:. --nn-preload default:GGML:GPU:Llama-3.2-3B-Instruct-Q5_K_M.gguf \
    --nn-preload embedding:GGML:GPU:nomic-embed-text-v1.5.Q5_K_M.gguf \
    rag-api-server.wasm \
    --model-name Llama-3.2-3B-Instruct-Q5_K_M,nomic-embed-text-v1.5.Q5_K_M \
    --ctx-size 4096,348 \
	--batch-size 512,512 \
    --prompt-template llama-3-chat,embedding \
    --rag-policy system-message \
    --qdrant-collection-name default \
    --qdrant-limit 3 \
    --qdrant-score-threshold 0.5 \
    --rag-prompt "Use the following pieces of context to answer the user's question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\n" \
    --port 8080

```

I then issue…

```auto
curl -X POST http://127.0.0.1:8080/v1/create/rag -F "file=@filename.txt"

```

And the return takes like 30 minutes. When I watch `htop` and `nvidia-smi`, I start loading individual CPU cores at 100% and cycling through them but the GPU is barely utilized, if at all.

Below are server logs, in case they’re helpful.

* * *

```auto
[2025-06-01 12:01:59.701] [info] rag_api_server in src/main.rs:189: log_level: info
[2025-06-01 12:01:59.701] [info] rag_api_server in src/main.rs:192: server_version: 0.13.15
[2025-06-01 12:01:59.701] [info] rag_api_server in src/main.rs:200: model_name: Llama-3.2-3B-Instruct-Q5_K_M,nomic-embed-text-v1.5.Q5_K_M
[2025-06-01 12:01:59.701] [info] rag_api_server in src/main.rs:208: model_alias: default,embedding
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:222: ctx_size: 4096,348
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:236: batch_size: 512,512
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:250: ubatch_size: 512,512
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:264: prompt_template: llama-3-chat,embedding
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:272: n_predict: -1
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:275: n_gpu_layers: 100
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:278: split_mode: layer
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:291: threads: 2
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:305: rag_prompt: Use the following pieces of context to answer the user's question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\n
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:326: qdrant_url: http://127.0.0.1:6333
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:353: qdrant_collection_name: default
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:362: qdrant_limit: 3
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:371: qdrant_score_threshold: 0.5
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:399: chunk_capacity: 100
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:402: context_window: 1
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:408: rag_policy: system-message
[2025-06-01 12:01:59.702] [info] rag_api_server in src/main.rs:427: include_usage: false
[2025-06-01 12:01:59.702] [info] llama_core in /home/laurentius/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/llama-core-0.30.0/src/lib.rs:189: Initializing the core context for RAG scenarios
[2025-06-01 12:01:59.702] [info] [WASI-NN] GGML backend: LLAMA_COMMIT cf0a43bb
[2025-06-01 12:01:59.702] [info] [WASI-NN] GGML backend: LLAMA_BUILD_NUMBER 5361
[2025-06-01 12:01:59.706] [info] [WASI-NN] llama.cpp: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
[2025-06-01 12:01:59.706] [info] [WASI-NN] llama.cpp: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
[2025-06-01 12:01:59.706] [info] [WASI-NN] llama.cpp: ggml_cuda_init: found 1 CUDA devices:
[2025-06-01 12:01:59.706] [info] [WASI-NN] llama.cpp: Device 0: NVIDIA GeForce RTX 4060, compute capability 8.9, VMM: yes
[2025-06-01 12:01:59.749] [info] [WASI-NN] llama.cpp: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4060) - 5561 MiB free
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: loaded meta data with 31 key-value pairs and 255 tensors from Llama-3.2-3B-Instruct-Q5_K_M.gguf (version GGUF V3 (latest))
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 0: general.architecture str = llama
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 1: general.type str = model
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 2: general.name str = Llama 3.2 3B Instruct
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 3: general.finetune str = Instruct
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 4: general.basename str = Llama-3.2
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 5: general.size_label str = 3B
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 6: general.license str = llama3.2
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam...
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ...
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 9: llama.block_count u32 = 28
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 10: llama.context_length u32 = 131072
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 11: llama.embedding_length u32 = 3072
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 12: llama.feed_forward_length u32 = 8192
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 13: llama.attention.head_count u32 = 24
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 17: llama.attention.key_length u32 = 128
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 18: llama.attention.value_length u32 = 128
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 19: llama.vocab_size u32 = 128256
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 128
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2
[2025-06-01 12:01:59.776] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 22: tokenizer.ggml.pre str = llama-bpe
[2025-06-01 12:01:59.788] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
[2025-06-01 12:01:59.793] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 128000
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 128009
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 28: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 29: general.quantization_version u32 = 2
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 30: general.file_type u32 = 17
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - type f32: 58 tensors
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - type q5_K: 168 tensors
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: llama_model_loader: - type q6_K: 29 tensors
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: print_info: file format = GGUF V3 (latest)
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: print_info: file type = Q5_K - Medium
[2025-06-01 12:01:59.817] [info] [WASI-NN] llama.cpp: print_info: file size = 2.16 GiB (5.76 BPW) 
[2025-06-01 12:01:59.913] [info] [WASI-NN] llama.cpp: load: special tokens cache size = 256
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: load: token to piece cache size = 0.7999 MB
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: arch = llama
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: vocab_only = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_ctx_train = 131072
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_embd = 3072
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_layer = 28
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_head = 24
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_head_kv = 8
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_rot = 128
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_swa = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_swa_pattern = 1
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_embd_head_k = 128
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_embd_head_v = 128
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_gqa = 3
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_embd_k_gqa = 1024
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_embd_v_gqa = 1024
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: f_norm_eps = 0.0e+00
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: f_norm_rms_eps = 1.0e-05
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: f_clamp_kqv = 0.0e+00
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: f_max_alibi_bias = 0.0e+00
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: f_logit_scale = 0.0e+00
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: f_attn_scale = 0.0e+00
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_ff = 8192
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_expert = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_expert_used = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: causal attn = 1
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: pooling type = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: rope type = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: rope scaling = linear
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: freq_base_train = 500000.0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: freq_scale_train = 1
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_ctx_orig_yarn = 131072
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: rope_finetuned = unknown
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: ssm_d_conv = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: ssm_d_inner = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: ssm_d_state = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: ssm_dt_rank = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: ssm_dt_b_c_rms = 0
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: model type = 3B
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: model params = 3.21 B
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: general.name = Llama 3.2 3B Instruct
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: vocab type = BPE
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_vocab = 128256
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: n_merges = 280147
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: BOS token = 128000 '<|begin_of_text|>'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: EOS token = 128009 '<|eot_id|>'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: EOT token = 128009 '<|eot_id|>'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: EOM token = 128008 '<|eom_id|>'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: LF token = 198 'Ċ'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: EOG token = 128008 '<|eom_id|>'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: EOG token = 128009 '<|eot_id|>'
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: print_info: max token length = 256
[2025-06-01 12:01:59.937] [info] [WASI-NN] llama.cpp: load_tensors: loading model tensors, this can take a while... (mmap = true)
[2025-06-01 12:01:59.987] [info] [WASI-NN] llama.cpp: load_tensors: offloading 28 repeating layers to GPU
[2025-06-01 12:01:59.987] [info] [WASI-NN] llama.cpp: load_tensors: offloading output layer to GPU
[2025-06-01 12:01:59.987] [info] [WASI-NN] llama.cpp: load_tensors: offloaded 29/29 layers to GPU
[2025-06-01 12:01:59.987] [info] [WASI-NN] llama.cpp: load_tensors: CPU_Mapped model buffer size = 308.23 MiB
[2025-06-01 12:01:59.987] [info] [WASI-NN] llama.cpp: load_tensors: CUDA0 model buffer size = 2207.10 MiB
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: constructing llama_context
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: n_seq_max = 1
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: n_ctx = 4096
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: n_ctx_per_seq = 4096
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: n_batch = 512
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: n_ubatch = 512
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: causal_attn = 1
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: flash_attn = 0
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: freq_base = 500000.0
[2025-06-01 12:02:00.263] [info] [WASI-NN] llama.cpp: llama_context: freq_scale = 1
[2025-06-01 12:02:00.263] [warning] [WASI-NN] llama.cpp: llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
[2025-06-01 12:02:00.264] [info] [WASI-NN] llama.cpp: llama_context: CUDA_Host output buffer size = 0.49 MiB
[2025-06-01 12:02:00.264] [info] [WASI-NN] llama.cpp: llama_kv_cache_unified: kv_size = 4096, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1, padding = 32
[2025-06-01 12:02:00.266] [info] [WASI-NN] llama.cpp: llama_kv_cache_unified: CUDA0 KV buffer size = 448.00 MiB
[2025-06-01 12:02:00.266] [info] [WASI-NN] llama.cpp: llama_kv_cache_unified: KV self size = 448.00 MiB, K (f16): 224.00 MiB, V (f16): 224.00 MiB
[2025-06-01 12:02:00.278] [info] [WASI-NN] llama.cpp: llama_context: CUDA0 compute buffer size = 256.50 MiB
[2025-06-01 12:02:00.278] [info] [WASI-NN] llama.cpp: llama_context: CUDA_Host compute buffer size = 14.01 MiB
[2025-06-01 12:02:00.278] [info] [WASI-NN] llama.cpp: llama_context: graph nodes = 958
[2025-06-01 12:02:00.278] [info] [WASI-NN] llama.cpp: llama_context: graph splits = 2
[2025-06-01 12:02:00.278] [info] [WASI-NN] GGML backend: llama_system_info: CUDA : ARCHS = 600,610,700,750,800,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | BMI2 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | 
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
[2025-06-01 12:02:00.278] [info] [WASI-NN] GGML backend: LLAMA_COMMIT cf0a43bb
[2025-06-01 12:02:00.278] [info] [WASI-NN] GGML backend: LLAMA_BUILD_NUMBER 5361
[2025-06-01 12:02:00.279] [info] [WASI-NN] llama.cpp: llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4060) - 2647 MiB free
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: loaded meta data with 23 key-value pairs and 112 tensors from nomic-embed-text-v1.5.Q5_K_M.gguf (version GGUF V3 (latest))
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 0: general.architecture str = nomic-bert
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 1: general.name str = nomic-embed-text-v1.5
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 2: nomic-bert.block_count u32 = 12
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 3: nomic-bert.context_length u32 = 2048
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 4: nomic-bert.embedding_length u32 = 768
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 5: nomic-bert.feed_forward_length u32 = 3072
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 6: nomic-bert.attention.head_count u32 = 12
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 7: nomic-bert.attention.layer_norm_epsilon f32 = 0.000000
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 8: general.file_type u32 = 17
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 9: nomic-bert.attention.causal bool = false
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 10: nomic-bert.pooling_type u32 = 1
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 11: nomic-bert.rope.freq_base f32 = 1000.000000
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 12: tokenizer.ggml.token_type_count u32 = 2
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 13: tokenizer.ggml.bos_token_id u32 = 101
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 14: tokenizer.ggml.eos_token_id u32 = 102
[2025-06-01 12:02:00.282] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 15: tokenizer.ggml.model str = bert
[2025-06-01 12:02:00.284] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,30522] = ["[PAD]", "[unused0]", "[unused1]", "...
[2025-06-01 12:02:00.289] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,30522] = [-1000.000000, -1000.000000, -1000.00...
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,30522] = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 100
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 20: tokenizer.ggml.seperator_token_id u32 = 102
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 0
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 22: general.quantization_version u32 = 2
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - type f32: 51 tensors
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - type q5_K: 43 tensors
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: llama_model_loader: - type q6_K: 18 tensors
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: print_info: file format = GGUF V3 (latest)
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: print_info: file type = Q5_K - Medium
[2025-06-01 12:02:00.291] [info] [WASI-NN] llama.cpp: print_info: file size = 94.25 MiB (5.78 BPW) 
[2025-06-01 12:02:00.301] [warning] [WASI-NN] llama.cpp: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
[2025-06-01 12:02:00.301] [info] [WASI-NN] llama.cpp: load: special tokens cache size = 5
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: load: token to piece cache size = 0.2032 MB
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: arch = nomic-bert
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: vocab_only = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_ctx_train = 2048
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_embd = 768
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_layer = 12
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_head = 12
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_head_kv = 12
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_rot = 64
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_swa = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_swa_pattern = 1
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_embd_head_k = 64
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_embd_head_v = 64
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_gqa = 1
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_embd_k_gqa = 768
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_embd_v_gqa = 768
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: f_norm_eps = 1.0e-12
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: f_norm_rms_eps = 0.0e+00
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: f_clamp_kqv = 0.0e+00
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: f_max_alibi_bias = 0.0e+00
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: f_logit_scale = 0.0e+00
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: f_attn_scale = 0.0e+00
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_ff = 3072
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_expert = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_expert_used = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: causal attn = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: pooling type = 1
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: rope type = 2
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: rope scaling = linear
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: freq_base_train = 1000.0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: freq_scale_train = 1
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: n_ctx_orig_yarn = 2048
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: rope_finetuned = unknown
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: ssm_d_conv = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: ssm_d_inner = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: ssm_d_state = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: ssm_dt_rank = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: ssm_dt_b_c_rms = 0
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: model type = 137M
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: model params = 136.73 M
[2025-06-01 12:02:00.303] [info] [WASI-NN] llama.cpp: print_info: general.name = nomic-embed-text-v1.5
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: vocab type = WPM
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: n_vocab = 30522
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: n_merges = 0
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: BOS token = 101 '[CLS]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: EOS token = 102 '[SEP]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: UNK token = 100 '[UNK]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: SEP token = 102 '[SEP]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: PAD token = 0 '[PAD]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: MASK token = 103 '[MASK]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: LF token = 0 '[PAD]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: EOG token = 102 '[SEP]'
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: print_info: max token length = 21
[2025-06-01 12:02:00.304] [info] [WASI-NN] llama.cpp: load_tensors: loading model tensors, this can take a while... (mmap = true)
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: offloading 12 repeating layers to GPU
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: offloading output layer to GPU
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: offloaded 13/13 layers to GPU
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: CPU_Mapped model buffer size = 15.38 MiB
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: CUDA0 model buffer size = 78.88 MiB
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: constructing llama_context
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: n_seq_max = 1
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: n_ctx = 348
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: n_ctx_per_seq = 348
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: n_batch = 512
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: n_ubatch = 512
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: causal_attn = 0
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: flash_attn = 0
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: freq_base = 1000.0
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: freq_scale = 1
[2025-06-01 12:02:00.321] [warning] [WASI-NN] llama.cpp: llama_context: n_ctx_per_seq (348) < n_ctx_train (2048) -- the full capacity of the model will not be utilized
[2025-06-01 12:02:00.321] [info] [WASI-NN] llama.cpp: llama_context: CUDA_Host output buffer size = 0.12 MiB
[2025-06-01 12:02:00.329] [info] [WASI-NN] GGML backend: llama_system_info: CUDA : ARCHS = 600,610,700,750,800,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | BMI2 = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | 
common_init_from_params: KV cache shifting is not supported for this context, disabling KV cache shifting
common_init_from_params: setting dry_penalty_last_n to ctx_size = 348
[2025-06-01 12:02:00.329] [info] rag_api_server in src/main.rs:542: plugin_ggml_version: b5361 (commit cf0a43bb)
[2025-06-01 12:02:00.329] [info] rag_api_server in src/main.rs:591: Listening on 0.0.0.0:8080

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 2, 2025, 3:14am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/2 "2025-06-02T03:14:22Z")

</div>

It seems that you cannot use GPU in Embedding if the version of Llama.cpp is old, but I am not sure if this is the problem.

> **[Embeddings and GPU usage · ggml-org llama.cpp · Discussion #2658](https://github.com/ggml-org/llama.cpp/discussions/2658)**
>
> When generating embeddings (both using the embedding utility compiled from this repo, and also using langchain and llama-cpp-python) on a MacBook M1, I've noticed that there seem to be no additiona...

---

<div class="post-metadata">

**Author:** ![User515](https://avatars.discourse-cdn.com/v4/letter/u/6bbea6/32.png) [@User515](https://discuss.huggingface.co/u/User515)\
**Post date:** [June 3, 2025, 4:23am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/3 "2025-06-03T04:23:56Z")

</div>

Silly question, but I am pretty new to this: if I’m using a GGUF that’s previously been built and I’m running on WASMEdge / WASI-NN / LlamaEdge, at which point would I have compiled llama.cpp directly?

Since I’m just trying to quickly demo a tool, I’ve been running the install shell scripts provided on the respective repos (e.g. [WasmEdge/utils/install\_v2.sh at master · WasmEdge/WasmEdge · GitHub](https://github.com/WasmEdge/WasmEdge/blob/master/utils/install_v2.sh)), and haven’t looked too deeply into the core libraries.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 3, 2025, 5:45am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/4 "2025-06-03T05:45:30Z")

</div>

If GGUF is converted correctly, even old files should work fine. (Very rarely, the supported format range may change, but we don’t usually need to worry about that.)  
As long as Llama.cpp is up-to-date and relatively bug-free, it shouldn’t cause any issues at runtime…

Overall, since the GPU is recognized and doesn’t error out, this issue is likely due to a configuration mistake or a bug, so it’s probably not structurally impossible. VRAM should also be sufficient…

> **[Common AI Model Formats](https://huggingface.co/blog/ngxson/common-ai-model-formats)**
>
> A Blog post by Xuan-Son Nguyen on Hugging Face

---

<div class="post-metadata">

**Author:** ![User515](https://avatars.discourse-cdn.com/v4/letter/u/6bbea6/32.png) [@User515](https://discuss.huggingface.co/u/User515)\
**Post date:** [June 3, 2025, 2:01pm UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/5 "2025-06-03T14:01:32Z")

</div>

Ok I think I understand now: llama.cpp is used to quantize the GGUF’s before I download them, so I’m not directly interacting with that project in the way I’ve set up my tool.

According to Nomic’s page ([second-state/nomic-embed-text-v1.5-GGUF · Hugging Face](https://huggingface.co/second-state/nomic-embed-text-v1.5-GGUF)) it was quantized with llama.cpp b4120, a release from Nov. 2024. Seems relatively new.

Where could I check for the configuration mistake? Is there something I can test with the WASMEdge call?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 4, 2025, 12:51am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/6 "2025-06-04T00:51:17Z")

</div>

I assume that the problem lies in the settings of the program running on wasmedge rather than wasmedge itself, but I don’t know which program you are using…😅

Assuming that this is the case, increasing the number in this option may be effective.

> **[GitHub - LlamaEdge/rag-api-server: A RAG API server written in Rust following OpenAI...](https://github.com/LlamaEdge/rag-api-server)**
>
> A RAG API server written in Rust following OpenAI specs

> -g, --n-gpu-layers \<N\_GPU\_LAYERS\>  
> Number of layers to run on the GPU

> ```
> [default: 100]
> 
> ```

Edit:  
I thought so, but looking at the log above, 100 seems to be enough…  
The cause must be something else…

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 4, 2025, 12:58am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/7 "2025-06-04T00:58:43Z")

</div>

The core is fixed to an old version…

I’m not sure if this is the cause, but the version of Llama.cpp that is actually used may be old.

> <https://github.com/LlamaEdge/rag-api-server/blob/main/Cargo.toml#L16>

```auto
llama-core = { version = "=0.30.0", features = ["logging", "rag", "index"] }

```

---

<div class="post-metadata">

**Author:** ![User515](https://avatars.discourse-cdn.com/v4/letter/u/6bbea6/32.png) [@User515](https://discuss.huggingface.co/u/User515)\
**Post date:** [June 4, 2025, 3:32am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/8 "2025-06-04T03:32:52Z")

</div>

Hmm I tried adjusting the llama-core crate version to newer versions (even the version immediately after, 0.31.0) and running `cargo build --target wasm32-wasip1 --release` again but the build fails because there are too many mismatched types (i.e. the method calls have been changed).

A few examples…

```auto
2438 | ... chat_prompt.build_with_tools(&mut chat_request.messages, Some(&[])) {
     | ---------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `ChatCompletionRequestMessage`, found a different `ChatCompletionRequestMessage`
     | |
     | arguments to this method are incorrect
     |

```

and

```auto
error[E0308]: mismatched types
    --> /home/laurentius/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/llama-core-0.31.0/src/chat.rs:2452:76
     |
2452 | ...chat_request.messages, Some(tools.as_slice()))
     | ---- ^^^^^^^^^^^^^^^^ expected `endpoints::chat::Tool`, found a different `endpoints::chat::Tool`
     | |
     | arguments to this enum variant are incorrect
     |

```

Changing the llama-core crate may be a heavy lift because I’d need to go in and understand many of the method calls and possibly rewrite some of them.

---

<div class="post-metadata">

**Author:** ![User515](https://avatars.discourse-cdn.com/v4/letter/u/6bbea6/32.png) [@User515](https://discuss.huggingface.co/u/User515)\
**Post date:** [June 4, 2025, 3:58am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/9 "2025-06-04T03:58:27Z")

</div>

According to this ([All releases of llama-core // Lib.rs](https://lib.rs/crates/llama-core/versions)), 0.30.0 is from April of this year, so its CUDA support should be up to date. I’ll try a few different embeddings models.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 4, 2025, 4:35am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/10 "2025-06-04T04:35:07Z")

</div>

When I checked the log, most of it seems to be loaded on the GPU…

Could it be that some of it is on the CPU, or is there some other reason why the calculations are slow, and there is no problem with the loading itself?

```auto
[2025-06-01 12:02:00.281] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 0: general.architecture str = nomic-bert
...
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: offloading 12 repeating layers to GPU
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: offloading output layer to GPU
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: offloaded 13/13 layers to GPU
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: CPU_Mapped model buffer size = 15.38 MiB
[2025-06-01 12:02:00.307] [info] [WASI-NN] llama.cpp: load_tensors: CUDA0 model buffer size = 78.88 MiB

```

Edit:  
Your GPU supports bfloat16, but a similar situation may occur with float32 or so in Embedding models, for example. Unnecessary data movement between the CPU and GPU will result in a speed penalty.

> <https://github.com/ggml-org/llama.cpp/issues/9881#issuecomment-2414516092>
>
> \### What happened?
> 
> llama.cpp is running slow on NVIDIA A100 80GB GPU
> 
> Steps… to reproduce:
> 1. git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
> 2. mkdir build && cd build
> 3. cmake .. -DGGML\_CUDA=ON
> 4. make GGML\_CUDA=1
> 3. command: ./build/bin/llama-cli -m ../gguf\_files/llama-3-8B.gguf -t 6912 --ctx-size 50 --n\_predict 50 --prompt "There are two persons named ram and krishna"
> Here threads are set to 6912 since GPU has 6912 CUDA cores.
> 
> It is slow on gpu compared to cpu.
> On gpu eval time is around 0.07 tokens per second.
> Is this expected behaviour or any tweak should be done while building llama.cpp?
> 
> \### Name and Version
> 
> version: 3902 (c81f3bbb)
> built with cc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3) for aarch64-redhat-linux
> 
> \### What operating system are you seeing the problem on?
> 
> Linux
> 
> \### Relevant log output
> 
> \`\`\`shell
> \`\`\`

Bug? aroud cache.

> **[Missing something - Slow time to first token · ggml-org llama.cpp ·...](https://github.com/ggml-org/llama.cpp/discussions/10141)**
>
> Hello! I think I am missing something. I am trying to test out llamacpp but I am getting very long delays before getting a response. I have tried a few different things and I read through the docs,...

> <https://github.com/ggml-org/llama.cpp/issues/9492>
>
> \### What happened?
> 
> I'm using the \`openai\` library to interact with \`llama-serve…r\` docker image on an A6000:
> 
> \`docker run -p 8080:8080 --name llama-server -v ~/gguf\_models:/models --gpus all ghcr.io/ggerganov/llama.cpp:server-cuda -m models/Meta-Llama-3.1-70B-Instruct-Q4\_K\_L.gguf -c 65536 -fa --host 0.0.0.0 --port 8080 --n-gpu-layers 99 -ctk q4\_0 -ctv q4\_0 -t 4\`
> 
> The first request I send takes about 80 seconds, during which at first a single CPU core gets 100% load for maybe ~55s (with GPU usage at 0%) and only then the GPU kicks in. The second time I execute the exact same call, it takes ~26s to respond and starts with both CPU (one core 100%) and GPU (~87%) working a the same time.
> 
> The API call itself is:
> 
> \`\`\`
> import openai # openai-1.45.0
> 
> api\_url = 'http://x.x.x.x:8080'
> 
> client = openai.OpenAI(
> base\_url=f"{api\_url}/v1",
> api\_key = "unused"
> )
> 
> messages = \[
> {"role": "system", "content": "You are a helpful assistant. Your top priority is answering user questions truthfully, based solely on the information provided."},
> {"role": "user", "content": prompt} # prompt is around 450 words
> \]
> 
> completion = client.chat.completions.create(
> model=None,
> temperature=0.0,
> messages=messages
> ).choices\[0\].message
> 
> \`\`\`
> 
> \### Name and Version
> 
> $./llama-server --version
> version: 0 (unknown)
> built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86\_64-linux-gnu
> 
> ^^^ not very helpful but I have just pulled a fresh docker image today i.e. 15/09/2024: 
> \`docker pull ghcr.io/ggerganov/llama.cpp:server-cuda\`
> 
> \### What operating system are you seeing the problem on?
> 
> Linux
> 
> \### Relevant log output
> 
> \`\`\`shell
> warn: LLAMA\_ARG\_HOST environment variable is set, but will be overwritten by command line argument --host
> INFO \[main\] build info | tid="126087453233152" timestamp=1726381552 build=0 commit="unknown"
> INFO \[main\] system info | tid="126087453233152" timestamp=1726381552 n\_threads=4 n\_threads\_batch=4 total\_threads=36 system\_info="AVX = 1 | AVX\_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512\_VBMI = 0 | AVX512\_VNNI = 0 | AVX512\_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM\_FMA = 0 | F16C = 1 | FP16\_VA = 0 | RISCV\_VECT = 0 | WASM\_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL\_INT8 = 0 | LLAMAFILE = 1 | "
> INFO \[main\] HTTP server is listening | tid="126087453233152" timestamp=1726381552 n\_threads\_http="35" port="8080" hostname="0.0.0.0"
> INFO \[main\] loading model | tid="126087453233152" timestamp=1726381552 n\_threads\_http="35" port="8080" hostname="0.0.0.0"
> llama\_model\_loader: loaded meta data with 33 key-value pairs and 724 tensors from models/Meta-Llama-3.1-70B-Instruct-Q4\_K\_L.gguf (version GGUF V3 (latest))
> llama\_model\_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
> llama\_model\_loader: - kv 0: general.architecture str = llama
> llama\_model\_loader: - kv 1: general.type str = model
> llama\_model\_loader: - kv 2: general.name str = Meta Llama 3.1 70B Instruct
> llama\_model\_loader: - kv 3: general.finetune str = Instruct
> llama\_model\_loader: - kv 4: general.basename str = Meta-Llama-3.1
> llama\_model\_loader: - kv 5: general.size\_label str = 70B
> llama\_model\_loader: - kv 6: general.license str = llama3.1
> llama\_model\_loader: - kv 7: general.tags arr\[str,6\] = \["facebook", "meta", "pytorch", "llam...
> llama\_model\_loader: - kv 8: general.languages arr\[str,8\] = \["en", "de", "fr", "it", "pt", "hi", ...
> llama\_model\_loader: - kv 9: llama.block\_count u32 = 80
> llama\_model\_loader: - kv 10: llama.context\_length u32 = 131072
> llama\_model\_loader: - kv 11: llama.embedding\_length u32 = 8192
> llama\_model\_loader: - kv 12: llama.feed\_forward\_length u32 = 28672
> llama\_model\_loader: - kv 13: llama.attention.head\_count u32 = 64
> llama\_model\_loader: - kv 14: llama.attention.head\_count\_kv u32 = 8
> llama\_model\_loader: - kv 15: llama.rope.freq\_base f32 = 500000.000000
> llama\_model\_loader: - kv 16: llama.attention.layer\_norm\_rms\_epsilon f32 = 0.000010
> llama\_model\_loader: - kv 17: general.file\_type u32 = 15
> llama\_model\_loader: - kv 18: llama.vocab\_size u32 = 128256
> llama\_model\_loader: - kv 19: llama.rope.dimension\_count u32 = 128
> llama\_model\_loader: - kv 20: tokenizer.ggml.model str = gpt2
> llama\_model\_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe
> llama\_model\_loader: - kv 22: tokenizer.ggml.tokens arr\[str,128256\] = \["!", "\\"", "#", "$", "%", "&", "'", ...
> llama\_model\_loader: - kv 23: tokenizer.ggml.token\_type arr\[i32,128256\] = \[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
> llama\_model\_loader: - kv 24: tokenizer.ggml.merges arr\[str,280147\] = \["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
> llama\_model\_loader: - kv 25: tokenizer.ggml.bos\_token\_id u32 = 128000
> llama\_model\_loader: - kv 26: tokenizer.ggml.eos\_token\_id u32 = 128009
> llama\_model\_loader: - kv 27: tokenizer.chat\_template str = {{- bos\_token }}\\n{%- if custom\_tools ...
> llama\_model\_loader: - kv 28: general.quantization\_version u32 = 2
> llama\_model\_loader: - kv 29: quantize.imatrix.file str = /models\_out/Meta-Llama-3.1-70B-Instru...
> llama\_model\_loader: - kv 30: quantize.imatrix.dataset str = /training\_dir/calibration\_datav3.txt
> llama\_model\_loader: - kv 31: quantize.imatrix.entries\_count i32 = 560
> llama\_model\_loader: - kv 32: quantize.imatrix.chunks\_count i32 = 125
> llama\_model\_loader: - type f32: 162 tensors
> llama\_model\_loader: - type q8\_0: 2 tensors
> llama\_model\_loader: - type q4\_K: 440 tensors
> llama\_model\_loader: - type q5\_K: 40 tensors
> llama\_model\_loader: - type q6\_K: 80 tensors
> llm\_load\_vocab: special tokens cache size = 256
> llm\_load\_vocab: token to piece cache size = 0.7999 MB
> llm\_load\_print\_meta: format = GGUF V3 (latest)
> llm\_load\_print\_meta: arch = llama
> llm\_load\_print\_meta: vocab type = BPE
> llm\_load\_print\_meta: n\_vocab = 128256
> llm\_load\_print\_meta: n\_merges = 280147
> llm\_load\_print\_meta: vocab\_only = 0
> llm\_load\_print\_meta: n\_ctx\_train = 131072
> llm\_load\_print\_meta: n\_embd = 8192
> llm\_load\_print\_meta: n\_layer = 80
> llm\_load\_print\_meta: n\_head = 64
> llm\_load\_print\_meta: n\_head\_kv = 8
> llm\_load\_print\_meta: n\_rot = 128
> llm\_load\_print\_meta: n\_swa = 0
> llm\_load\_print\_meta: n\_embd\_head\_k = 128
> llm\_load\_print\_meta: n\_embd\_head\_v = 128
> llm\_load\_print\_meta: n\_gqa = 8
> llm\_load\_print\_meta: n\_embd\_k\_gqa = 1024
> llm\_load\_print\_meta: n\_embd\_v\_gqa = 1024
> llm\_load\_print\_meta: f\_norm\_eps = 0.0e+00
> llm\_load\_print\_meta: f\_norm\_rms\_eps = 1.0e-05
> llm\_load\_print\_meta: f\_clamp\_kqv = 0.0e+00
> llm\_load\_print\_meta: f\_max\_alibi\_bias = 0.0e+00
> llm\_load\_print\_meta: f\_logit\_scale = 0.0e+00
> llm\_load\_print\_meta: n\_ff = 28672
> llm\_load\_print\_meta: n\_expert = 0
> llm\_load\_print\_meta: n\_expert\_used = 0
> llm\_load\_print\_meta: causal attn = 1
> llm\_load\_print\_meta: pooling type = 0
> llm\_load\_print\_meta: rope type = 0
> llm\_load\_print\_meta: rope scaling = linear
> llm\_load\_print\_meta: freq\_base\_train = 500000.0
> llm\_load\_print\_meta: freq\_scale\_train = 1
> llm\_load\_print\_meta: n\_ctx\_orig\_yarn = 131072
> llm\_load\_print\_meta: rope\_finetuned = unknown
> llm\_load\_print\_meta: ssm\_d\_conv = 0
> llm\_load\_print\_meta: ssm\_d\_inner = 0
> llm\_load\_print\_meta: ssm\_d\_state = 0
> llm\_load\_print\_meta: ssm\_dt\_rank = 0
> llm\_load\_print\_meta: ssm\_dt\_b\_c\_rms = 0
> llm\_load\_print\_meta: model type = 70B
> llm\_load\_print\_meta: model ftype = Q4\_K - Medium
> llm\_load\_print\_meta: model params = 70.55 B
> llm\_load\_print\_meta: model size = 40.32 GiB (4.91 BPW)
> llm\_load\_print\_meta: general.name = Meta Llama 3.1 70B Instruct
> llm\_load\_print\_meta: BOS token = 128000 '\<|begin\_of\_text|\>'
> llm\_load\_print\_meta: EOS token = 128009 '\<|eot\_id|\>'
> llm\_load\_print\_meta: LF token = 128 'Ä'
> llm\_load\_print\_meta: EOT token = 128009 '\<|eot\_id|\>'
> llm\_load\_print\_meta: max token length = 256
> ggml\_cuda\_init: GGML\_CUDA\_FORCE\_MMQ: no
> ggml\_cuda\_init: GGML\_CUDA\_FORCE\_CUBLAS: no
> ggml\_cuda\_init: found 1 CUDA devices:
> Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
> llm\_load\_tensors: ggml ctx size = 0.68 MiB
> llm\_load\_tensors: offloading 80 repeating layers to GPU
> llm\_load\_tensors: offloading non-repeating layers to GPU
> llm\_load\_tensors: offloaded 81/81 layers to GPU
> llm\_load\_tensors: CPU buffer size = 1064.62 MiB
> llm\_load\_tensors: CUDA0 buffer size = 40222.18 MiB
> .................................................................................................
> llama\_new\_context\_with\_model: n\_ctx = 65536
> llama\_new\_context\_with\_model: n\_batch = 2048
> llama\_new\_context\_with\_model: n\_ubatch = 512
> llama\_new\_context\_with\_model: flash\_attn = 1
> llama\_new\_context\_with\_model: freq\_base = 500000.0
> llama\_new\_context\_with\_model: freq\_scale = 1
> llama\_kv\_cache\_init: CUDA0 KV buffer size = 5760.00 MiB
> llama\_new\_context\_with\_model: KV self size = 5760.00 MiB, K (q4\_0): 2880.00 MiB, V (q4\_0): 2880.00 MiB
> llama\_new\_context\_with\_model: CUDA\_Host output buffer size = 0.98 MiB
> llama\_new\_context\_with\_model: CUDA0 compute buffer size = 266.50 MiB
> llama\_new\_context\_with\_model: CUDA\_Host compute buffer size = 144.01 MiB
> llama\_new\_context\_with\_model: graph nodes = 2247
> llama\_new\_context\_with\_model: graph splits = 2
> INFO \[init\] initializing slots | tid="126087453233152" timestamp=1726381574 n\_slots=1
> INFO \[init\] new slot | tid="126087453233152" timestamp=1726381574 id\_slot=0 n\_ctx\_slot=65536
> INFO \[main\] model loaded | tid="126087453233152" timestamp=1726381574
> INFO \[main\] chat template | tid="126087453233152" timestamp=1726381574 chat\_example="\<|start\_header\_id|\>system\<|end\_header\_id|\>\\n\\nYou are a helpful assistant\<|eot\_id|\>\<|start\_header\_id|\>user\<|end\_header\_id|\>\\n\\nHello\<|eot\_id|\>\<|start\_header\_id|\>assistant\<|end\_header\_id|\>\\n\\nHi there\<|eot\_id|\>\<|start\_header\_id|\>user\<|end\_header\_id|\>\\n\\nHow are you?\<|eot\_id|\>\<|start\_header\_id|\>assistant\<|end\_header\_id|\>\\n\\n" built\_in=true
> INFO \[update\_slots\] all slots are idle | tid="126087453233152" timestamp=1726381574
> INFO \[log\_server\_request\] request | tid="126086914506752" timestamp=1726381582 remote\_addr="127.0.0.1" remote\_port=35744 status=200 method="GET" path="/health" params={}
> INFO \[launch\_slot\_with\_task\] slot is processing task | tid="126087453233152" timestamp=1726381587 id\_slot=0 id\_task=0
> INFO \[update\_slots\] kv cache rm \[p0, end) | tid="126087453233152" timestamp=1726381587 id\_slot=0 id\_task=0 p0=0
> INFO \[log\_server\_request\] request | tid="126086815940608" timestamp=1726381613 remote\_addr="127.0.0.1" remote\_port=59038 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086805454848" timestamp=1726381643 remote\_addr="127.0.0.1" remote\_port=35914 status=200 method="GET" path="/health" params={}
> INFO \[release\] slot released | tid="126087453233152" timestamp=1726381666 id\_slot=0 id\_task=0 n\_past=1011 truncated=false
> INFO \[print\_timings\] prompt eval time = 39264.22 ms / 685 tokens ( 57.32 ms per token, 17.45 tokens per second) | tid="126087453233152" timestamp=1726381666 id\_slot=0 id\_task=0 t\_prompt\_processing=39264.217 n\_prompt\_tokens\_processed=685 t\_token=57.320024817518245 n\_tokens\_second=17.44591010181102
> INFO \[print\_timings\] generation eval time = 40157.40 ms / 327 runs ( 122.81 ms per token, 8.14 tokens per second) | tid="126087453233152" timestamp=1726381666 id\_slot=0 id\_task=0 t\_token\_generation=40157.401 n\_decoded=327 t\_token=122.80550764525994 n\_tokens\_second=8.142957259609506
> INFO \[print\_timings\] total time = 79421.62 ms | tid="126087453233152" timestamp=1726381666 id\_slot=0 id\_task=0 t\_prompt\_processing=39264.217 t\_token\_generation=40157.401 t\_total=79421.61799999999
> INFO \[update\_slots\] all slots are idle | tid="126087453233152" timestamp=1726381666
> INFO \[log\_server\_request\] request | tid="126086904020992" timestamp=1726381666 remote\_addr="10.147.20.194" remote\_port=47472 status=200 method="POST" path="/v1/chat/completions" params={}
> INFO \[launch\_slot\_with\_task\] slot is processing task | tid="126087453233152" timestamp=1726381670 id\_slot=0 id\_task=328
> INFO \[update\_slots\] kv cache rm \[p0, end) | tid="126087453233152" timestamp=1726381670 id\_slot=0 id\_task=328 p0=0
> INFO \[log\_server\_request\] request | tid="126086794969088" timestamp=1726381673 remote\_addr="127.0.0.1" remote\_port=44812 status=200 method="GET" path="/health" params={}
> INFO \[release\] slot released | tid="126087453233152" timestamp=1726381696 id\_slot=0 id\_task=328 n\_past=1011 truncated=false
> INFO \[print\_timings\] prompt eval time = 1319.62 ms / 685 tokens ( 1.93 ms per token, 519.09 tokens per second) | tid="126087453233152" timestamp=1726381696 id\_slot=0 id\_task=328 t\_prompt\_processing=1319.625 n\_prompt\_tokens\_processed=685 t\_token=1.9264598540145985 n\_tokens\_second=519.0868617978592
> INFO \[print\_timings\] generation eval time = 24725.42 ms / 327 runs ( 75.61 ms per token, 13.23 tokens per second) | tid="126087453233152" timestamp=1726381696 id\_slot=0 id\_task=328 t\_token\_generation=24725.416 n\_decoded=327 t\_token=75.61289296636086 n\_tokens\_second=13.225257767149397
> INFO \[print\_timings\] total time = 26045.04 ms | tid="126087453233152" timestamp=1726381696 id\_slot=0 id\_task=328 t\_prompt\_processing=1319.625 t\_token\_generation=24725.416 t\_total=26045.041
> INFO \[update\_slots\] all slots are idle | tid="126087453233152" timestamp=1726381696
> INFO \[log\_server\_request\] request | tid="126086904020992" timestamp=1726381696 remote\_addr="10.147.20.194" remote\_port=47472 status=200 method="POST" path="/v1/chat/completions" params={}
> INFO \[log\_server\_request\] request | tid="126086763511808" timestamp=1726381703 remote\_addr="127.0.0.1" remote\_port=57226 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086773997568" timestamp=1726381733 remote\_addr="127.0.0.1" remote\_port=58518 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086784483328" timestamp=1726381763 remote\_addr="127.0.0.1" remote\_port=36938 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086753026048" timestamp=1726381793 remote\_addr="127.0.0.1" remote\_port=34366 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086742540288" timestamp=1726381823 remote\_addr="127.0.0.1" remote\_port=36110 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086721568768" timestamp=1726381853 remote\_addr="127.0.0.1" remote\_port=41308 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086732054528" timestamp=1726381883 remote\_addr="127.0.0.1" remote\_port=44500 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086711083008" timestamp=1726381913 remote\_addr="127.0.0.1" remote\_port=51098 status=200 method="GET" path="/health" params={}
> INFO \[log\_server\_request\] request | tid="126086700597248" timestamp=1726381943 remote\_addr="127.0.0.1" remote\_port=39654 status=200 method="GET" path="/health" params={}
> \`\`\`

---

<div class="post-metadata">

**Author:** ![User515](https://avatars.discourse-cdn.com/v4/letter/u/6bbea6/32.png) [@User515](https://discuss.huggingface.co/u/User515)\
**Post date:** [June 5, 2025, 3:36am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/11 "2025-06-05T03:36:58Z")

</div>

Hmm well I tried a few things.

First, I tried using a few different embedding models: two different nomic-text-1.5’s, MiniLM-L6-v2, and one more that was too big to load in my VRAM. No change.

Next, I tried loading the embedding model as CPU (i.e.  
`--nn-preload default:GGML:GPU:Llama-3.2-3B-Instruct-Q5_K_M.gguf \ --nn-preload embedding:GGML:CPU:nomic-embed-text-v1.5.Q5_K_M.gguf \`  
) to see if I could speed it up just by not repeatedly transferring between GPU and CPU; still extremely slow while chunking the text file and nvidia-smi reports a very similar VRAM consumption as before.

I tried to see if the `rag-api-server.wasm` could be loaded with embeddings models only (for testing) and then asked to chunk a file but it requires the chat model is also loaded.

Lastly, I looked to see if I could adjust bfloat16 vs float32 usage anywhere but my understanding is that these vector types are compiled into the GGUF (or generated during training?) and llama-core libraries, so I don’t think I can change them. I don’t see any reference to them in the script that WASMEdge supplies to build WASMEdge and the NN-GGML plugin, either.

What would be an obvious next step for debugging? Or a workaround to get a reasonably quick RAG going on middle-range consumer hardware?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 5, 2025, 5:00am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/12 "2025-06-05T05:00:58Z")

</div>

The behavior of the embedding model on the GPU in WasmEdge seems strange…  
I’m not sure if the issue is with WasmEdge, Llama.cpp, or something else…  
Considering that the calculation results are being output, it seems like the model is being loaded to VRAM, but the GPU isn’t being utilized properly during the computation, resulting in unusual behavior.

> Or a workaround to get a reasonably quick RAG going on middle-range consumer hardware?

If you’re not tied to WasmEdge, Ollama could be a lighter option for simple tasks, and vLLM could be used for longer token lengths (processing text longer than 8,000 tokens). In either case, there should be no hardware specifications issues.

> **[Embedding models · Ollama Blog](https://ollama.com/blog/embedding-models)**
>
> Embedding models are available in Ollama, making it easy to generate vector embeddings for use in search and retrieval augmented generation (RAG) applications.

> **[LangChain - vLLM](https://docs.vllm.ai/en/stable/serving/integrations/langchain.html)**

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 5, 2025, 5:37am UTC](https://discuss.huggingface.co/t/unable-to-push-embeddings-onto-gpu/157574/13 "2025-06-05T05:37:50Z")

</div>

Probably I found similar unresolved issue.

> <https://github.com/LlamaEdge/rag-api-server/issues/9>
>
> after try step from readme
> \`\`\`
> curl -X POST http://127.0.0.1:8080/v1/create/ra…g -F "file=@paris.txt"
> \`\`\`
> It took 590824.84 ms = nearly 1 minute for only chunking 306 lines (91KB) file on m3 max.
> 
> This is just me or I miss some flag?