kasys/llm-jp-4-8b-thinking-q4f16_1-MLC

llm-jp-4-8b-thinking · q4f16_1 · MLC (WebLLM / WebGPU)

A conversion of llm-jp/llm-jp-4-8b-thinking to the MLC LLM q4f16_1 format, runnable entirely in the browser (WebGPU) via WebLLM.

Base model

• llm-jp/llm-jp-4-8b-thinking (LLM-jp, National Institute of Informatics)
• 8.59B parameters, Llama architecture (GQA, 32 layers, hidden 4096, vocab 196608, ctx 65536)
• A thinking / reasoning model using the OpenAI Harmony response format (analysis / final channels)
• License: Apache-2.0

How it was made

Converted with a source build of mlc-llm (TVM Unity):

1. mlc_llm convert_weight --quantization q4f16_1 --model-type llama — 4-bit weight quantization (~4.5 GB, 3.5 bits per parameter)
2. mlc_llm gen_config --conv-template llm-jp-4 --context-window-size 4096 — using a custom conversation template llm-jp-4 that reproduces the base model's Harmony chat format
3. mlc_llm compile --device webgpu — produces the WebGPU wasm library (*-webgpu.wasm)

The llm-jp-4 conversation template mirrors the base model's chat_template.jinja: it uses <|start|> / <|channel|> / <|message|> / <|end|> and stops generation on <|return|> (id 2).

Usage (WebLLM)

import * as webllm from "@mlc-ai/web-llm";

const MODEL_ID = "llm-jp-4-8b-thinking-q4f16_1-MLC";
const REPO = "https://huggingface.co/kasys/llm-jp-4-8b-thinking-q4f16_1-MLC";

const appConfig: webllm.AppConfig = {
  model_list: [{
    model: REPO,
    model_id: MODEL_ID,
    model_lib: `${REPO}/resolve/main/llm-jp-4-8b-thinking-q4f16_1-webgpu.wasm`,
  }],
};

const engine = await webllm.CreateMLCEngine(MODEL_ID, { appConfig });
// The Harmony system prompt is injected by the template, so pass only the user turn.
const res = await engine.chat.completions.create({
  messages: [{ role: "user", content: "富士山の標高は？" }],
});

The model emits an analysis (reasoning) channel followed by a final (answer) channel. To show only the answer, extract the final channel.

Client requirements

• A WebGPU-capable browser (Chrome / Edge 113+, or recent Safari)
• A GPU with the shader-f16 feature and ~6 GB of available GPU memory (4.5 GB weights + KV cache). Rough guide: an 8 GB+ discrete GPU, or Apple Silicon with 16 GB+ unified memory
• The model (~4.5 GB) is downloaded on first use and cached by the browser afterwards

License

Apache-2.0, following the base model.