Instructions to use mlx-community/diffusiongemma-26B-A4B-it-5bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/diffusiongemma-26B-A4B-it-5bit with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("mlx-community/diffusiongemma-26B-A4B-it-5bit")
config = load_config("mlx-community/diffusiongemma-26B-A4B-it-5bit")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use mlx-community/diffusiongemma-26B-A4B-it-5bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/diffusiongemma-26B-A4B-it-5bit"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mlx-community/diffusiongemma-26B-A4B-it-5bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use mlx-community/diffusiongemma-26B-A4B-it-5bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/diffusiongemma-26B-A4B-it-5bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mlx-community/diffusiongemma-26B-A4B-it-5bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mlx-community/diffusiongemma-26B-A4B-it-5bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/diffusiongemma-26B-A4B-it-5bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mlx-community/diffusiongemma-26B-A4B-it-5bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

prince-canuma commited on Jun 11

Commit

6156b10

verified ·

1 Parent(s): 4201d02

Upload folder using huggingface_hub

Browse files

Files changed (6) hide show

config.json +1094 -166
model-00001-of-00004.safetensors +2 -2
model-00002-of-00004.safetensors +2 -2
model-00003-of-00004.safetensors +2 -2
model-00004-of-00004.safetensors +2 -2
model.safetensors.index.json +115 -115

config.json CHANGED Viewed

@@ -37,6 +37,26 @@
         "group_size": 64,
         "bits": 5,
         "mode": "affine",
         "model.decoder.layers.0.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -53,6 +73,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.1.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -69,6 +105,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.2.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -85,6 +137,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.3.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -101,6 +169,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.4.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -117,6 +201,18 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.5.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -133,6 +229,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.6.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -149,6 +261,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.7.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -165,6 +293,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.8.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -181,6 +325,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.9.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -197,6 +357,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.10.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -213,6 +389,18 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.11.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -229,6 +417,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.12.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -245,6 +449,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.13.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -261,6 +481,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.14.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -277,6 +513,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.15.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -293,6 +545,22 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.16.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -309,6 +577,18 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.17.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
@@ -325,648 +605,1284 @@
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.28.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.28.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.28.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.28.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.29.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.29.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.29.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.29.router.proj": {
             "group_size": 64,
             "bits": 8
-        }
-    },
-    "quantization_config": {
-        "group_size": 64,
-        "bits": 5,
-        "mode": "affine",
-        "model.decoder.layers.0.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.0.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.0.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.0.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.1.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.1.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.1.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.1.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.2.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.2.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.2.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.2.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.3.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.3.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.3.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.3.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.4.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.4.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.4.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.4.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.5.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.5.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.5.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.5.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.6.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.6.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.6.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.6.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.7.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.7.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.7.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.7.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.8.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.8.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.8.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.8.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.9.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.9.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.9.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.9.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.10.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.10.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.10.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.10.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.11.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.11.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.11.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.11.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.12.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.12.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.12.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.12.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.13.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.13.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.13.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.13.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.14.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.14.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.14.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.14.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.15.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.15.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.15.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.15.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.16.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.16.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.16.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.16.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.17.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.17.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.17.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.17.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.18.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.19.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.20.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.21.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.22.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.23.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.24.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.25.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.26.router.proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
-        "model.decoder.layers.27.router.proj": {
             "group_size": 64,
             "bits": 8
         },
@@ -986,6 +1902,18 @@
             "group_size": 64,
             "bits": 8
         },
         "model.decoder.layers.29.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8

         "group_size": 64,
         "bits": 5,
         "mode": "affine",
+        "model.decoder.embed_tokens": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.0.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.1.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.1.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.2.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.2.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.3.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.3.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.4.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.4.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.5.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.5.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.6.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.6.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.7.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.7.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.8.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.8.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.8.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.8.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.9.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.9.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.9.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.9.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.10.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.10.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.10.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.10.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.11.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.11.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.11.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.12.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.12.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.12.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.12.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.13.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.13.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.13.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.13.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.14.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.14.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.14.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.14.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.15.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.15.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.15.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.15.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.16.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.16.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.16.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.16.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.17.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.17.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.17.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.18.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.19.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.20.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.21.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.22.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.23.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.24.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.25.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.26.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.27.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.28.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        }
+    },
+    "quantization_config": {
+        "group_size": 64,
+        "bits": 5,
+        "mode": "affine",
+        "model.decoder.embed_tokens": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.0.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.1.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.2.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.3.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.4.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.5.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.6.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.v_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.7.router.proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.8.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.8.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.8.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.9.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.10.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.11.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.12.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.13.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
+        },
+        "model.decoder.layers.14.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.14.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.15.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.16.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.17.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.18.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.19.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.20.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.21.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.22.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.23.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.24.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.25.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.26.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.mlp.up_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.mlp.down_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.27.router.proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.28.self_attn.q_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.28.self_attn.k_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.28.self_attn.v_proj": {
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.28.self_attn.o_proj": {
             "group_size": 64,
             "bits": 8
         },
             "group_size": 64,
             "bits": 8
         },
+        "model.decoder.layers.29.self_attn.q_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.self_attn.k_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "model.decoder.layers.29.self_attn.o_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
         "model.decoder.layers.29.mlp.gate_proj": {
             "group_size": 64,
             "bits": 8

model-00001-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a30b9bacc1c74568d825ed830d417cc0a580af5c82257b63d3e4bd98eeec0485
-size 5093001807

 version https://git-lfs.github.com/spec/v1
+oid sha256:7124986683c442659692863370157777acda16b7ae1f30f8b188def2b90e6f8a
+size 5261403888

model-00002-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d3d737ff1c982c5f4fa7240558bc9ce9c4a1867042e2c74e0a5a64b5b2e61a4b
-size 5118792725

 version https://git-lfs.github.com/spec/v1
+oid sha256:2c29bb68b6b4c0231c84bd22917bd5e94924c5b8c6b5b3421fce194d02523a18
+size 5231071623

model-00003-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1b1992b12de471d15115f0dbbe46fdf3365a84be551e515df311c75b8cb21557
-size 5108879738

 version https://git-lfs.github.com/spec/v1
+oid sha256:377406d17476c7308739f5ab4413215223f62963d2ec311614942ec138d9e5a4
+size 5246391324

model-00004-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e0f470ef3c02c4704faede735053ca96dad2589a60911e34abb4bf6a4fd294b8
-size 3386623829

 version https://git-lfs.github.com/spec/v1
+oid sha256:203fdfa2eacde7f193363a863120144b2a0665fd9f55a14187f00200a3db097c
+size 3661572919

model.safetensors.index.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
     "metadata": {
-        "total_size": 18707087064
     },
     "weight_map": {
         "model.decoder.embed_tokens.biases": "model-00001-of-00004.safetensors",
@@ -339,14 +339,14 @@
         "model.decoder.layers.15.self_attn.v_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.15.self_attn.v_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.experts.down_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.experts.down_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.experts.down_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.experts.gate_up_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.experts.gate_up_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.experts.gate_up_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.layer_scalar": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.mlp.down_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.mlp.down_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
@@ -358,10 +358,10 @@
         "model.decoder.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.post_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.post_feedforward_layernorm_1.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.post_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.pre_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.16.pre_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.router.per_expert_scale": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.router.proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.router.proj.scales": "model-00002-of-00004.safetensors",
@@ -387,39 +387,39 @@
         "model.decoder.layers.17.experts.gate_up_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.experts.gate_up_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.experts.gate_up_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.17.layer_scalar": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.down_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.down_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.gate_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.gate_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.up_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.up_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.post_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.17.post_feedforward_layernorm_1.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.post_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.17.pre_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.17.pre_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.17.router.per_expert_scale": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.router.proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.router.proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.router.proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.router.scale": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.k_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.k_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.o_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.o_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.q_proj.biases": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.q_proj.scales": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.18.experts.down_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.18.experts.down_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.18.experts.down_proj.weight": "model-00003-of-00004.safetensors",
@@ -753,14 +753,14 @@
         "model.decoder.layers.24.self_attn.v_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.24.self_attn.v_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.experts.down_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.experts.down_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.experts.down_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.experts.gate_up_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.experts.gate_up_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.experts.gate_up_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.layer_scalar": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.mlp.down_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.mlp.down_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
@@ -772,10 +772,10 @@
         "model.decoder.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.post_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.post_feedforward_layernorm_1.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.post_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.pre_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.25.pre_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.router.per_expert_scale": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.router.proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.router.proj.scales": "model-00003-of-00004.safetensors",
@@ -801,42 +801,42 @@
         "model.decoder.layers.26.experts.gate_up_proj.biases": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.experts.gate_up_proj.scales": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.experts.gate_up_proj.weight": "model-00004-of-00004.safetensors",
-        "model.decoder.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.26.layer_scalar": "model-00004-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.down_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.down_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.gate_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.gate_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.up_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.up_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.post_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.26.post_feedforward_layernorm_1.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.post_feedforward_layernorm_2.weight": "model-00004-of-00004.safetensors",
-        "model.decoder.layers.26.pre_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.26.pre_feedforward_layernorm_2.weight": "model-00004-of-00004.safetensors",
-        "model.decoder.layers.26.router.per_expert_scale": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.router.proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.router.proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.router.proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.router.scale": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.k_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.k_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.o_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.o_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.q_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.q_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.v_proj.biases": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.v_proj.scales": "model-00003-of-00004.safetensors",
-        "model.decoder.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.27.experts.down_proj.biases": "model-00004-of-00004.safetensors",
         "model.decoder.layers.27.experts.down_proj.scales": "model-00004-of-00004.safetensors",
         "model.decoder.layers.27.experts.down_proj.weight": "model-00004-of-00004.safetensors",
@@ -1125,14 +1125,14 @@
         "model.decoder.layers.6.self_attn.v_proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.6.self_attn.v_proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.experts.down_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.experts.down_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.experts.down_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.experts.gate_up_proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.experts.gate_up_proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.experts.gate_up_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.layer_scalar": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.mlp.down_proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.mlp.down_proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
@@ -1144,10 +1144,10 @@
         "model.decoder.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.post_feedforward_layernorm_1.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.post_feedforward_layernorm_2.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.pre_feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.7.pre_feedforward_layernorm_2.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.router.per_expert_scale": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.router.proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.router.proj.scales": "model-00001-of-00004.safetensors",
@@ -1173,42 +1173,42 @@
         "model.decoder.layers.8.experts.gate_up_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.experts.gate_up_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.experts.gate_up_proj.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.8.layer_scalar": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.down_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.down_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.gate_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.gate_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.up_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.up_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.8.post_feedforward_layernorm_1.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.post_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.8.pre_feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.8.pre_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
-        "model.decoder.layers.8.router.per_expert_scale": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.router.proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.router.proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.router.proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.router.scale": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.k_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.k_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.o_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.o_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.q_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.q_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.v_proj.biases": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.v_proj.scales": "model-00001-of-00004.safetensors",
-        "model.decoder.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.9.experts.down_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.9.experts.down_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.9.experts.down_proj.weight": "model-00002-of-00004.safetensors",

 {
     "metadata": {
+        "total_size": 19400228568
     },
     "weight_map": {
         "model.decoder.embed_tokens.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.15.self_attn.v_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.15.self_attn.v_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.16.experts.down_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.16.experts.down_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.16.experts.down_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.16.experts.gate_up_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.experts.gate_up_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.experts.gate_up_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.16.layer_scalar": "model-00003-of-00004.safetensors",
         "model.decoder.layers.16.mlp.down_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.mlp.down_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.post_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.16.post_feedforward_layernorm_1.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.16.post_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.16.pre_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.16.pre_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.16.router.per_expert_scale": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.router.proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.16.router.proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.17.experts.gate_up_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.experts.gate_up_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.experts.gate_up_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.input_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.layer_scalar": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.down_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.down_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.gate_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.gate_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.up_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.up_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.post_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.post_feedforward_layernorm_1.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.post_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.pre_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.17.pre_feedforward_layernorm_2.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.router.per_expert_scale": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.router.proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.router.proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.router.proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.router.scale": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.k_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.k_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.o_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.o_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.q_proj.biases": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.q_proj.scales": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.17.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.18.experts.down_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.18.experts.down_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.18.experts.down_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.24.self_attn.v_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.24.self_attn.v_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.25.experts.down_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.25.experts.down_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.25.experts.down_proj.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.25.experts.gate_up_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.experts.gate_up_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.experts.gate_up_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.25.layer_scalar": "model-00004-of-00004.safetensors",
         "model.decoder.layers.25.mlp.down_proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.mlp.down_proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.post_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.25.post_feedforward_layernorm_1.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.25.post_feedforward_layernorm_2.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.25.pre_feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
+        "model.decoder.layers.25.pre_feedforward_layernorm_2.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.25.router.per_expert_scale": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.router.proj.biases": "model-00003-of-00004.safetensors",
         "model.decoder.layers.25.router.proj.scales": "model-00003-of-00004.safetensors",
         "model.decoder.layers.26.experts.gate_up_proj.biases": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.experts.gate_up_proj.scales": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.experts.gate_up_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.input_layernorm.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.layer_scalar": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.down_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.down_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.gate_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.gate_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.up_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.up_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.post_feedforward_layernorm.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.post_feedforward_layernorm_1.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.post_feedforward_layernorm_2.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.pre_feedforward_layernorm.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.26.pre_feedforward_layernorm_2.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.router.per_expert_scale": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.router.proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.router.proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.router.proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.router.scale": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.k_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.k_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.o_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.o_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.q_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.q_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.v_proj.biases": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.v_proj.scales": "model-00004-of-00004.safetensors",
+        "model.decoder.layers.26.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.27.experts.down_proj.biases": "model-00004-of-00004.safetensors",
         "model.decoder.layers.27.experts.down_proj.scales": "model-00004-of-00004.safetensors",
         "model.decoder.layers.27.experts.down_proj.weight": "model-00004-of-00004.safetensors",
         "model.decoder.layers.6.self_attn.v_proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.6.self_attn.v_proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+        "model.decoder.layers.7.experts.down_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.7.experts.down_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.7.experts.down_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.7.experts.gate_up_proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.experts.gate_up_proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.experts.gate_up_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+        "model.decoder.layers.7.layer_scalar": "model-00002-of-00004.safetensors",
         "model.decoder.layers.7.mlp.down_proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.mlp.down_proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
+        "model.decoder.layers.7.post_feedforward_layernorm_1.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.7.post_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.7.pre_feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
+        "model.decoder.layers.7.pre_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.7.router.per_expert_scale": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.router.proj.biases": "model-00001-of-00004.safetensors",
         "model.decoder.layers.7.router.proj.scales": "model-00001-of-00004.safetensors",
         "model.decoder.layers.8.experts.gate_up_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.experts.gate_up_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.experts.gate_up_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.layer_scalar": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.down_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.down_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.gate_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.gate_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.up_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.up_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.post_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.post_feedforward_layernorm_1.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.post_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.pre_feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.8.pre_feedforward_layernorm_2.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.router.per_expert_scale": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.router.proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.router.proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.router.proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.router.scale": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.k_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.k_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.o_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.o_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.q_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.q_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.v_proj.biases": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.v_proj.scales": "model-00002-of-00004.safetensors",
+        "model.decoder.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
         "model.decoder.layers.9.experts.down_proj.biases": "model-00002-of-00004.safetensors",
         "model.decoder.layers.9.experts.down_proj.scales": "model-00002-of-00004.safetensors",
         "model.decoder.layers.9.experts.down_proj.weight": "model-00002-of-00004.safetensors",