Update README.md

README.md

@@ -1,28 +1,1173 @@
-# Stentor-12M
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- text-generation
+- llama
+- small-language-model
+- efficient
+- edge-deployment
+- speculative-decoding
+- tiny-model
+- 12m-parameters
+- kaggle-trained
+- educational
+- research
+- low-resource
+- cpu-inference
+- mobile-deployment
+- stentor2
+- tokenmonster
+pipeline_tag: text-generation
+datasets:
+- StentorLabs/StenCore
+- epfml/FineWeb-HQ
+widget:
+- text: "Once upon a time"
+  example_title: "Story Generation"
+- text: "Explain neural networks in simple terms."
+  example_title: "Toy Explanation (Often Wrong)"
+- text: "def fibonacci(n):"
+  example_title: "Code Continuation"
+- text: "The laws of thermodynamics describe"
+  example_title: "Science Continuation"
+model_card_authors:
+- StentorLabs
+model-index:
+- name: Stentor2-12M
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      name: FineWeb-HQ / StenCore (validation split)
+      type: epfml/FineWeb-HQ
+    metrics:
+    - name: Final Validation Loss
+      type: loss
+      value: 3.2565
+    - name: Final Perplexity
+      type: perplexity
+      value: 25.96
+    - name: Best Checkpoint Validation Loss
+      type: loss
+      value: 3.2814
+    - name: BLiMP Overall Accuracy (67 tasks)
+      type: accuracy
+      value: 68.95
+---
 
-This checkpoint was produced by the Stentor-12M training pipeline.
+# Stentor2-12M
 
-## Metadata
+![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)
+![Model Size](https://img.shields.io/badge/parameters-12.3M-green.svg)
+![Training Time](https://img.shields.io/badge/training-2.13h-orange.svg)
+![Hardware](https://img.shields.io/badge/hardware-2x%20T4-red.svg)
+![Context Length](https://img.shields.io/badge/context-1024%20tokens-purple.svg)
+![Vocab Size](https://img.shields.io/badge/vocab-8064%20tokens-blue.svg)
+[![Hugging Face](https://img.shields.io/badge/🤗-Hugging%20Face-yellow.svg)](https://huggingface.co/StentorLabs)
+![BLiMP](https://img.shields.io/badge/BLiMP-68.95%25-brightgreen.svg)
+
+> 🔬 **Research Artifact — Not a Production Model.** This is a base language model released for research, experimentation, and community use. It is not suitable for deployment in any user-facing application without significant additional work. See [Intended Uses](#use-cases--intended-uses) for details.
+
+---
+
+## Table of Contents
+
+1. [What Is This?](#what-is-this)
+2. [The Core Design Insight: Vocabulary Efficiency](#the-core-design-insight-vocabulary-efficiency)
+3. [Why Full MHA Instead of GQA](#why-full-mha-instead-of-gqa)
+4. [Head-to-Head: Stentor v1 → Stentor2-Preview → Stentor2-12M](#head-to-head)
+5. [Quick Start](#quick-start)
+6. [Important Limitations](#important-limitations)
+7. [Model Architecture — Full Specification](#model-architecture--full-specification)
+8. [The Tokenizer: TokenMonster](#the-tokenizer-tokenmonster)
+9. [Training Infrastructure](#training-infrastructure)
+10. [Training Hyperparameters — Complete Reference](#training-hyperparameters--complete-reference)
+11. [The T4 Mixed-Precision Recipe — Deep Dive](#the-t4-mixed-precision-recipe--deep-dive)
+12. [Data Pipeline](#data-pipeline)
+13. [Weight Initialization](#weight-initialization)
+14. [Evaluation & Results](#evaluation--results)
+15. [BLiMP Benchmark](#blimp-benchmark)
+16. [Training Dynamics](#training-dynamics)
+17. [Use Cases & Intended Uses](#use-cases--intended-uses)
+18. [Out-of-Scope Uses](#out-of-scope-uses)
+19. [Ethical Considerations & Societal Impact](#ethical-considerations--societal-impact)
+20. [Inference Guide](#inference-guide)
+21. [Real Model Responses](#real-model-responses)
+22. [Quantization](#quantization)
+23. [Format Conversion](#format-conversion)
+24. [Speculative Decoding](#speculative-decoding)
+25. [Bias, Risks & Limitations](#bias-risks--limitations)
+26. [Related Work](#related-work)
+27. [Environmental Impact](#environmental-impact)
+28. [Citation](#citation)
+
+---
+
+## What Is This?
+
+Stentor2-12M is the official release of the **Stentor2** model family — a ground-up redesign of the original Stentor v1 line. At ~12.3M parameters, it is a compact base language model built entirely from scratch on free-tier Kaggle compute using two NVIDIA Tesla T4 GPUs in a true dual-GPU distributed training configuration.
+
+Like all Stentor models, this is a **base next-token predictor**, not a chat assistant. It will not reliably follow instructions, has no safety tuning, and is best used for research, prototyping, speculative decoding, and edge-deployment experimentation. The value of this model is not its conversational capability — it is what it represents architecturally: dramatically improved efficiency over both Stentor v1 and the Stentor2-Preview, achieved by fixing the root vocabulary imbalance of v1 and removing architectural overhead that actively harmed training at this scale.
+
+---
+
+## The Core Design Insight: Vocabulary Efficiency
+
+The most consequential change in Stentor2 is the replacement of the standard Llama/Mistral 32,768-token vocabulary with a purpose-built **8,000-token English vocabulary** from the TokenMonster project (`english-8000-strict-nocapcode-v1`, padded to 8,064 for hardware alignment).
+
+This is not a minor tweak — it is the entire architectural story of Stentor2.
+
+### Why Vocabulary Size Matters So Much at This Scale
+
+In a transformer language model, the embedding table has shape `[vocab_size × hidden_size]`. When you tie word embeddings (share the embedding and output projection weights, which Stentor does), this table appears once in the parameter count. At 12M total parameters, the fraction consumed by this table dictates how much "brain" is left over for the actual transformer layers.
+
+**Stentor-12M (v1)** used a 32,768-token vocabulary. At a hidden size of 192:
+
+```
+embedding_params = 32,768 × 192 = 6,291,456
+total_params     = 12,047,040
+embedding_share  = 52.2%
+```
+
+Over half of the model was a lookup table. The transformer stack — the part that actually *learns language patterns* — had fewer than 6 million parameters to work with. It was more dictionary than reasoner.
+
+**Stentor2-12M** uses an 8,064-token vocabulary. At a hidden size of 256:
+
+```
+embedding_params = 8,064 × 256 = 2,064,384
+total_params     = 12,294,400
+embedding_share  = 16.8%
+```
+
+By shrinking the vocabulary, the embedding table was cut from 6.3M to 2.1M parameters — freeing up ~4.2M parameters that were redistributed into transformer depth (12 layers vs 9) and width (hidden size 256 vs 192), where they contribute directly to language modeling quality.
+
+The combined result of vocabulary efficiency, full MHA (see next section), a richer dataset pipeline, and true dual-GPU training is a **~70.8% reduction in perplexity** versus Stentor v1 (89.01 → 25.96).
+
+---
+
+## Why Full MHA Instead of GQA
+
+Stentor2-12M uses standard **full multi-head attention** (MHA) where the number of key/value heads equals the number of query heads (both = 4). The earlier Stentor2-Preview experimented with Grouped Query Attention (GQA), but it was removed for this release. This decision deserves a detailed explanation because it runs counter to the trend in larger models.
+
+### The GQA Problem at Small Scale
+
+Grouped Query Attention was designed to reduce the KV cache memory footprint in billion-parameter models where KV cache size becomes a serious inference bottleneck. At 12M parameters with a 1,024-token context, this problem simply does not exist. The KV cache per sequence per layer is:
+
+```
+2 (K and V) × 4 heads × 64 head_dim × 2 bytes (FP16) = 1,024 bytes per layer
+× 12 layers = 12,288 bytes ≈ 12 KB per sequence
+```
+
+GQA saves nothing meaningful here. But it costs something real during training.
+
+### How GQA Introduces Optimizer Noise
+
+With only 4 attention heads total, the valid GQA configurations are 1, 2, or 4 KV heads. Using fewer than 4 (say, 2) means each KV projection must simultaneously serve 2 query heads.
+
+During backpropagation, the gradients from those 2 query heads are **accumulated into a single KV projection update**. The optimizer (AdamW) then computes a single step direction for a gradient that is actually the sum of two independent signals — signals that may be pointing in meaningfully different directions depending on what each query head has learned to attend to. At scale, these signals tend to be well-correlated (which is part of why GQA works in large models). At 4 heads in a 12M parameter model, that correlation assumption breaks down. The optimizer is forced to find a compromise update that fully satisfies neither head, which introduces noise into the gradient estimates, destabilizes the loss curve, and slows convergence.
+
+This effect was directly observed in preliminary training runs for the Stentor2 family. GQA produced a visibly noisier training loss curve and slower convergence toward low perplexity — the exact opposite of its intended benefit.
+
+### The Throughput Cost
+
+Beyond gradient noise, GQA at 4 heads adds computational overhead with no offsetting gain. The head-sharing logic creates synchronization cost in the attention kernel that measurably reduced tokens-per-second throughput on T4 GPUs compared to clean full MHA. Every efficiency argument for GQA — memory, throughput, inference speed — inverts at this parameter scale.
+
+**The conclusion is straightforward:** GQA is a tool designed for large models with large KV caches. Applying it to a 12M parameter model with 4 heads actively harms training without providing any of its intended benefits. Full MHA is the correct choice here.
+
+---
+
+## Head-to-Head
+
+<a name="head-to-head"></a>
+
+| Property | Stentor-12M (v1) | Stentor2-12M-Preview | **Stentor2-12M** |
+|---|---|---|---|
+| **Vocabulary** | 32,768 (Mistral BPE) | 8,064 (TokenMonster) | **8,064 (TokenMonster)** |
+| **Hidden Size** | 192 | 256 | **256** |
+| **Intermediate Size** | 576 | 768 | **768** |
+| **Num Layers** | 9 | 12 | **12** |
+| **Attention Heads** | 3 | 4 | **4** |
+| **KV Heads (GQA)** | 3 (full MHA) | 4 (full MHA) | **4 (full MHA)** |
+| **Head Dimension** | 64 | 64 | **64** |
+| **Context Length** | 512 tokens | 1,024 tokens | **1,024 tokens** |
+| **Total Parameters** | 12,047,040 | 12,294,400 | **12,294,400** |
+| **Embedding Share** | 52.2% | 16.8% | **16.8%** |
+| **Non-Embedding Params** | ~5.76M | ~10.23M | **~10.23M** |
+| **Token Budget** | 200M | 240M | **480M (2 epochs)** |
+| **Training Processes** | 1 | 1 (2 GPUs available) | **2 (true distributed)** |
+| **Training Time** | ~1.3h | ~4.4h | **~2.13h** |
+| **Final Perplexity** | 89.01 | ~50.07 | **25.96** |
+| **Perplexity vs v1** | — | ↓43.8% | **↓70.8%** |
+| **Dataset** | FineWeb-Edu + Cosmopedia | FineWeb-Edu | **StenCore + FineWeb-HQ** |
+| **Architecture** | LlamaForCausalLM | LlamaForCausalLM | **LlamaForCausalLM** |
+| **Training Precision** | fp16 | fp16 | **fp16** |
+| **AutoModel compatible** | ✅ | ❌ (custom loader) | **✅** |
+
+---
+
+## 🚀 Quick Start
+
+### 1. Install Dependencies
+
+```bash
+pip install transformers torch safetensors tokenmonster
+```
+
+### 2. Load the Model
+
+Stentor2-12M loads cleanly with the standard HuggingFace API — no custom loader required.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model     = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M")
+tokenizer = AutoTokenizer.from_pretrained("StentorLabs/Stentor2-12M", trust_remote_code=True)
+```
+
+**GPU (FP16) — recommended if you have a CUDA GPU:**
+
+```python
+model = AutoModelForCausalLM.from_pretrained(
+    "StentorLabs/Stentor2-12M",
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+```
+
+### 3. Generate Text
+
+This is a **base model** — it continues text rather than answering questions. Give it the beginning of a sentence and it will complete it.
+
+```python
+input_ids      = torch.tensor([tokenizer.encode("The history of computing")], dtype=torch.long).to(next(model.parameters()).device)
+attention_mask = torch.ones_like(input_ids)
+
+with torch.inference_mode():
+    output = model.generate(
+        input_ids,
+        attention_mask=attention_mask,
+        max_new_tokens=80,
+        do_sample=True,
+        temperature=1.1,
+        top_p=0.55,
+        repetition_penalty=1.15,
+        pad_token_id=tokenizer.pad_token_id,
+    )
+
+print(tokenizer.decode(output[0].tolist()))
+```
+
+> **Why `attention_mask`?** The model's pad token and EOS token share the same ID. Without an explicit attention mask, HuggingFace throws a warning because it cannot tell which tokens are real vs padding. Passing `torch.ones_like(input_ids)` tells the model that every token in the input is real — which is always true for single-sequence inference.
+
+### 4. Recommended Generation Settings
+
+| Parameter | Recommended Range | Notes |
+|---|---|---|
+| `temperature` | 0.65 – 1.2 | Lower = more focused, higher = more creative |
+| `top_p` | 0.5 – 0.8 | Nucleus sampling cutoff |
+| `max_new_tokens` | 10 – 60 | Keep outputs short for best coherence |
+| `repetition_penalty` | 1.1 – 1.2 | Helps prevent looping |
+
+> ⚠️ **Keep `max_new_tokens` low.** This is a 12M parameter base model — coherence degrades beyond ~60 tokens. Short completions are significantly more on-topic than long ones.
+
+---
+
+## ⚠️ Important Limitations
+
+- **Not Instruction-Tuned:** This is a base model. It will often ignore prompts, continue in unexpected directions, or respond off-topic.
+- **No Safety Tuning:** No RLHF, no constitutional AI, no content filtering. Use with appropriate caution.
+- **Limited World Knowledge:** ~12M parameters cannot store meaningful world knowledge. Do not treat outputs as factual.
+- **Context Window:** Hard limit of 1,024 tokens. Longer contexts are untested and likely to degrade.
+- **English Only:** The TokenMonster `english-8000-strict-nocapcode-v1` vocabulary is English-specific. Non-English text will tokenize very poorly.
+- **Custom Tokenizer:** This model uses a TokenMonster adapter, not a standard HuggingFace fast tokenizer. The `tokenmonster` package must be installed. `trust_remote_code=True` is required when loading the tokenizer.
+- **`skip_special_tokens` Not Supported:** The TokenMonster tokenizer does **not** support the `skip_special_tokens` argument in its decode method. Always decode without it and strip special tokens manually if needed.
+- **Shared Tensor Warning:** When saving or loading, you may see: `Removed shared tensor {'lm_head.weight'} while saving`. This is expected behavior from tied word embeddings and is safe to ignore.
+
+---
+
+## Model Architecture — Full Specification
+
+Stentor2-12M is a `LlamaForCausalLM` model. All architecture values below were derived directly from the training script and validated against the logged parameter counts.
+
+### Core Configuration
+
+| Component | Value | Derivation |
+|---|---|---|
+| **Architecture** | `LlamaForCausalLM` | Hard-coded in training script |
+| **Hidden Size** | 256 | Inferred: embedding_params (2,064,384) ÷ vocab_size (8,064) = 256 ✓ |
+| **Intermediate Size (FFN)** | 768 | Hidden × 3 (verified via total param count) |
+| **Num Hidden Layers** | 12 | Verified via total param count formula |
+| **Num Attention Heads** | 4 | Hidden ÷ head_dim = 256 ÷ 64 = 4 |
+| **Num Key/Value Heads** | 4 | Full MHA — no GQA (see [above](#why-full-mha-instead-of-gqa)) |
+| **Head Dimension** | 64 | Enforced by training script hardware constraint |
+| **Vocab Size** | 8,064 | TokenMonster 8K base + 62 padding tokens (padded to multiple of 128) |
+| **Max Position Embeddings** | 1,024 | `block_size` in training configuration |
+| **Hidden Activation** | SiLU | LlamaForCausalLM default |
+| **Positional Encoding** | RoPE | `rope_theta = 10,000.0` |
+| **RMS Norm Epsilon** | 1e-5 | Training script default |
+| **Tie Word Embeddings** | True | Shared embedding / LM head weights |
+| **Attention Implementation** | SDPA | PyTorch Scaled Dot Product Attention |
+| **Attention Pattern** | Full causal | No sliding window, no sparse patterns |
+
+### Parameter Count Breakdown
+
+The total parameter count can be reproduced exactly using the following formula from the training script:
+
+```python
+def estimate_llama_params(vocab_size, hidden_size, intermediate_size,
+                          num_hidden_layers, num_attention_heads, num_key_value_heads):
+    kv_dim = int(hidden_size * num_key_value_heads / num_attention_heads)
+    attn   = 2 * hidden_size * hidden_size + 2 * hidden_size * kv_dim
+    mlp    = 3 * hidden_size * intermediate_size
+    norm   = 2 * hidden_size
+    total  = vocab_size * hidden_size + num_hidden_layers * (attn + mlp + norm) + hidden_size
+    return total
+```
+
+Plugging in Stentor2-12M values:
+
+```
+kv_dim    = 256 * 4 / 4 = 256
+attn      = 2×256×256 + 2×256×256 = 262,144
+mlp       = 3×256×768  = 589,824
+norm      = 2×256      = 512
+per_layer = 852,480
+
+embedding  = 8,064 × 256  = 2,064,384
+layers     = 12 × 852,480 = 10,229,760
+final_norm = 256
+
+total = 2,064,384 + 10,229,760 + 256 = 12,294,400 ✓
+```
+
+| Component | Parameters | % of Total |
+|---|---|---|
+| Embedding Table (tied with LM Head) | 2,064,384 | 16.8% |
+| Transformer Layers × 12 | 10,229,760 | 83.2% |
+| — Attention (per layer × 12) | 3,145,728 | 25.6% |
+| — FFN/MLP (per layer × 12) | 7,077,888 | 57.5% |
+| — Layer Norms (per layer × 12) | 6,144 | 0.05% |
+| Final RMS Norm | 256 | 0.002% |
+| **Total** | **12,294,400** | **100%** |
+
+---
+
+## The Tokenizer: TokenMonster
+
+Stentor2-12M uses a custom tokenizer adapter wrapping the **TokenMonster** `english-8000-strict-nocapcode-v1` vocabulary rather than a standard BPE tokenizer from the HuggingFace ecosystem.
+
+### What Is TokenMonster?
+
+TokenMonster ([alasdairforsythe/tokenmonster](https://huggingface.co/alasdairforsythe/tokenmonster)) is an alternative tokenization approach optimized for compact English vocabulary sizes. The `english-8000-strict-nocapcode-v1` vocabulary is a purpose-built ~8,000-token English vocabulary designed for efficiency at small model scales.
+
+### ⚠️ `skip_special_tokens` Is Not Supported
+
+The TokenMonster tokenizer **does not support the `skip_special_tokens` argument** in its decode method. Always decode without it:
+
+```python
+# ✅ Correct
+text = tokenizer.decode(output_ids)
+
+# ❌ This will raise an error
+text = tokenizer.decode(output_ids, skip_special_tokens=True)
+```
+
+If you are using HuggingFace's `TextIteratorStreamer` or any wrapper that internally passes `skip_special_tokens=True`, you will need to patch or replace that wrapper.
+
+### Tokenizer Efficiency vs. v1
+
+You may notice this tokenizer produces **more tokens per word** compared to Stentor v1. This is expected and by design. The v1 models used a 32,768-token Mistral BPE vocabulary, which encodes common English words as single tokens very efficiently. Stentor2 uses an 8,064-token TokenMonster vocabulary — smaller vocabulary means more tokens per word on average. This is the direct tradeoff for freeing up ~4.2M parameters for the transformer layers. The ~70.8% perplexity improvement shows the tradeoff was worth it.
+
+### Vocabulary Construction
+
+1. Base vocabulary loaded from `alasdairforsythe/tokenmonster` → `vocabs/english-8000-strict-nocapcode-v1.vocab`
+2. Special tokens added: `</s>` (EOS), `<s>` (BOS), `<pad>` (set equal to EOS)
+3. A default chat template is injected for structural compatibility
+4. Vocabulary padded to the nearest multiple of 128 using dummy tokens `<|extra_0|>` through `<|extra_61|>`, resulting in a final vocabulary size of **8,064 tokens**
+
+### Tokenizer Configuration
 
 ```json
 {
-  "hidden_size": 256,
-  "intermediate_size": 768,
-  "learning_rate": 0.0008,
-  "max_position_embeddings": 1024,
-  "mixed_precision": "fp16",
-  "model_arch": "llama",
-  "num_attention_heads": 4,
-  "num_hidden_layers": 12,
-  "num_key_value_heads": 4,
-  "optimizer": "adamw",
-  "pad_vocab_to_multiple": 128,
-  "rope_theta": null,
-  "scheduler": "cosine",
-  "stable_ratio": 0.8,
-  "torch_compile": false,
-  "vocab_size": 8064,
-  "warmup_ratio": 0.05,
-  "weight_decay": 0.01
+  "tokenizer_type": "tokenmonster",
+  "vocab_file": "tokenmonster.vocab",
+  "model_max_length": 1024,
+  "eos_token": "</s>",
+  "bos_token": "<s>",
+  "pad_token": "</s>",
+  "vocab_size": 8064
 }
 ```
+
+### Chat Template
+
+A simple chat template is injected for structural compatibility with chat formatting tools. The base model is **not trained to follow it**:
+
+```jinja
+{% for message in messages %}
+<|{{ message['role'] }}|>
+{{ message['content'] }}
+{% endfor %}
+{% if add_generation_prompt %}<|assistant|>
+{% endif %}
+```
+
+### Loading the Tokenizer
+
+```bash
+pip install tokenmonster
+```
+
+```python
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("StentorLabs/Stentor2-12M", trust_remote_code=True)
+```
+
+---
+
+## Training Infrastructure
+
+### Hardware
+
+| Component | Specification |
+|---|---|
+| GPU Count | 2× NVIDIA Tesla T4 |
+| VRAM per GPU | 15.64 GB |
+| Total VRAM | ~31.3 GB |
+| Training Processes | 2 (true data-parallel distributed training) |
+| Platform | Kaggle Notebooks (free tier) |
+| Accelerator Library | HuggingFace Accelerate |
+| Communication Backend | NCCL |
+
+> **Note on Dual-GPU Setup:** Unlike Stentor2-12M-Preview (which had 2 GPUs available but ran as a single process), Stentor2-12M ran as a true 2-process distributed job with `world_size=2`. Both GPUs contributed to training, which is why total_batch_size=128 with per_device_batch_size=32 and gradient_accumulation=2 (32 × 2 processes × 2 = 128).
+
+### Software Stack
+
+| Package | Role |
+|---|---|
+| PyTorch | Core tensor operations and autograd |
+| HuggingFace Transformers | Model architecture (LlamaForCausalLM) |
+| HuggingFace Accelerate | Distributed training loop and device management |
+| HuggingFace Datasets | Streaming data loading |
+| bitsandbytes | Quantization primitives |
+| tokenmonster | Custom vocabulary |
+| safetensors | Model serialization |
+
+---
+
+## Training Hyperparameters — Complete Reference
+
+### Core Training Parameters
+
+| Hyperparameter | Value | Notes |
+|---|---|---|
+| `learning_rate` | 8e-4 | AdamW LR |
+| `weight_decay` | 0.01 | Applied to non-embedding, non-norm, non-bias params |
+| `max_grad_norm` | 1.0 | Gradient clipping threshold |
+| `optimizer` | AdamW | With `betas=(0.9, 0.95)`, `eps=1e-8` |
+| `scheduler` | Cosine | Cosine decay with linear warmup |
+| `warmup_steps` | 477 | ~5% of max_train_steps |
+| `stable_steps` | 7,641 | ~80% of max_train_steps |
+| `source_token_budget` | 240,000,000 | Raw source tokens per epoch |
+| `token_budget` | 480,000,000 | Total tokens across all epochs (2 epochs × 240M) |
+| `max_train_steps` | 9,552 | Theoretical max; training stopped early at token budget |
+| `actual_steps_trained` | ~3,600 | Token budget reached at step ~3,600 in epoch 1 |
+| `num_train_epochs` | 2 | |
+| `seed` | 42 | Reproducibility seed |
+| `mixed_precision` | fp16 | All activations/gradients in FP16 |
+
+### Batch & Sequence Parameters
+
+| Hyperparameter | Value | Notes |
+|---|---|---|
+| `per_device_train_batch_size` | 32 | Per GPU per gradient accumulation step |
+| `per_device_eval_batch_size` | 16 | Evaluation batch size |
+| `gradient_accumulation_steps` | 2 | Optimizer step every 2 forward passes |
+| `num_processes` | 2 | True distributed, 2 GPUs |
+| `total_batch_size` | 128 | `32 × 2 processes × 2 grad_accum` |
+| `block_size` | 1,024 | Sequence length; training packed to this size |
+| `tokens_per_optimizer_step` | 131,072 | `total_batch_size × block_size` |
+
+### Evaluation & Checkpointing
+
+| Hyperparameter | Value |
+|---|---|
+| `eval_steps` | 900 |
+| `best_eval_steps` | 900 |
+| `best_eval_start_step` | 1,500 |
+| `save_every_minutes` | 30 |
+| `save_total_limit` | 2 |
+| `logging_steps` | 300 |
+| `max_eval_samples` | 5,000 |
+
+### Precision Stability Configuration
+
+| Setting | Value |
+|---|---|
+| `fp32_norms` | True |
+| `fp32_norm_modules` | 25 |
+| `fp32_softmax_modules` | 0 (SDPA-compatible path) |
+| `fp32_critical_layers` | 2 (first 1 + last 1) |
+| `torch_compile` | False |
+| `gradient_checkpointing` | False |
+
+### AdamW Optimizer — Detailed
+
+The optimizer uses a decoupled parameter group strategy:
+
+- **Decay group:** All `nn.Linear` weight matrices (excludes bias, norm weights, embedding)
+  - `weight_decay = 0.01`
+- **No-decay group:** Bias terms, normalization parameters, embedding parameters
+  - `weight_decay = 0.0`
+- **Betas:** `(0.9, 0.95)` — the 0.95 β₂ is a modern LLM default (vs the 0.999 PyTorch default)
+- **Epsilon:** `1e-8`
+- **Fused kernel:** Enabled when CUDA is available (`fused=True`)
+
+### Learning Rate Schedule
+
+```
+Phase 1 — Warmup (steps 0–477):
+  LR ramps linearly from 0 → 8e-4
+
+Phase 2 — Cosine Decay (steps 477–9,552):
+  LR follows cosine curve from 8e-4 → 0
+  (Training stopped early at ~3,600 steps due to token budget)
+```
+
+---
+
+## The T4 Mixed-Precision Recipe — Deep Dive
+
+The most technically interesting aspect of Stentor2's training pipeline is its custom **T4 Mixed-Precision Recipe** — a bespoke approach to stable FP16 training on NVIDIA Tesla T4 GPUs, which lack BF16 support and have known numerical instability issues with FP16 on certain operations.
+
+This recipe involves three distinct techniques applied simultaneously:
+
+### 1. FP32 Critical Layers (2 layers: first 1 + last 1)
+
+The first and last transformer layers run entirely in FP32:
+
+- Their weights are cast to `.float()` at setup time
+- Their `forward()` method is monkey-patched to cast all inputs to FP32 before the call and cast outputs back to the original dtype afterward
+- `torch.amp.autocast("cuda", enabled=False)` context prevents autocast from re-downcasting inside the layer
+
+**Rationale:** The first layer is responsible for embedding projection and initial feature extraction; instability here corrupts the entire forward pass. The last layer handles final token prediction; numerical errors here directly impact loss. Running these in FP32 provides a stability floor at minimal compute cost.
+
+### 2. FP32 Normalization Layers (25 modules)
+
+All RMSNorm and LayerNorm modules are monkey-patched to run their computation in FP32 regardless of input dtype:
+
+```python
+def _fp32_norm_forward(hidden_states, *args, **kwargs):
+    input_dtype = hidden_states.dtype
+    output = original_forward(hidden_states.float().contiguous(), *args, **kwargs)
+    return output.clone().to(input_dtype)
+```
+
+The `.clone()` call prevents returning graph-managed buffers that could be overwritten across replay steps. The `.contiguous()` call prevents strided-tensor issues in FP32 norm operations.
+
+**Why 25 modules:** 12 transformer layers × 2 norms each (input norm + post-attention norm) + 1 final norm = 25 total.
+
+**This is why `torch_compile` is disabled.** The FP32 norm wrappers are incompatible with CUDAGraph replay, which `torch.compile` uses under `reduce-overhead` mode.
+
+### 3. SDPA-Compatible Path (No FP32 Softmax Wrapper)
+
+Unlike earlier experiments, Stentor2-12M uses the SDPA (Scaled Dot Product Attention) path from PyTorch, which handles numerical stability in the attention kernel natively. The FP32 softmax monkey-patch used in some experimental runs was intentionally skipped here because SDPA and FlashAttention kernels require FP16/BF16 inputs for their fast paths. Wrapping the softmax in FP32 would disable these kernels and cause a significant throughput regression. SDPA's internal numerics are already stable enough for 1,024-token sequences on T4 hardware.
+
+### T4 Recipe Summary Table
+
+| Technique | Count | Scope |
+|---|---|---|
+| FP32 critical layers | 2 | First and last transformer layers |
+| FP32 norm modules | 25 | All RMSNorm / LayerNorm |
+| FP32 softmax modules | 0 | Skipped — SDPA handles this natively |
+
+---
+
+## Data Pipeline
+
+### Dataset
+
+Stentor2-12M was trained on a **cascaded StenCore → FineWeb-HQ pipeline**, a two-stage data mixture combining StentorLabs' curated source corpus with high-quality filtered web text. This represents an upgrade from the FineWeb-Edu-only data used in Stentor2-12M-Preview.
+
+**Total tokens processed:** 480,116,736 (2 epochs × 240M source tokens per epoch)
+
+### Cascaded Data Pipeline
+
+The training pipeline loads data in priority order:
+
+1. **StenCore** ([StentorLabs/StenCore](https://huggingface.co/datasets/StentorLabs/StenCore)) — StentorLabs' curated training corpus, loaded first up to the source token budget
+2. **FineWeb-HQ** ([epfml/FineWeb-HQ](https://huggingface.co/datasets/epfml/FineWeb-HQ)) — High-quality filtered web text, used to fill the remaining token budget after StenCore is exhausted. FineWeb-HQ documents are deduplicated against StenCore by content fingerprint to prevent overlap
+
+The pipeline materializes the dataset on rank 0 and distributes it across both GPU processes via a shared disk cache, eliminating duplicated downloads in the distributed setting.
+
+### Dataset Scale
+
+| Component | Value |
+|---|---|
+| Materialized raw rows | 1,137,414 |
+| Source token budget per epoch | 240,000,000 |
+| Total token budget | 480,000,000 |
+| Actual tokens trained on | 480,116,736 |
+| Source download budget | 3 GB |
+
+### Text Preprocessing
+
+Each raw text sample undergoes the following cleaning pipeline before tokenization:
+
+```python
+def clean_text(text: str) -> str:
+    text = unicodedata.normalize("NFKC", text)   # normalize unicode
+    lines = [line.strip() for line in text.splitlines() if line.strip()]
+    text = " ".join(lines)                         # collapse newlines
+    text = " ".join(text.split())                  # normalize whitespace
+    return text
+```
+
+**NFKC normalization** maps visually equivalent Unicode characters to a single canonical form (e.g., full-width `Ａ` → `A`, ligature `fi` → `fi`). This is the standard approach used in T5, BERT, and most modern LLM preprocessing pipelines.
+
+### Tokenization
+
+- `add_special_tokens=False` during tokenization
+- EOS token (`</s>`) appended to every sample
+- Attention mask generated (all 1s for real tokens)
+
+### Sequence Packing
+
+After tokenization, samples are **packed** into fixed 1,024-token blocks:
+
+```
+Sample 1:  [tok, tok, ..., </s>]   (e.g., 347 tokens)
+Sample 2:  [tok, tok, ..., </s>]   (e.g., 891 tokens)
+                                    ↓
+Block 1:   [<sample1...>, <first 677 tokens of sample2>]   (1024 tokens)
+Block 2:   [<remaining 214 tokens of sample2>, <sample3...>]
+```
+
+Packing eliminates all padding waste and ensures every training token is a real content token. Labels for packed sequences are identical to `input_ids` — there is no special boundary masking between packed samples, which is standard practice.
+
+### Data Collation
+
+- `input_ids`: padded with `pad_token_id`
+- `labels`: padded with `-100` (ignored in loss computation)
+- `attention_mask`: padded with `0`
+- All batches padded to nearest multiple of 8 for hardware alignment
+
+---
+
+## Weight Initialization
+
+All parameters are initialized using a **truncated normal distribution** with `std=0.02` — the same initialization used in GPT-2 and most modern LLMs:
+
+```python
+def initialize_weights(model, std=0.02):
+    for module in model.modules():
+        if isinstance(module, (nn.Linear, nn.Embedding)):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif "rmsnorm" in type(module).__name__.lower():
+            if module.weight is not None:
+                module.weight.data.fill_(1.0)   # scale initialized to 1
+            if module.bias is not None:
+                module.bias.data.zero_()
+```
+
+- Linear layers: normal(0, 0.02)
+- Embeddings: normal(0, 0.02)
+- RMSNorm scale weights: initialized to 1.0 (identity transform at start)
+- All biases: zero
+
+---
+
+## Evaluation & Results
+
+### Metrics
+
+- **Validation Loss:** Cross-entropy loss over a held-out validation prefix drawn from the tokenized train set (5,000 samples)
+- **Perplexity (PPL):** `exp(loss)` — lower means the model is less "surprised" by unseen text
+
+### Training Loss Progression
+
+| Step | Loss (train batch) |
+|---|---|
+| 300 | 4.9760 |
+| 600 | 4.1367 |
+| 900 | 3.4398 |
+| 1200 | 3.6314 |
+| 1500 | 3.3766 |
+| 1800 | 3.3155 |
+| 2100 | 3.2452 |
+| 2400 | 3.3610 |
+| 2700 | 3.2639 |
+| 3000 | 3.3475 |
+| 3300 | 3.0684 |
+| 3600 | 2.9352 |
+
+### Evaluation Results
+
+| Checkpoint | Step | Eval Loss | Perplexity |
+|---|---|---|---|
+| End of Epoch 0 | ~2,400 | 3.3776 | 29.30 |
+| Best Checkpoint | 3,300 | 3.2814 | ~26.62 |
+| **Final Model** | **~3,600** | **3.2565** | **25.96** |
+
+### Comparison to Previous Stentor Models
+
+| Model | Final Eval Loss | Final Perplexity | vs v1 |
+|---|---|---|---|
+| Stentor-12M (v1) | 4.4887 | 89.01 | — |
+| Stentor2-12M-Preview | ~3.91 | ~50.07 | ↓43.8% |
+| **Stentor2-12M** | **3.2565** | **25.96** | **↓70.8%** |
+
+> **Comparison note:** These three models were trained on different datasets and with different configurations. The comparison is meaningful directionally but is not a rigorously controlled ablation. Stentor v1 used FineWeb-Edu + Cosmopedia; Stentor2-Preview used FineWeb-Edu only; Stentor2-12M used StenCore + FineWeb-HQ.
+
+---
+
+## BLiMP Benchmark
+
+BLiMP (Benchmark of Linguistic Minimal Pairs) evaluates a language model's grammatical knowledge by presenting pairs of sentences — one grammatical and one ungrammatical — and measuring how often the model assigns higher probability to the grammatical sentence. Random chance is 50%.
+
+**Overall BLiMP Average (67 tasks): 68.95%**  
+**Random Baseline: 50.00%**  
+**Above Chance: +18.95 percentage points**
+
+### Full BLiMP Results
+
+| Task | Accuracy |
+|---|---|
+| **Overall Average** | **68.95%** |
+| blimp_adjunct_island | 72.00% |
+| blimp_anaphor_gender_agreement | 65.70% |
+| blimp_anaphor_number_agreement | 95.70% |
+| blimp_animate_subject_passive | 68.70% |
+| blimp_animate_subject_trans | 82.80% |
+| blimp_causative | 60.80% |
+| blimp_complex_NP_island | 41.00% |
+| blimp_coordinate_structure_constraint_complex_left_branch | 40.20% |
+| blimp_coordinate_structure_constraint_object_extraction | 75.00% |
+| blimp_determiner_noun_agreement_1 | 97.40% |
+| blimp_determiner_noun_agreement_2 | 92.50% |
+| blimp_determiner_noun_agreement_irregular_1 | 75.50% |
+| blimp_determiner_noun_agreement_irregular_2 | 90.00% |
+| blimp_determiner_noun_agreement_with_adj_2 | 85.40% |
+| blimp_determiner_noun_agreement_with_adj_irregular_1 | 78.60% |
+| blimp_determiner_noun_agreement_with_adj_irregular_2 | 81.40% |
+| blimp_determiner_noun_agreement_with_adjective_1 | 89.60% |
+| blimp_distractor_agreement_relational_noun | 70.90% |
+| blimp_distractor_agreement_relative_clause | 42.50% |
+| blimp_drop_argument | 70.20% |
+| blimp_ellipsis_n_bar_1 | 67.30% |
+| blimp_ellipsis_n_bar_2 | 79.70% |
+| blimp_existential_there_object_raising | 79.70% |
+| blimp_existential_there_quantifiers_1 | 95.80% |
+| blimp_existential_there_quantifiers_2 | 29.60% |
+| blimp_existential_there_subject_raising | 72.80% |
+| blimp_expletive_it_object_raising | 73.30% |
+| blimp_inchoative | 49.50% |
+| blimp_intransitive | 60.60% |
+| blimp_irregular_past_participle_adjectives | 99.70% |
+| blimp_irregular_past_participle_verbs | 78.50% |
+| blimp_irregular_plural_subject_verb_agreement_1 | 77.40% |
+| blimp_irregular_plural_subject_verb_agreement_2 | 87.40% |
+| blimp_left_branch_island_echo_question | 40.00% |
+| blimp_left_branch_island_simple_question | 38.80% |
+| blimp_matrix_question_npi_licensor_present | 16.30% |
+| blimp_npi_present_1 | 65.50% |
+| blimp_npi_present_2 | 63.80% |
+| blimp_only_npi_licensor_present | 52.90% |
+| blimp_only_npi_scope | 41.60% |
+| blimp_passive_1 | 86.10% |
+| blimp_passive_2 | 85.80% |
+| blimp_principle_A_c_command | 65.60% |
+| blimp_principle_A_case_1 | 100.00% |
+| blimp_principle_A_case_2 | 85.90% |
+| blimp_principle_A_domain_1 | 88.90% |
+| blimp_principle_A_domain_2 | 64.10% |
+| blimp_principle_A_domain_3 | 50.50% |
+| blimp_principle_A_reconstruction | 40.20% |
+| blimp_regular_plural_subject_verb_agreement_1 | 87.30% |
+| blimp_regular_plural_subject_verb_agreement_2 | 79.70% |
+| blimp_sentential_negation_npi_licensor_present | 99.20% |
+| blimp_sentential_negation_npi_scope | 46.40% |
+| blimp_sentential_subject_island | 39.30% |
+| blimp_superlative_quantifiers_1 | 48.00% |
+| blimp_superlative_quantifiers_2 | 75.60% |
+| blimp_tough_vs_raising_1 | 34.00% |
+| blimp_tough_vs_raising_2 | 83.10% |
+| blimp_transitive | 76.80% |
+| blimp_wh_island | 63.40% |
+| blimp_wh_questions_object_gap | 59.10% |
+| blimp_wh_questions_subject_gap | 87.60% |
+| blimp_wh_questions_subject_gap_long_distance | 92.40% |
+| blimp_wh_vs_that_no_gap | 92.10% |
+| blimp_wh_vs_that_no_gap_long_distance | 97.50% |
+| blimp_wh_vs_that_with_gap | 33.80% |
+| blimp_wh_vs_that_with_gap_long_distance | 11.00% |
+
+### BLiMP Interpretation
+
+**Strong performance (>85%):** Determiner-noun agreement, passive constructions, anaphor number agreement, irregular past participles, existential quantifiers, sentential negation licensing, wh-movement without gaps, Principle A case conditions. These tasks test local agreement and well-practiced syntactic patterns — areas where token-level statistics are strong predictors.
+
+**Moderate performance (60–85%):** Most agreement tasks, animate subject constructions, ellipsis, raising constructions, WH questions with subject gaps. The model has learned these patterns but applies them inconsistently across surface variations.
+
+**Below-chance or near-chance performance (<50%):** NPI licensing (particularly `matrix_question_npi_licensor_present` at 16.3%, `wh_vs_that_with_gap_long_distance` at 11%), island constraints, some scope tasks. These require tracking long-distance syntactic dependencies or operator scope — phenomena that demand more representational capacity than 12M parameters can reliably support at 1,024-token context.
+
+**Key takeaway:** At 68.95% vs 50% random, this model demonstrates genuine grammatical knowledge well above chance at only 12.3M parameters. The profile of strengths (local agreement) and weaknesses (long-distance dependencies, operator scope) is consistent with what scaling laws predict for a model of this size.
+
+---
+
+## Training Dynamics
+
+The training run proceeded for 2 epochs, stopping when the 480M token budget was reached at ~step 3,600 of a theoretical 9,552-step schedule.
+
+**Epoch 0 (steps 0–~2,400, 3,497 seconds):** Loss dropped rapidly from ~8+ initially, stabilizing around 3.2–3.6 by step 2,100. The first best checkpoint was recorded at step 1,500 (eval loss 3.5410), with a better checkpoint at step 2,400 (eval loss 3.3743) marking the end of epoch 0.
+
+**Epoch 1 (steps ~2,400–3,600, 1,906 seconds):** Training continued with a fresh data pass. The best checkpoint was achieved at step 3,300 (eval loss 3.2814). Notably, the final epoch evaluation after the token budget was reached produced 3.2565 (ppl 25.96) — slightly better than the best saved checkpoint eval, consistent with the cosine schedule still having meaningful learning rate at that point.
+
+**Throughput:** ~91,500–92,000 tokens/second average across both GPUs throughout training — a substantial improvement from the single-process ~91,600 tps of the Preview, now representing genuine dual-GPU utilization.
+
+---
+
+## Use Cases & Intended Uses
+
+> 🔬 **Reminder:** This is a **research artifact**. It is a base language model with no safety tuning, no instruction following, and no factual grounding.
+
+### Intended Uses
+
+| Use Case | Suitability | Notes |
+|---|---|---|
+| Studying transformer training dynamics | ✅ High | Small enough to train/fine-tune on free compute |
+| Tokenization efficiency research | ✅ High | 8K vs 32K vocab tradeoff is directly observable |
+| GQA vs MHA ablation research | ✅ High | Contrast directly with Stentor2-Preview |
+| Speculative decoding experiments | ✅ High | Fast enough to serve as a draft model |
+| Benchmarking CPU/edge inference latency | ✅ High | ~25 MB in FP16, runs on any hardware |
+| Testing quantization/conversion pipelines | ✅ High | GGUF, ONNX, INT8 pipeline validation |
+| Teaching material for LLM courses | ✅ High | Architecture is simple enough to trace by hand |
+| LoRA / QLoRA fine-tuning experiments | ✅ Moderate | Base model only; start from scratch for any task |
+| Text continuation / creative prompting | ✅ Moderate | Works best on short completions ≤60 tokens |
+| Domain-specific fine-tuning research | ✅ Moderate | Small enough to iterate rapidly |
+| Factual Q&A | ❌ Not suitable | Model has no reliable world knowledge |
+| Production deployment | ❌ Not suitable | No safety tuning; base model only |
+| Non-English text | ❌ Not suitable | TokenMonster vocab is English-only |
+| Long coherent outputs (>60 tokens) | ❌ Not suitable | Coherence degrades quickly at this scale |
+
+---
+
+## Out-of-Scope Uses
+
+- **User-facing applications of any kind** — This model has no safety filtering, no alignment, and no factual reliability.
+- **Medical, legal, or financial advice** — 12M parameters cannot store or reason over specialized knowledge reliably.
+- **Generating content about real people** — The model has no awareness of real people. Outputs mentioning real people are likely fabricated.
+- **Automated content pipelines** — Do not use this model to generate content at scale without human review.
+- **Non-English use** — The 8,064-token TokenMonster vocabulary is built exclusively for English.
+- **Instruction following** — This is a base model. It does not reliably follow instructions, answer questions, or complete structured tasks.
+
+---
+
+## Ethical Considerations & Societal Impact
+
+### Inherited Data Biases
+
+Stentor2-12M was trained on StenCore and FineWeb-HQ. Despite quality filtering, this data inherits biases present in English-language web text:
+
+- **Western-centric perspective** — Educational content on the web skews heavily toward Western viewpoints and examples
+- **English monolingualism** — The training data and vocabulary are both English-only
+- **Demographic underrepresentation** — Groups underrepresented in English-language web text will be underrepresented in outputs
+- **Temporal cutoff** — The model has no knowledge of events after the data collection cutoff
+
+### No Safety Tuning
+
+This model has received **no safety training of any kind**. It is a raw base model that predicts the next token based on statistical patterns. It should not be used in any context where harmful outputs would cause real-world harm.
+
+### Positive Societal Aspects
+
+- **Democratizing AI research** — Trained entirely on free-tier Kaggle compute, this model demonstrates that meaningful LLM research does not require significant financial resources.
+- **Transparency** — Full training hyperparameters, architecture details, training script, and benchmark results are published. This is a contribution to reproducible ML research.
+- **Minimal environmental footprint** — ~2.13 hours of dual-GPU compute. Estimated carbon footprint well under 0.5 kg CO₂e.
+
+---
+
+## Inference Guide
+
+### Basic Generation
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model     = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("StentorLabs/Stentor2-12M", trust_remote_code=True)
+
+device = next(model.parameters()).device
+
+def generate(prompt, max_new_tokens=50, temperature=0.9, top_p=0.65):
+    input_ids      = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long).to(device)
+    attention_mask = torch.ones_like(input_ids)
+    with torch.inference_mode():
+        output = model.generate(
+            input_ids,
+            attention_mask=attention_mask,
+            max_new_tokens=max_new_tokens,
+            do_sample=True,
+            temperature=temperature,
+            top_p=top_p,
+            repetition_penalty=1.15,
+            pad_token_id=tokenizer.pad_token_id,
+        )
+    new_ids = output[0][input_ids.shape[1]:].tolist()
+    return tokenizer.decode(new_ids).strip()
+
+print(generate("The history of computing began"))
+```
+
+### CPU (FP32)
+
+```python
+model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float32)
+model = model.to("cpu")
+```
+
+### GPU (FP16)
+
+```python
+model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16, device_map="auto")
+```
+
+### From a Local Checkpoint
+
+```python
+model     = AutoModelForCausalLM.from_pretrained("./path/to/local/checkpoint")
+tokenizer = AutoTokenizer.from_pretrained("./path/to/local/checkpoint", trust_remote_code=True)
+```
+
+---
+
+## Real Model Responses
+
+*[Placeholder — generation examples coming soon]*
+
+---
+
+## Quantization
+
+### FP16 — Recommended First Step
+
+```python
+model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16)
+model = model.to("cuda")
+```
+
+### Dynamic INT8 Quantization (CPU, PyTorch native)
+
+```python
+import torch
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float32)
+model = model.to("cpu").eval()
+
+model_int8 = torch.quantization.quantize_dynamic(
+    model,
+    {torch.nn.Linear},
+    dtype=torch.qint8,
+)
+# Approximate memory: ~12 MB — 75% reduction from FP32
+```
+
+### bitsandbytes INT8 (GPU)
+
+```python
+from transformers import AutoModelForCausalLM, BitsAndBytesConfig
+import torch
+
+bnb_config = BitsAndBytesConfig(load_in_8bit=True)
+model = AutoModelForCausalLM.from_pretrained(
+    "StentorLabs/Stentor2-12M",
+    quantization_config=bnb_config,
+    device_map="auto"
+)
+```
+
+---
+
+## Format Conversion
+
+### Convert to GGUF (for llama.cpp)
+
+```bash
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+pip install -r requirements.txt
+
+huggingface-cli download StentorLabs/Stentor2-12M --local-dir stentor2-12m
+
+python convert_hf_to_gguf.py stentor2-12m/ \
+  --outfile stentor2-12m.gguf \
+  --outtype f16
+
+./llama-quantize stentor2-12m.gguf stentor2-12m-q4_0.gguf q4_0
+./llama-cli -m stentor2-12m-q4_0.gguf -p "The science of" -n 50
+```
+
+> **Note on GGUF + TokenMonster:** The custom TokenMonster tokenizer may require manual vocabulary mapping when using llama.cpp. The standard `convert_hf_to_gguf.py` script expects a HuggingFace tokenizer format and may need adjustment for the TokenMonster vocabulary.
+
+### Convert to ONNX
+
+```bash
+pip install optimum[exporters]
+
+optimum-cli export onnx \
+  --model StentorLabs/Stentor2-12M \
+  --task text-generation-with-past \
+  stentor2-12m-onnx/
+```
+
+---
+
+## Speculative Decoding
+
+Stentor2-12M can serve as a fast **draft model** to accelerate inference from larger Llama-family target models.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+draft_model = AutoModelForCausalLM.from_pretrained(
+    "StentorLabs/Stentor2-12M",
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+
+target_model = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-3.2-1B",
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+target_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
+
+prompt  = "Explain the concept of recursion"
+inputs  = target_tokenizer(prompt, return_tensors="pt")
+
+outputs = target_model.generate(
+    **inputs,
+    assistant_model=draft_model,
+    do_sample=True,
+    max_new_tokens=100
+)
+print(target_tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+**Important caveat:** Stentor2 uses a different vocabulary (8,064-token TokenMonster) than standard Llama models (32,000-token BPE). This vocabulary mismatch means the target model's token acceptance rate may be lower than it would be with a vocabulary-compatible draft model. Speedups depend heavily on the similarity of the generated distribution between draft and target.
+
+---
+
+## Bias, Risks & Limitations
+
+- **Prompt Relevance:** Outputs are frequently off-topic for complex prompts. The model is pattern-completing, not comprehending.
+- **Factual Accuracy:** All factual claims should be treated as unreliable. 12M parameters cannot store meaningful world knowledge.
+- **Context Boundary:** Hard limit of 1,024 tokens.
+- **Short Output Window for Coherence:** Even within the context limit, coherence degrades beyond ~60 tokens. `max_new_tokens` at 60 or below is strongly recommended.
+- **English Bias:** TokenMonster English vocabulary — other languages will produce poor output.
+- **Training Data Bias:** Inherits biases present in English-language web data.
+- **Hallucination:** Like all LLMs, this model may confidently produce plausible-sounding but entirely fabricated content.
+- **No Alignment:** No RLHF, no DPO, no constitutional training.
+- **Tokenizer Efficiency:** The 8K TokenMonster vocabulary produces more tokens per word than standard 32K BPE vocabularies. This is expected and is the direct architectural tradeoff for the perplexity improvement.
+- **Shared Tensor Warning:** `Removed shared tensor {'lm_head.weight'} while saving` is expected from tied word embeddings. Safe to ignore.
+
+---
+
+## Related Work
+
+### Comparable Sub-50M Models
+
+| Model | Parameters | Perplexity | Vocab | Notes |
+|---|---|---|---|---|
+| **Stentor2-12M** (this model) | 12.3M | 25.96 | 8,064 | Full MHA, StenCore+FineWeb-HQ, dual-GPU |
+| Stentor2-12M-Preview | 12.3M | ~50.07 | 8,064 | Single-process, FineWeb-Edu only |
+| Stentor-12M (v1) | 12.0M | 89.01 | 32,768 | Mistral BPE, FineWeb-Edu+Cosmopedia |
+| Stentor-30M (v1) | 30.4M | 33.02 | 32,768 | Larger v1 model |
+| Pythia-14M | 14M | varies (Pile) | 50,254 | EleutherAI; well-studied scaling baseline |
+| TinyStories-33M | ~33M | varies | ~50K | Synthetic story data; Eldan & Li, 2023 |
+
+> **Comparison caveats:** Perplexity numbers are not directly comparable across models — different validation sets, vocabularies, tokenizers, and training data all affect the number. The table is a rough orientation, not a rigorous benchmark.
+
+### Related Research Papers
+
+| Paper | Relevance |
+|---|---|
+| [TinyStories](https://arxiv.org/abs/2305.07759) — Eldan & Li, 2023 | Demonstrates meaningful language generation from sub-33M models |
+| [Pythia](https://arxiv.org/abs/2304.01373) — Biderman et al., 2023 | Systematic study of small model scaling |
+| [Scaling Laws](https://arxiv.org/abs/2001.08361) — Kaplan et al., 2020 | Foundational work on compute-optimal training |
+| [Chinchilla](https://arxiv.org/abs/2203.15556) — Hoffmann et al., 2022 | Revised scaling laws; informs token budget decisions |
+| [GQA](https://arxiv.org/abs/2305.13245) — Ainslie et al., 2023 | Grouped Query Attention — and why it doesn't help here |
+| [RoPE](https://arxiv.org/abs/2104.09864) — Su et al., 2021 | Positional encoding used in this model |
+| [Speculative Decoding](https://arxiv.org/abs/2211.17192) — Leviathan et al., 2023 | Primary deployment use case for this model |
+| [Model Cards](https://arxiv.org/abs/1810.03993) — Mitchell et al., 2018 | Methodology underlying this model card |
+| [T5](https://arxiv.org/abs/1910.10683) — Raffel et al., 2020 | Source of NFKC text normalization approach |
+
+### StentorLabs Models
+
+- [Stentor-30M](https://huggingface.co/StentorLabs/Stentor-30M) — Larger v1 base model
+- [Stentor-12M](https://huggingface.co/StentorLabs/Stentor-12M) — v1 baseline this model improves upon
+- [Stentor-30M-Instruct](https://huggingface.co/StentorLabs/Stentor-30M-Instruct) — Instruction-tuned v1
+- [Stentor-12M-Instruct](https://huggingface.co/StentorLabs/Stentor-12M-Instruct) — Instruction-tuned v1
+- [Stentor2-12M-Preview](https://huggingface.co/StentorLabs/Stentor2-12M-Preview) — Preview checkpoint preceding this release
+- [StentorLabs Collection](https://huggingface.co/StentorLabs) — All models from StentorLabs
+
+### Referenced Tools & Datasets
+
+- [StenCore](https://huggingface.co/datasets/StentorLabs/StenCore) — Primary training data
+- [FineWeb-HQ](https://huggingface.co/datasets/epfml/FineWeb-HQ) — Secondary training data
+- [TokenMonster](https://huggingface.co/alasdairforsythe/tokenmonster) — Tokenizer vocabulary
+- [HuggingFace Accelerate](https://github.com/huggingface/accelerate) — Training framework
+- [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) — Quantization library
+
+---
+
+## Environmental Impact
+
+| Factor | Value |
+|---|---|
+| Hardware | 2× NVIDIA Tesla T4 (both active) |
+| Active Training Duration | ~2.13 hours |
+| Cloud Provider | Kaggle (free tier) |
+| Compute Region | Western USA |
+| Estimated Carbon | Minimal (< 0.5 kg CO₂e estimated) |
+
+Training on free-tier cloud compute demonstrates that meaningful SLM research is accessible to independent researchers and students without significant hardware investment or carbon cost.
+
+---
+
+## Citation
+
+```bibtex
+@misc{izumoto2026stentor2_12m,
+  title        = {Stentor2-12M},
+  author       = {Kai Izumoto},
+  year         = {2026},
+  publisher    = {StentorLabs},
+  howpublished = {\url{https://huggingface.co/StentorLabs/Stentor2-12M}},
+  note         = {Official release of the Stentor2 model family.
+                  12.3M parameter LlamaForCausalLM base model trained on
+                  StenCore and FineWeb-HQ with a TokenMonster 8K vocabulary.
+                  Dual-GPU distributed training on 2x NVIDIA T4.
+                  Apache 2.0 license.}
+}
+```
+
+---
+
+## Model Card Contact
+
+Questions, benchmarks, or feedback: [StentorLabs@gmail.com](mailto:StentorLabs@gmail.com) or open a [discussion](https://huggingface.co/StentorLabs/Stentor2-12M/discussions).
+
+---
+
+<p align="center">
+  Made with ❤️ by <a href="https://huggingface.co/StentorLabs">StentorLabs</a><br>
+  <i>Democratizing AI through accessible, efficient models</i>
+</p>