Instructions to use StentorLabs/Stentor2-12M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use StentorLabs/Stentor2-12M with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="StentorLabs/Stentor2-12M")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("StentorLabs/Stentor2-12M")
model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use StentorLabs/Stentor2-12M with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "StentorLabs/Stentor2-12M"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "StentorLabs/Stentor2-12M",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/StentorLabs/Stentor2-12M

SGLang

How to use StentorLabs/Stentor2-12M with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "StentorLabs/Stentor2-12M" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "StentorLabs/Stentor2-12M",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "StentorLabs/Stentor2-12M" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "StentorLabs/Stentor2-12M",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use StentorLabs/Stentor2-12M with Docker Model Runner:
```
docker model run hf.co/StentorLabs/Stentor2-12M
```

StentorLabs commited on Mar 8

Commit

b077f11

verified ·

1 Parent(s): 6de881a

Delete README.md

Browse files

Files changed (1) hide show

README.md +0 -982

README.md DELETED Viewed

@@ -1,982 +0,0 @@
----
-language:
-- en
-license: apache-2.0
-library_name: transformers
-tags:
-- text-generation
-- llama
-- small-language-model
-- efficient
-- edge-deployment
-- speculative-decoding
-- tiny-model
-- 12m-parameters
-- kaggle-trained
-- educational
-- research
-- low-resource
-- cpu-inference
-- mobile-deployment
-- stentor2
-- tokenmonster
-pipeline_tag: text-generation
-datasets:
-- epfml/FineWeb-HQ
-- StentorLabs/StenCore
-thumbnail: https://huggingface.co/StentorLabs/Stentor2-12M/resolve/main/thumbnail.png
-widget:
-- text: Once upon a time
-  example_title: Story Generation
-- text: Explain neural networks in simple terms.
-  example_title: Toy Explanation (Often Wrong)
-- text: 'def fibonacci(n):'
-  example_title: Code Continuation
-- text: The laws of thermodynamics describe
-  example_title: Science Continuation
-model_card_authors:
-- StentorLabs
-model-index:
-- name: Stentor2-12M
-  results:
-  - task:
-      type: text-generation
-    dataset:
-      name: FineWeb-HQ (validation split)
-      type: epfml/FineWeb-HQ
-    metrics:
-    - name: Best Validation Loss
-      type: loss
-      value: 3.2814
-    - name: Best Perplexity (at best checkpoint)
-      type: perplexity
-      value: 26.61
-    - name: Final Epoch Validation Loss
-      type: loss
-      value: 3.2565
-    - name: Final Epoch Perplexity
-      type: perplexity
-      value: 25.96
----
-# Stentor2-12M
-![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)
-![Model Size](https://img.shields.io/badge/parameters-12.3M-green.svg)
-![Training Time](https://img.shields.io/badge/training-2.1h-orange.svg)
-![Hardware](https://img.shields.io/badge/hardware-2x%20T4-red.svg)
-![Context Length](https://img.shields.io/badge/context-1024%20tokens-purple.svg)
-![Vocab Size](https://img.shields.io/badge/vocab-8064%20tokens-blue.svg)
-[![Hugging Face](https://img.shields.io/badge/🤗-Hugging%20Face-yellow.svg)](https://huggingface.co/StentorLabs)
-> 🔬 **Research Artifact — Not a Production Model.** This model has no safety tuning and is not suitable for deployment in any user-facing application. See [Intended Uses](#use-cases--intended-uses) for details.
----
-## Table of Contents
-1. [What Is This?](#what-is-this)
-2. [The Core Design Insight: Vocabulary Efficiency](#the-core-design-insight-vocabulary-efficiency)
-3. [Head-to-Head: Stentor v1 vs Stentor2](#head-to-head-stentor-v1-vs-stentor2)
-4. [Quick Start](#quick-start)
-5. [Important Limitations](#important-limitations)
-6. [Honest Notices](#-honest-notices)
-7. [PDF Tokens & The Replacement Character](#pdf-tokens--the-replacement-character)
-8. [Model Architecture — Full Specification](#model-architecture--full-specification)
-9. [The Tokenizer: TokenMonster](#the-tokenizer-tokenmonster)
-10. [Training Infrastructure](#training-infrastructure)
-11. [Training Hyperparameters — Complete Reference](#training-hyperparameters--complete-reference)
-12. [The T4 Mixed-Precision Recipe — Deep Dive](#the-t4-mixed-precision-recipe--deep-dive)
-13. [Data Pipeline](#data-pipeline)
-14. [Weight Initialization](#weight-initialization)
-15. [Evaluation & Results](#evaluation--results)
-16. [Training Dynamics](#training-dynamics)
-17. [Use Cases & Intended Uses](#use-cases--intended-uses)
-18. [Out-of-Scope Uses](#out-of-scope-uses)
-19. [Ethical Considerations & Societal Impact](#ethical-considerations--societal-impact)
-20. [Inference Guide](#inference-guide)
-21. [Free Inference — Try It Now](#-free-inference--try-it-now)
-22. [Real Model Responses](#real-model-responses)
-23. [Quantization](#quantization)
-24. [Format Conversion](#format-conversion)
-25. [Speculative Decoding](#speculative-decoding)
-26. [Bias, Risks & Limitations](#bias-risks--limitations)
-27. [Related Work](#related-work)
-28. [Environmental Impact](#environmental-impact)
-29. [Citation](#citation)
----
-## What Is This?
-Stentor2-12M is the first production release from the **Stentor2** model family — a ground-up redesign of the original Stentor v1 line. At ~12.3M parameters, it is a compact base language model built entirely from scratch on free-tier Kaggle compute using two NVIDIA Tesla T4 GPUs.
-Like all Stentor models, this is a **base next-token predictor**, not a chat assistant. It will not reliably follow instructions, has no safety tuning, and is best used for research, prototyping, speculative decoding, and edge-deployment experimentation. The value of this model is not its conversational capability — it's what it represents architecturally: a dramatic efficiency gain over v1 at the same scale, achieved by fixing the root cause of v1's underperformance.
----
-## The Core Design Insight: Vocabulary Efficiency
-The most consequential change in Stentor2 is the replacement of the standard Llama/Mistral 32,768-token vocabulary with a purpose-built **8,000-token English vocabulary** from the TokenMonster project (`english-8000-strict-nocapcode-v1`, padded to 8,064 for hardware alignment).
-This is not a minor tweak — it is the entire architectural story of Stentor2.
-### Why Vocabulary Size Matters So Much at This Scale
-In a transformer language model, the embedding table has shape `[vocab_size × hidden_size]`. When you tie word embeddings (share the embedding and output projection weights, which Stentor does), this table appears once in the parameter count. At 12M total parameters, the fraction consumed by this table dictates how much "brain" is left over for the actual transformer layers.
-**Stentor-12M (v1)** used a 32,768-token vocabulary. At a hidden size of 192:
-```
-embedding_params = 32,768 × 192 = 6,291,456
-total_params     = 12,047,040
-embedding_share  = 52.2%
-```
-Over half of the model was a lookup table. The transformer stack — the part that actually *learns language patterns* — had fewer than 6 million parameters to work with.
-**Stentor2-12M** uses an 8,064-token vocabulary. At a hidden size of 256:
-```
-embedding_params = 8,064 × 256 = 2,064,384
-total_params     = 12,294,400
-embedding_share  = 16.8%
-```
-By shrinking the vocabulary, the embedding table was cut from 6.3M to 2.1M parameters — freeing up ~4.2M parameters redistributed into transformer depth (12 layers vs 9) and width (hidden size 256 vs 192), where they contribute directly to language modeling quality.
-The result is a **~70.1% reduction in perplexity** (89.01 → ~26.6) compared to Stentor-12M v1.
----
-## Head-to-Head: Stentor v1 vs Stentor2
-| Property | Stentor-12M (v1) | Stentor2-12M |
-|---|---|---|
-| **Vocabulary** | 32,768 (Mistral BPE) | 8,064 (TokenMonster English) |
-| **Hidden Size** | 192 | 256 |
-| **Intermediate Size** | 576 | 768 |
-| **Num Layers** | 9 | 12 |
-| **Attention Heads** | 3 | 4 |
-| **Head Dimension** | 64 | 64 |
-| **Context Length** | 512 tokens | 1,024 tokens |
-| **Total Parameters** | 12,047,040 | 12,294,400 |
-| **Embedding Share** | 52.2% | 16.8% |
-| **Non-Embedding Params** | ~5.76M | ~10.23M |
-| **Source Token Budget** | 200M | 240M |
-| **Total Token Budget** | ~200M | 480M |
-| **Training Time** | ~1.3h | ~2.1h |
-| **Training Processes** | 1 | 2 |
-| **Best Perplexity** | 89.01 | **26.61** |
-| **Perplexity Reduction** | — | **~70.1%** |
-| **Tokenizer** | Mistral BPE | TokenMonster |
-| **Architecture** | LlamaForCausalLM | LlamaForCausalLM |
-| **Training Precision** | fp16 | fp16 + FP32 norms/critical layers |
----
-## 🚀 Quick Start
-### 1. Install Dependencies
-```bash
-pip install transformers torch safetensors huggingface_hub tokenmonster
-```
-### 2. Load the Model
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
-model = AutoModelForCausalLM.from_pretrained(
-    "StentorLabs/Stentor2-12M",
-    torch_dtype=torch.float16,
-)
-tokenizer = AutoTokenizer.from_pretrained(
-    "StentorLabs/Stentor2-12M",
-    trust_remote_code=True,
-)
-```
-### 3. Generate Text
-```python
-input_ids      = torch.tensor([tokenizer.encode("The history of computing")], dtype=torch.long).to(next(model.parameters()).device)
-attention_mask = torch.ones_like(input_ids)
-with torch.inference_mode():
-    output = model.generate(
-        input_ids,
-        attention_mask=attention_mask,
-        max_new_tokens=80,
-        do_sample=True,
-        temperature=1.1,
-        top_p=0.55,
-        repetition_penalty=1.15,
-        pad_token_id=tokenizer.pad_token_id,
-    )
-print(tokenizer.decode(output[0].tolist()))
-```
-> **Why `attention_mask`?** The model's pad token and EOS token are the same ID. Without an explicit attention mask, HuggingFace throws a warning because it can't tell which tokens are real vs padding. Passing `torch.ones_like(input_ids)` tells the model that every token in the input is real.
-### 4. Recommended Generation Settings
-| Parameter | Recommended Range | Personal Favorite | Notes |
-|---|---|---|---|
-| `temperature` | 0.6 – 0.9 | **0.8** | Below 0.6 causes heavy `<22>` (PDF token) output; above 0.9 gets chaotic |
-| `top_p` | 0.6 – 0.8 | **0.7** | Below 0.6 also increases `<22>` tokens significantly |
-| `repetition_penalty` | 1.1 – 1.4 | **1.2** | Without this the model will loop; 1.2–1.3 hits the sweet spot |
-| `max_new_tokens` | 10 – 500 | — | Model stays mostly on topic but may drift at longer lengths |
-> ⚠️ **Always use `repetition_penalty ≥ 1.1`.** Without it, this model will fall into repetitive loops. With it on, output quality is mostly stable.
-> ⚠️ **Keep temperature and top_p above 0.6.** Going below this threshold causes the model to lean heavily on PDF-derived tokens that render as `<22>` (the Unicode replacement character). See [PDF Tokens](#pdf-tokens--the-replacement-character) for details.
----
-## ⚠️ Important Limitations
-- **Not Instruction-Tuned:** This is a base model. It will often ignore prompts, continue in unexpected directions, or respond off-topic.
-- **No Safety Tuning:** No RLHF, no constitutional AI, no content filtering.
-- **Limited World Knowledge:** ~12M parameters cannot store meaningful world knowledge.
-- **Context Window:** Hard limit of 1,024 tokens.
-- **English Only:** The TokenMonster vocabulary is English-specific.
-- **TokenMonster Required:** Uses a TokenMonster adapter. Make sure `tokenmonster` is installed (`pip install tokenmonster`). You do not need a custom model loader — standard `AutoModelForCausalLM` works — but the tokenmonster package is still required for the tokenizer.
-- **PDF Tokens (`<22>`):** The model was trained on data containing PDF-extracted text, and will sometimes generate tokens that render as `<22>` (the Unicode replacement character). See the [PDF Tokens](#pdf-tokens--the-replacement-character) section below.
-- **Repetition Without Penalty:** Without `repetition_penalty`, the model will fall into loops. Always use `repetition_penalty ≥ 1.1`.
-- **Shared Tensor Warning:** When saving or loading, you may see: `Removed shared tensor {'lm_head.weight'} while saving`. This is expected from tied word embeddings and is safe to ignore.
----
-## 📋 Honest Notices
-These are candid, first-hand observations about how this model actually behaves.
-1. **Significantly more coherent than its predecessors.** Output quality is a clear step up from Stentor2-12M-Preview and Stentor-12M v1. It is competitive with — and in some runs slightly better than — Stentor-30M, despite being less than half the parameter count.
-2. **Generates PDF tokens (`<22>`).** The model will sometimes output tokens that display as `<22>` — the Unicode replacement character. These are valid tokens from PDF-extracted training data that most systems cannot natively render, and that most language models never produce. They are not a decoding error. See [PDF Tokens](#pdf-tokens--the-replacement-character) for details.
-3. **Prone to repetition without `repetition_penalty`.** Left unchecked, the model loops. With `repetition_penalty` set to 1.1–1.4, this is mostly resolved and output quality is stable.
-4. **No custom model loader required. 🎉** Unlike Stentor2-12M-Preview, this model loads with standard `AutoModelForCausalLM.from_pretrained()`. No special loading code needed.
-5. **You still need to install `tokenmonster`.** The tokenizer wraps TokenMonster and requires `pip install tokenmonster`. It's a one-line install and nothing like the complexity of a fully custom tokenizer, but it is a required dependency.
-6. **The model talks about education and academics — a lot.** Trained on FineWeb-HQ (a high-quality filtered web corpus with significant PDF and educational content) and StenCore (100% PDFs), the model has a strong prior toward academic language, school systems, curriculum, research, and formal writing. Prompts unrelated to education will frequently be redirected toward educational framing anyway.
----
-## PDF Tokens & The Replacement Character
-Stentor2-12M was trained on FineWeb-HQ and StenCore, which includes a substantial amount of text extracted from PDF documents. PDFs often contain binary sequences or encoding artifacts that survive text extraction as raw bytes outside the standard UTF-8 range. These get tokenized and trained on as valid tokens.
-As a result, the model has learned to generate these tokens — and will do so, especially at lower temperatures or lower top-p values. When decoded, they render as **`<22>`** (U+FFFD, the Unicode replacement character), because most systems substitute this symbol for unrepresentable byte sequences.
-**What this looks like:**
-```
-Academics → <22><22><22><22><22><22><22><22><22><22><22><22><22><22>...
-```
-**How to reduce it:** Keep `temperature ≥ 0.6` and `top_p ≥ 0.6`. Below these thresholds, PDF token probability rises sharply. At well-tuned settings (temp 0.8, top_p 0.7), `<22>` tokens appear occasionally but do not dominate output.
-**This is not a bug.** Most language models are trained on clean text and never encounter these sequences. Stentor2-12M is unusual in that it can generate them — a side effect of training on real-world PDF-extracted data.
----
-## Model Architecture — Full Specification
-Stentor2-12M is a `LlamaForCausalLM` model.
-### Core Configuration
-| Component | Value | Derivation |
-|---|---|---|
-| **Architecture** | `LlamaForCausalLM` | Hard-coded in training script |
-| **Hidden Size** | 256 | embedding_params (2,064,384) ÷ vocab_size (8,064) = 256 ✓ |
-| **Intermediate Size (FFN)** | 768 | Hidden × 3 (verified via total param count) |
-| **Num Hidden Layers** | 12 | Verified via total param count formula |
-| **Num Attention Heads** | 4 | Hidden ÷ head_dim = 256 ÷ 64 = 4 |
-| **Num Key/Value Heads** | 4 | Full MHA (no GQA at this scale) |
-| **Head Dimension** | 64 | Enforced by training script |
-| **Vocab Size** | 8,064 | TokenMonster 8K base + 62 padding tokens (multiple of 128) |
-| **Max Position Embeddings** | 1,024 | `block_size` default in training script |
-| **Hidden Activation** | SiLU | LlamaForCausalLM default |
-| **Positional Encoding** | RoPE | `rope_theta = 10,000.0` |
-| **RMS Norm Epsilon** | 1e-5 | Default in training script |
-| **Tie Word Embeddings** | True | Shared embedding / LM head weights |
-| **Attention Implementation** | SDPA | PyTorch Scaled Dot Product Attention |
-### Parameter Count Breakdown
-```python
-def estimate_llama_params(vocab_size, hidden_size, intermediate_size,
-                          num_hidden_layers, num_attention_heads, num_key_value_heads):
-    kv_dim = int(hidden_size * num_key_value_heads / num_attention_heads)
-    attn = 2 * hidden_size * hidden_size + 2 * hidden_size * kv_dim
-    mlp  = 3 * hidden_size * intermediate_size
-    norm = 2 * hidden_size
-    total = vocab_size * hidden_size + num_hidden_layers * (attn + mlp + norm) + hidden_size
-    return total
-```
-Plugging in Stentor2 values:
-```
-kv_dim  = 256 * 4 / 4 = 256
-attn    = 2×256×256 + 2×256×256 = 262,144
-mlp     = 3×256×768 = 589,824
-norm    = 2×256 = 512
-per_layer = 852,480
-embedding  = 8,064 × 256  = 2,064,384
-layers     = 12 × 852,480 = 10,229,760
-final_norm = 256
-total = 2,064,384 + 10,229,760 + 256 = 12,294,400 ✓
-```
-| Component | Parameters | % of Total |
-|---|---|---|
-| Embedding Table (tied with LM Head) | 2,064,384 | 16.8% |
-| Transformer Layers × 12 | 10,229,760 | 83.2% |
-| — Attention (per layer × 12) | 3,145,728 | 25.6% |
-| — FFN/MLP (per layer × 12) | 7,077,888 | 57.5% |
-| — Layer Norms (per layer × 12) | 6,144 | 0.05% |
-| Final RMS Norm | 256 | 0.002% |
-| **Total** | **12,294,400** | **100%** |
----
-## The Tokenizer: TokenMonster
-Stentor2 uses a custom tokenizer adapter wrapping the **TokenMonster** `english-8000-strict-nocapcode-v1` vocabulary.
-### What Is TokenMonster?
-TokenMonster ([alasdairforsythe/tokenmonster](https://huggingface.co/alasdairforsythe/tokenmonster)) is an alternative tokenization approach optimized for compact English vocabulary sizes.
-### Tokenizer Efficiency vs. v1
-This tokenizer produces **more tokens per word** compared to Stentor v1. This is expected — smaller vocabulary means more tokens per word on average. The ~70.1% perplexity improvement shows the tradeoff was worth it.
-### Vocabulary Construction
-1. Base vocabulary loaded from `alasdairforsythe/tokenmonster` → `vocabs/english-8000-strict-nocapcode-v1.vocab`
-2. Special tokens added: `</s>` (EOS), `<s>` (BOS), `<pad>` (set equal to EOS)
-3. A default chat template injected for structural compatibility
-4. Vocabulary padded to the nearest multiple of 128 → **8,064 tokens**
-### Tokenizer Configuration
-```json
-{
-  "tokenizer_type": "tokenmonster",
-  "vocab_file": "tokenmonster.vocab",
-  "model_max_length": 1024,
-  "eos_token": "</s>",
-  "bos_token": "<s>",
-  "pad_token": "</s>",
-  "vocab_size": 8064
-}
-```
-### Chat Template
-```jinja
-{% for message in messages %}
-<|{{ message['role'] }}|>
-{{ message['content'] }}
-{% endfor %}
-{% if add_generation_prompt %}<|assistant|>
-{% endif %}
-```
----
-## Training Infrastructure
-### Hardware
-| Component | Specification |
-|---|---|
-| GPU Count | 2× NVIDIA Tesla T4 |
-| VRAM per GPU | 15.64 GB |
-| Total VRAM | ~31.3 GB |
-| Active Training Processes | **2** (dual-process via HuggingFace Accelerate) |
-| Platform | Kaggle Notebooks (free tier) |
-| Accelerator Library | HuggingFace Accelerate |
-### Software Stack
-| Package | Role |
-|---|---|
-| PyTorch | Core tensor operations and autograd |
-| HuggingFace Transformers | Model architecture (LlamaForCausalLM) |
-| HuggingFace Accelerate | Training loop and device management |
-| HuggingFace Datasets | Data loading |
-| bitsandbytes | Dual-GPU optimization |
-| tokenmonster | Custom vocabulary |
-| safetensors | Model serialization |
----
-## Training Hyperparameters — Complete Reference
-### Core Training Parameters
-| Hyperparameter | Value | Notes |
-|---|---|---|
-| `learning_rate` | 8e-4 | AdamW LR (script default) |
-| `weight_decay` | 0.01 | Applied to non-embedding, non-norm, non-bias params |
-| `max_grad_norm` | 1.0 | Gradient clipping threshold |
-| `optimizer` | AdamW | With `betas=(0.9, 0.95)`, `eps=1e-8` |
-| `scheduler` | Cosine | Cosine decay with linear warmup |
-| `warmup_ratio` | 0.05 | → **477 warmup steps** |
-| `stable_ratio` | 0.80 | → **7,641 stable steps** |
-| `source_token_budget` | 240,000,000 | Source data token cap |
-| `token_budget` | 480,000,000 | Total training tokens; **hit at ~3,662 steps** |
-| `max_train_steps` | 9,552 | Configured limit; token budget hit first |
-| `seed` | 42 | Reproducibility seed |
-| `mixed_precision` | fp16 | All activations/gradients in FP16 |
-### Batch & Sequence Parameters
-| Hyperparameter | Value | Notes |
-|---|---|---|
-| `per_device_train_batch_size` | **32** | Per GPU per gradient accumulation step |
-| `per_device_eval_batch_size` | 16 | Evaluation batch size |
-| `gradient_accumulation_steps` | **2** | Effective optimizer steps every 2 forward passes |
-| `total_batch_size` | **128** | `per_device × processes × grad_accum = 32×2×2` |
-| `block_size` | 1,024 | Sequence length; training packed to this size |
-| `tokens_per_optimizer_step` | **131,072** | `total_batch_size × block_size = 128×1024` |
-| `num_train_epochs` | 2 | Both epochs completed before token budget |
-### Evaluation & Checkpointing
-| Hyperparameter | Value |
-|---|---|
-| `eval_steps` | **900** |
-| `best_eval_steps` | 900 |
-| `best_eval_start_step` | 1,500 |
-| `save_every_minutes` | 30 |
-| `save_total_limit` | 2 |
-| `logging_steps` | **300** |
-| `max_eval_samples` | 5,000 |
-### AdamW Optimizer — Detailed
-- **Decay group:** All `nn.Linear` weight matrices → `weight_decay = 0.01`
-- **No-decay group:** Bias terms, normalization parameters, embedding parameters → `weight_decay = 0.0`
-- **Betas:** `(0.9, 0.95)`
-- **Epsilon:** `1e-8`
-- **Fused kernel:** Enabled when CUDA is available
-### Learning Rate Schedule
-```
-Phase 1 — Warmup (steps 0–477):
-  LR ramps linearly from 0 → 8e-4
-Phase 2 — Cosine Decay (steps 477–9,552):
-  LR follows cosine curve from 8e-4 → 0
-  (Training ended early at ~3,662 steps due to token budget)
-```
----
-## The T4 Mixed-Precision Recipe — Deep Dive
-The training pipeline uses a custom **T4 Mixed-Precision Recipe** designed for stable fp16 training on NVIDIA Tesla T4 GPUs.
-### 1. FP32 Normalization Layers (25 modules)
-All RMSNorm modules are monkey-patched to run in FP32 regardless of input dtype:
-```python
-def _fp32_norm_forward(hidden_states, *args, **kwargs):
-    input_dtype = hidden_states.dtype
-    output = original_forward(hidden_states.float().contiguous(), *args, **kwargs)
-    return output.clone().to(input_dtype)
-```
-The `.clone()` call prevents returning graph-managed buffers that can be overwritten. The `.contiguous()` prevents strided-tensor issues in FP32 norm ops.
-**Count:** 12 layers × 2 norms each (input + post-attention) + 1 final norm = **25 modules total**.
-### 2. FP32 Critical Layers (2 layers)
-The **first 2 transformer layers** are designated as critical and run entirely in FP32:
-- Weights cast to `.float()` at setup time
-- `forward()` monkey-patched to cast all inputs to FP32 and outputs back to original dtype
-- `torch.amp.autocast("cuda", enabled=False)` prevents re-downcasting inside the layer
-**Rationale:** The first layers handle embedding projection and initial feature extraction. Instability here corrupts the entire forward pass. Running these in FP32 provides a stability floor at minimal compute cost.
-### 3. FP32 Attention Softmax — Skipped
-The FP32 softmax wrapper was **not applied** (0 modules). As logged during training:
-```
-Stability tricks: skipping FP32 softmax wrapper because
-SDPA/FlashAttention kernels require fp16/bf16 inputs for fast paths.
-```
-PyTorch's SDPA implementation handles numerical stability internally and requires fp16/bf16 inputs for its optimized code paths. Wrapping softmax in FP32 would bypass these fast kernels.
-### T4 Recipe Summary
-| Technique | Count | Scope |
-|---|---|---|
-| FP32 norm modules | **25** | All RMSNorm layers |
-| FP32 critical layers | **2** | First 2 transformer layers |
-| FP32 softmax modules | **0** | Skipped — SDPA incompatible |
----
-## Data Pipeline
-### Dataset
-The model was trained on **FineWeb-HQ and StenCore** ([epfml/FineWeb-HQ](https://huggingface.co/datasets/epfml/FineWeb-HQ)) and ([StentorLabs/StenCore](https://huggingface.co/datasets/StentorLabs/StenCore))— high-quality filtered web and PDF corpuses.
-**Total tokens processed:** 480,116,736 (~480M, budget-limited run)
-**Source tokens (raw data):** 240,000,000 (240M source token budget)
-### Text Preprocessing
-```python
-def clean_text(text: str) -> str:
-    text = unicodedata.normalize("NFKC", text)
-    lines = [line.strip() for line in text.splitlines() if line.strip()]
-    text = " ".join(lines)
-    text = " ".join(text.split())
-    return text
-```
-- **NFKC normalization** maps visually equivalent Unicode characters to canonical form
-- **Whitespace collapse** ensures consistent tokenization
-### Sequence Packing
-After tokenization, samples are packed into fixed 1,024-token blocks. Labels for packed sequences are identical to `input_ids` (causal LM). No special boundary masking between packed samples.
----
-## Weight Initialization
-```python
-def initialize_weights(model, std=0.02):
-    for module in model.modules():
-        if isinstance(module, (nn.Linear, nn.Embedding)):
-            module.weight.data.normal_(mean=0.0, std=std)
-            if module.bias is not None:
-                module.bias.data.zero_()
-        elif "rmsnorm" in type(module).__name__.lower():
-            if module.weight is not None:
-                module.weight.data.fill_(1.0)
-            if module.bias is not None:
-                module.bias.data.zero_()
-```
-- Linear/Embedding layers: normal(0, 0.02)
-- RMSNorm scale weights: 1.0 (identity at start)
-- Biases: zero
----
-## Evaluation & Results
-### Results Summary
-| Checkpoint | Step | Eval Loss | Perplexity |
-|---|---|---|---|
-| Early | 900 | — | — |
-| First best | 1,500 | 3.5410 | 34.50 |
-| Second best | 2,400 | 3.3743 | 29.20 |
-| Epoch 0 end | ~2,350 | 3.3776 | 29.30 |
-| **Best Checkpoint** | **3,300** | **3.2814** | **26.61** |
-| Epoch 1 end (final) | ~3,662 | 3.2565 | 25.96 |
-### Comparison to Stentor v1
-| Model | Best Eval Loss | Best Perplexity | Improvement |
-|---|---|---|---|
-| Stentor-12M (v1) | 4.4887 | 89.01 | — |
-| **Stentor2-12M** | **3.2814** | **26.61** | **↓70.1% perplexity** |
-> **Note:** The comparison is close but not perfectly controlled — v1 trained on a mix of FineWeb-Edu and Cosmopedia v2, while Stentor2 trained on FineWeb-HQ and StenCore. Both use educational-quality text at the same parameter count.
----
-## Training Dynamics
-The training run processed **480,116,736 tokens** across **2 epochs**, stopping when the token budget was reached at approximately **step 3,662** (well before the configured `max_train_steps` of 9,552).
-**Epoch 0 (~2,350 steps, 3,497s):** Loss dropped rapidly from ~5.0 to ~3.2. Best checkpoint checkpoints updated at steps 1,500 (3.5410) and 2,400 (3.3743). Epoch-end eval: loss 3.3776, ppl 29.30.
-**Epoch 1 (~1,310 additional steps, 1,906s):** Continued improvement. New best at step 3,300 (loss 3.2814). Token budget hit shortly after step 3,600. Epoch-end eval: loss 3.2565, ppl 25.96.
-**Throughput:** ~91,600–92,000 tokens/sec average throughout training.
----
-## Use Cases & Intended Uses
-| Use Case | Suitability | Notes |
-|---|---|---|
-| Studying transformer training dynamics | ✅ High | Small enough to train/fine-tune on free compute |
-| Tokenization efficiency research | ✅ High | 8K vs 32K vocab tradeoff directly observable |
-| Speculative decoding experiments | ✅ High | Fast enough to serve as a draft model |
-| Benchmarking CPU/edge inference latency | ✅ High | ~12MB in FP16, runs on any hardware |
-| Testing quantization/conversion pipelines | ✅ High | GGUF, ONNX, INT8 pipeline validation |
-| Teaching material for LLM courses | ✅ High | Architecture simple enough to trace by hand |
-| Text continuation / creative prompting | ✅ High | Works well enough for basic continuation |
-| Domain-specific fine-tuning research | ✅ High | Small enough to iterate rapidly |
-| Factual Q&A | ❌ Not suitable | No reliable world knowledge |
-| Production deployment | ❌ Not suitable | No safety tuning |
-| Non-English text | ❌ Not suitable | TokenMonster vocab is English-only |
-| Long-document tasks (Only 1k context) | ❌ Not suitable | Context too small |
----
-## Out-of-Scope Uses
-- **User-facing applications of any kind** — No safety filtering, no alignment, no factual reliability.
-- **Medical, legal, or financial advice** — 12M parameters cannot store or reason over specialized knowledge.
-- **Generating content about real people** — Outputs mentioning real people are likely to be fabricated.
-- **Automated content pipelines** — Output quality is insufficient for unreviewed publication.
-- **Non-English use** — Vocabulary is English-only.
-- **Instruction following** — This is a base model.
----
-## Ethical Considerations & Societal Impact
-### Inherited Data Biases
-Trained on FineWeb-HQ and StenCore, a filtered subset of Common Crawls on the web and PDFs. Inherits:
-- **Western-centric perspective** — Educational content skews toward Western viewpoints.
-- **English monolingualism** — Training data and vocabulary are English-only.
-- **Demographic underrepresentation** — Groups underrepresented in English educational web content will be underrepresented in outputs.
-### No Safety Tuning
-No RLHF, no DPO, no constitutional AI, no content filtering.
-### Positive Aspects
-- **Democratizing AI research** — Trained entirely on free-tier Kaggle compute.
-- **Transparency** — Full training hyperparameters, architecture details, and deep details published.
-- **Minimal environmental footprint** — ~2.1 hours of dual-GPU compute.
----
-## Inference Guide
-### Basic Generation
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
-model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16)
-tokenizer = AutoTokenizer.from_pretrained("StentorLabs/Stentor2-12M", trust_remote_code=True)
-model = model.to("cuda").eval()
-def generate(prompt, max_new_tokens=50, temperature=0.9, top_p=0.65):
-    input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long).to(model.device)
-    attention_mask = torch.ones_like(input_ids)
-    with torch.inference_mode():
-        output = model.generate(
-            input_ids,
-            attention_mask=attention_mask,
-            max_new_tokens=max_new_tokens,
-            do_sample=True,
-            temperature=temperature,
-            top_p=top_p,
-            repetition_penalty=1.15,
-            pad_token_id=tokenizer.pad_token_id,
-        )
-    new_ids = output[0][input_ids.shape[1]:].tolist()
-    return tokenizer.decode(new_ids).strip()
-print(generate("The history of computing began"))
-```
----
-## 🚀 Free Inference — Try It Now
-**No GPU, no setup, no API key required.**
-StentorLabs hosts a free, unlimited inference demo for all Stentor models at:
-> 🔗 **[https://huggingface.co/spaces/StentorLabs/StentorLabs-demo_space](https://huggingface.co/spaces/StentorLabs/StentorLabs-demo_space)**
-### Features
-| Mode | Description |
-|---|---|
-| **Normal Generation** | Standard text completion with adjustable temp, top-p, repetition penalty, and max tokens |
-| **Token Probability Explorer** | Inspect per-token probability distributions as the model generates |
-| **Temperature Sweep** | Generate the same prompt across multiple temperatures simultaneously to compare behavior |
-| **Head-to-Head Comparison** | Run two Stentor models side-by-side on the same prompt |
-All parameters (temperature, top-p, repetition penalty, max tokens) are fully customizable. Inference is free and unlimited.
----
-## Real Model Responses
-The following are unfiltered outputs from Stentor2-12M. The first three are the very first generations run (no cherry-picking), the fourth is a deliberately bad example, and the fifth is a deliberately good one.
----
-**Sample 1** — `temp=0.8, top_p=0.7, rep_penalty=1.2, max_tokens=140`
-**Prompt:** `5 Tips and Tricks for Difficult Homework.`
-```
-SPECIAL STUDY SUBJECT: The Earth's Fall Times <22><22><22> Review the article with reference to
-"A Tyre" <22><22><22> Revise your answers carefully and read it carefully. Write down what you
-think you did and what you think it had to do with your answer. <22><22><22> Review your notes.
-Read a book from the book, and review your answers. What would you do? <22><22><22> Do some
-research findings on your own reading. <22><22><22> Invite students to read them. <22><22><22>
-Try to learn the text by using different sources. <22>
-```
-> The model completely ignores the prompt and defaults to academic/educational framing. PDF tokens (`<22>`) appear throughout as list separators — a characteristic artifact of the training data.
----
-**Sample 2** — `temp=0.8, top_p=0.7, rep_penalty=1.3, max_tokens=140`
-**Prompt:** `How to Properly Cite Sources`
-```
-The Yellowstone Elementary School (CFP) is committed to providing an integrated and well-rounded
-education system for students of all ages. The curriculum is designed to promote academic
-achievement, instructional opportunities and a wider range of subject areas of study and practice
-as well as teaching the student through its own learning style. This program is an important part
-of the school's success in school and homework are set out in the district. We believe that all
-pupils who receive a free school year will be able to attend classroom lessons from grade level
-levels and will be required to complete their classes at any time during their summer break. We
-are pleased that our current PSHCE
-```
-> Coherent, grammatically sound institutional prose — but completely unrelated to citing sources. Classic Stentor2 behavior: latches onto educational register and produces confident, fluent non-sequitur.
----
-**Sample 3** — `temp=0.6, top_p=0.75, rep_penalty=1.3, max_tokens=120`
-**Prompt:** `How to Lose Weight Fast`
-```
-The U.S. Department of Agriculture (USDA) is committed to providing the necessary services to
-the economy and professions of urban development, as well as support for the protection of the
-environment and the preservation of all living things within the community. The U.S. Government
-provides a framework for developing this information in partnership with local organizations that
-provide education, health care, and employment opportunities for all people affected by the
-COVID-19 pandemic. We aim to ensure that the government's commitment to sustainable housing
-through increased infrastructure and public transportation systems; therefore, we seek to
-```
-> No connection to weight loss. Government/policy language — another common register for this model. Fluent, well-structured, completely off-topic.
----
-**Sample 4 (Hand-Picked Bad)** — `temp=0.8, top_p=0.5, rep_penalty=1.0, max_tokens=40`
-**Prompt:** `Academics`
-```
-<22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22><22>
-<22><22><22><22><22><22><22><22><22><22><22><22><22><22>
-```
-> Pure PDF token collapse. This is what happens with `rep_penalty=1.0` (off), `top_p=0.5` (too low), and a one-word prompt. All three conditions together push the model into its PDF-token attractor state. Do not use these settings.
----
-**Sample 5 (Hand-Picked Good)** — `temp=1.0, top_p=0.7, rep_penalty=1.4, max_tokens=45`
-**Prompt:** `How to do Homework Assignments`
-```
-Reasoning and Decision Making This Project is a great way to help students to succeed in school.
-The project helps the student achieve better, the more they need to grow their faiths and to
-support them in success as they become
-```
-> Clean, readable, no PDF tokens, no repetition. Not exactly on-topic (pivot to "project" framing is typical), but this is what good Stentor2 output looks like: coherent educational prose, correct grammar, sensible continuation.
----
-## Quantization
-### FP16
-```python
-model = AutoModelForCausalLM.from_pretrained("StentorLabs/Stentor2-12M", torch_dtype=torch.float16)
-model = model.to("cuda")
-```
-### Dynamic INT8 (CPU)
-```python
-import torch
-model_int8 = torch.quantization.quantize_dynamic(
-    model.to("cpu"),
-    {torch.nn.Linear},
-    dtype=torch.qint8,
-)
-```
----
-## Format Conversion
-### Convert to GGUF (for llama.cpp)
-```bash
-git clone https://github.com/ggerganov/llama.cpp
-cd llama.cpp && pip install -r requirements.txt
-huggingface-cli download StentorLabs/Stentor2-12M --local-dir stentor2-12m
-python convert_hf_to_gguf.py stentor2-12m/ \
-  --outfile stentor2-12m.gguf \
-  --outtype f16
-./llama-quantize stentor2-12m.gguf stentor2-12m-q4_0.gguf q4_0
-./llama-cli -m stentor2-12m-q4_0.gguf -p "The science of" -n 50
-```
-### Convert to ONNX
-```bash
-pip install optimum[exporters]
-optimum-cli export onnx \
-  --model StentorLabs/Stentor2-12M \
-  --task text-generation-with-past \
-  stentor2-12m-onnx/
-```
----
-## Speculative Decoding
-Stentor2-12M can serve as a fast **draft model** to accelerate inference from larger Llama-family target models.
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
-draft_model = AutoModelForCausalLM.from_pretrained(
-    "StentorLabs/Stentor2-12M", torch_dtype=torch.float16
-).to("cuda")
-target_model = AutoModelForCausalLM.from_pretrained(
-    "meta-llama/Llama-3.2-1B",
-    torch_dtype=torch.float16,
-    device_map="auto"
-)
-target_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
-inputs = target_tokenizer("Explain the concept of recursion", return_tensors="pt")
-outputs = target_model.generate(
-    **inputs,
-    assistant_model=draft_model,
-    do_sample=True,
-    max_new_tokens=100
-)
-print(target_tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
-**Important caveat:** Stentor2 uses a different vocabulary (8,064-token TokenMonster) than standard Llama models (32,000-token BPE). This vocabulary mismatch means the target model's acceptance rate may be lower than with a vocabulary-compatible draft model.
----
-## Bias, Risks & Limitations
-- **Prompt Relevance:** Outputs frequently off-topic for complex prompts.
-- **Factual Accuracy:** All factual claims should be treated as unreliable.
-- **Context Boundary:** Hard limit of 1,024 tokens.
-- **English Bias:** TokenMonster English vocabulary; other languages tokenize poorly.
-- **Training Data Bias:** Inherits biases in English-language educational web and PDF text.
-- **Hallucination:** May produce confident but fabricated content.
-- **No Alignment:** No RLHF, no DPO, no constitutional training.
-- **Tokenizer Efficiency:** 8K TokenMonster vocabulary produces more tokens per word than standard 32K BPE. This is expected and not a bug.
-- **Shared Tensor Warning:** `Removed shared tensor {'lm_head.weight'}` is expected behavior from tied word embeddings. Safe to ignore.
----
-## Related Work
-### Comparable Sub-50M Models
-| Model | Parameters | Best Perplexity | Training Data | Notes |
-|---|---|---|---|---|
-| **Stentor2-12M** (this model) | 12.3M | **26.61** | FineWeb-HQ 480M tokens | Base model, TokenMonster vocab |
-| Stentor-12M (v1) | 12.0M | 89.01 | FineWeb-Edu + Cosmopedia 200M | Baseline this model improves on |
-| Stentor-30M (v1) | 30.4M | 33.02 | FineWeb-Edu + Cosmopedia 600M | Larger v1 model |
-| TinyStories-33M | ~33M | varies | TinyStories (synthetic) | Story generation focused |
-| Pythia-14M | 14M | varies (Pile) | The Pile 300B tokens | EleutherAI scaling baseline |
-> **Comparison caveats:** Perplexity numbers are not directly comparable across models — different validation sets, vocabularies, and tokenizers all affect the number.
-### Related Research Papers
-| Paper | Relevance |
-|---|---|
-| [TinyStories](https://arxiv.org/abs/2305.07759) — Eldan & Li, 2023 | Demonstrates meaningful generation from 1M–33M parameter models |
-| [Pythia](https://arxiv.org/abs/2304.01373) — Biderman et al., 2023 | Systematic study of small model scaling |
-| [Scaling Laws](https://arxiv.org/abs/2001.08361) — Kaplan et al., 2020 | Informs token budget decisions |
-| [Chinchilla](https://arxiv.org/abs/2203.15556) — Hoffmann et al., 2022 | ~480M tokens for 12M params is compute-optimal under this analysis |
-| [RoPE](https://arxiv.org/abs/2104.09864) — Su et al., 2021 | Positional encoding used in this model |
-| [Speculative Decoding](https://arxiv.org/abs/2211.17192) — Leviathan et al., 2023 | Primary use case for a fast draft model |
-| [T5](https://arxiv.org/abs/1910.10683) — Raffel et al., 2020 | Source of NFKC text normalization approach |
----
-## Environmental Impact
-| Factor | Value |
-|---|---|
-| Hardware | 2× NVIDIA Tesla T4 |
-| Active Training Duration | **~2.1 hours** |
-| Cloud Provider | Kaggle (free tier) |
-| Compute Region | Western USA |
-| Estimated Carbon | Minimal (< 0.3 kg CO₂e estimated) |
----
-## Citation
-```bibtex
-@misc{izumoto2026stentor2_12m,
-  title        = {Stentor2-12M},
-  author       = {Kai Izumoto},
-  year         = {2026},
-  publisher    = {StentorLabs},
-  howpublished = {\url{https://huggingface.co/StentorLabs/Stentor2-12M}},
-  note         = {12.3M parameter LlamaForCausalLM base model trained on
-                  FineWeb-HQ and StenCore with a TokenMonster 8K vocabulary.
-                  Trained on 2x Tesla T4 GPUs. Apache 2.0 license.}
-}
-```
----
-## Model Card Contact
-Questions, benchmarks, or feedback: [StentorLabs@gmail.com](mailto:StentorLabs@gmail.com) or open a [discussion](https://huggingface.co/StentorLabs/Stentor2-12M/discussions).
----
-<p align="center">
-  Made with ❤️ by <a href="https://huggingface.co/StentorLabs">StentorLabs</a><br>
-  <i>Democratizing AI through accessible, efficient models</i>
-</p>