Titanius-1.1-88m-base-fp16

Titanius-1.1 is an experimental decoder-only base language model. This release is the final checkpoint of a 2-billion-token base pretraining run, saved at 30,518 optimizer steps.

Release status: This is a research artifact, not an instruction-tuned or production-ready assistant. It may generate inaccurate, biased, unsafe, or otherwise undesirable text.

Training overview

Setting	Value
Training steps	30,518
Training tokens	2,000,027,648
Tokens per optimizer step	65,536
Context length	2,048 tokens
Device batch size	1 sequence
Gradient accumulation	32 steps
Optimizer	AdamW
Peak learning rate	4e-4
Final learning rate	4e-5
Warmup	2%
Schedule	Cosine decay
Gradient clipping	1.0
Final smoothed training loss	3.0181
Final held-out validation loss	3.1133
Training dataset	karpathy/climbmix-400b-shuffle

The checkpoint records the exact packed-token count. The reported validation loss comes from the final checkpoint's held-out evaluation. No standardized benchmark evaluation was recorded, so these losses should not be interpreted as downstream task scores.

Model architecture

Component	Value
Architecture	Decoder-only causal Transformer
Layers	10
Hidden size	768
Query heads	12
Key/value heads	4
Head dimension	64
MLP width	3,072
Vocabulary size	32,768
Positional encoding	Rotary embeddings (RoPE)
Attention	Grouped-query causal attention
Activation	Squared ReLU
Context window	2,048 tokens
Output logit soft cap	15

The model uses RMS normalization in the forward path, QK normalization, tied input/output embeddings, per-layer residual scaling, an input-embedding residual, and a mid-layer "backout" residual.

Parameter count

The parameter count was recalculated directly from model_step_30518.pt:

Count	Parameters
Unique trainable parameters	88,080,405
Non-embedding parameters	62,914,581
Shared token embedding/output parameters	25,165,824

The repository's "88m" label counts unique trainable parameters. The input embedding and output projection share one weight matrix. PyTorch lists that shared tensor under both state-dictionary keys, but both keys refer to the same storage. The uploaded FP16 checkpoint contains model weights and inference metadata only. Optimizer, dataloader, and other training state were removed.

Tokenizer

The model uses a custom byte-level BPE tokenizer with a vocabulary of 32,768 tokens. Numbers are pre-tokenized in groups of up to three digits. Its special tokens are:

<|bos|>
<|user_start|>
<|user_end|>
<|assistant_start|>
<|assistant_end|>

Although conversation boundary tokens exist in the vocabulary, this checkpoint is a base pretrained model and has not been instruction-tuned.

Intended use

This checkpoint is intended for:

• research and experimentation with small causal language models;
• continued pretraining or supervised fine-tuning;
• inspection of the custom architecture and training run;
• non-critical text-generation experiments.

It is not intended for autonomous decisions, high-stakes applications, factual lookup, or deployment without application-specific evaluation and safeguards.

Limitations

• No standardized benchmark, safety, bias, or memorization evaluation is reported.
• The model is not instruction-tuned and may not reliably follow prompts.
• Outputs may be incorrect, incoherent, biased, offensive, or repetitive.
• The training corpus may contain problematic, copyrighted, personal, or low-quality material inherited from its upstream sources.
• Loading and inference require the repository's custom PyTorch model code. This is not a drop-in standard Transformers architecture.

Training data

The run streams the training split of karpathy/climbmix-400b-shuffle. Users should review the upstream dataset card, composition, license information, and usage restrictions before using or redistributing this model.

Reproducibility notes

The training configuration, model dimensions, token count, final smoothed loss, and final validation loss were read from the saved checkpoint and training log. The uploaded checkpoint intentionally omits training state and contains only what the included inference code needs. Hardware and independent benchmark results are not claimed here.

Citation

If you use this model, cite the Hugging Face repository:

@misc{salem2026titanius11,
  author       = {Nadav Salem},
  title        = {Titanius-1.1-88m-base-fp16},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/NadavSalem/Titanius-1.1-88m-base-fp16}}
}