Text Generation
Transformers
Safetensors
vectabass
scaling-laws
uk-sovereign-ai
custom_code

vectabass-1.03b-2.18bt-base-cooled

Paper cell N5-D2 of the AR square (919.4M non-embedding parameters).

A 1029M-parameter base language model (919.4M non-embedding), trained on 2.18B tokens. It is one cell of the Phase 1 AR square: one model ladder from 100M to 1.6B non-embedding parameters, each size trained to several token budgets, released together so the scaling law can be checked.

This is a base model. It is not instruction tuned or safety tuned.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-1.03b-2.18bt-base-cooled")
model = AutoModelForCausalLM.from_pretrained("VectaBass/vectabass-1.03b-2.18bt-base-cooled", trust_remote_code=True, dtype=torch.bfloat16)
ids = tok("The Senedd is", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0]))

trust_remote_code loads modeling_vectabass.py from this repo. The network is Llama-like plus exclusive self-attention (XSA), which Llama does not have. Run it in bf16, as trained.

Model

Non-embedding parameters 919,351,296
Total parameters 1,028,812,800 (embeddings tied)
Width, layers 2176, 18
Heads 34 query, 17 key-value, dim 64
Feed-forward SwiGLU, 5648
Norm RMSNorm, pre-norm
Positions RoPE, base 10000, context 2048
Attention causal, GQA 2:1, XSA
Vocabulary 50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding

Training

Tokens 2,182,742,016 (2 per non-embedding parameter)
Batch 458,752 tokens
Optimizer Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95)
Peak LR 0.00125
Warmup 437M tokens
Schedule warmup, constant, then a 1-sqrt cooldown to 0 over the last 20%
Weight decay 0.0507 on matrices
Precision bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits
Hardware 4 x NVIDIA GH200 120GB

The data is the phase-1-release view: 47 bins of openly licensed text, about 96B unique tokens, with UK public-sector, parliamentary and Celtic-language sources. See DATA.md for every bin, its share and its licence.

Evaluation

Validation loss on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens): 2.3236 nats per token.

This repo's weights reproduce the training code's logits bitwise and score 2.3237 here under PyTorch SDPA (training used FlashAttention-4).

Zero-shot benchmark scores for this model are not measured yet.

Limitations

  • A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
  • Context is 2048 tokens.
  • Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
  • No safety tuning and no filtering of outputs.

Provenance

Run contract ar-square--N5-D2 (revision 311bdffe7a2e). Source checkpoint sha256 b9197ab5694ae2df57fdf7551d088d6ec2dee7e5852245f2802fb8fbf4057c5f. See provenance.json.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Citation

Paper forthcoming. Until then, cite this repository.

Downloads last month
42
Safetensors
Model size
1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train VectaBass/vectabass-1.03b-2.18bt-base-cooled

Collection including VectaBass/vectabass-1.03b-2.18bt-base-cooled