Feature Extraction
Transformers
Safetensors
vectabass
scaling-laws
uk-sovereign-ai
diffusion
masked-diffusion
mdlm
custom_code

vectabass-59m-5.01bt-mdlm-cooled

Paper cell N0-D3 of the MDLM addendum (27.3M non-embedding parameters).

A 59M-parameter masked diffusion language model (MDLM; 27.3M non-embedding), trained from scratch on 5.01B tokens. It is one cell of the Phase 1 MDLM addendum: one extra rung, N0 (27.3M non-embedding parameters), below the MDLM square's ladder of 100M to 1.6B, trained to several token budgets with the same recipe rules and data, so the scaling law can be checked at a smaller size. A matching autoregressive (AR) addendum is released separately.

This is a base model trained for a scaling study. It is not instruction tuned or safety tuned.

Use

import torch
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-59m-5.01bt-mdlm-cooled")
model = AutoModel.from_pretrained("VectaBass/vectabass-59m-5.01bt-mdlm-cooled", trust_remote_code=True, dtype=torch.bfloat16).cuda()
ids = tok("The Senedd is", return_tensors="pt")["input_ids"]
out = model.diffusion_generate(ids, order='confidence', argmax=True, presence_penalty=4.0, t_start=0.0, schedule='linear', ban_eot=True, gen_length=256, steps=256, seed=0)
print(tok.decode(out[0]))

trust_remote_code loads modeling_vectabass.py from this repo. The network is the AR models' Llama-like stack with exclusive self-attention (XSA), made bidirectional. model(ids).logits gives, at each position, the distribution of the clean token; put the mask token (id 50283) where a token is unknown. diffusion_generate appends gen_length masks to the prompt and fills them over steps forward passes. order="random" unmasks in random order, the reverse process the model was trained for. order="confidence" is MaskGIT-style: it commits the most confident positions first. argmax=True with presence_penalty=4.0 and one token per step is the setting we recommend: see Sampling. Run it in bf16, as trained.

Sampling

We compared about 30 samplers on 32 held-out document starts (32-token prompt, 256 tokens generated, one seed). Each continuation was scored by Qwen3-8B-Base in bits per UTF-8 byte (lower reads more fluently, but very low means repetition), by its share of distinct bigrams and by its share of repeated 4-grams. Numbers below are for cell N6-D5.

Sampler Bits per byte Distinct bigrams Repeated 4-grams
real continuation 0.66 0.80 7%
confidence, argmax, presence penalty 4 incl. prompt (the default above) 0.64 0.74 18%
same, no penalty 0.15 0.16 80%
random order, exact ancestral sampling 1.39 0.66 22%

What we found:

  • Without a penalty the confidence order loops. Subtracting 4 from the logit of every token already in the canvas or the prompt forces it to say something new. Counting the prompt matters: without it, short prompts are copied in a loop.
  • Penalty 3 still repeats. Penalty 6 is less fluent than real text. 4 is the tested value.
  • It is a heuristic, not exact sampling: it biases against repeated words. Lists, tables and code, where repetition is correct, were not tested, and lists still repeat.
  • Keep one token per step (steps = gen_length). 4 tokens per step costs quality (0.73 bits per byte, distinct bigrams 0.49).
  • Exact ancestral sampling draws from the model's own distribution and is much less fluent than our AR model of the same size. Temperature, top-k, top-p and min-p trade fluency for diversity and never reach real text.
  • Facts are often wrong. One seed and 32 prompts: standard errors on bits per byte are about 0.06.

Model

Non-embedding parameters 27,279,360
Total parameters 59,473,920 (embeddings tied)
Width, layers 640, 6
Heads 10 query, 5 key-value, dim 64
Feed-forward SwiGLU, 1728
Norm RMSNorm, pre-norm
Positions RoPE, base 10000, context 2048
Attention bidirectional, GQA 2:1, XSA
Mask token id 50283
Vocabulary 50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding

Training

Objective MDLM: linear alpha_t = 1 - t, loss is the continuous-time negative ELBO (1/t-weighted masked cross-entropy over all positions)
Tokens 5,014,290,432 (184 per non-embedding parameter)
Batch 393,216 tokens
Optimizer Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95)
Peak LR 0.00255
Warmup 39M tokens
Schedule warmup, constant, then a 1-sqrt cooldown to 0 over the last 20%
Weight decay 0.136 on matrices
Precision bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits
Hardware 1 x NVIDIA GH200 120GB

Warmup is pinned to 100 steps (39.3M tokens), a floor for this small size: the recipe rule gives 69.4 steps. Batch, LR and weight decay follow the rule.

The data is the phase-1-release view: 47 bins of openly licensed text, about 96B unique tokens, with UK public-sector, parliamentary and Celtic-language sources. See DATA.md for every bin, its share and its licence.

Evaluation

Negative ELBO on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens), 16 mask draws per sequence: 3.2341 ± 0.0026 nats per token (± one standard error). The negative ELBO is an upper bound on the model's negative log-likelihood, so it can be set against an AR model's cross-entropy on the same canon as a bound, not as an equal measure.

This repo's weights reproduce the training code's logits bitwise on masked canon rows. An independent one-draw estimate here, under PyTorch SDPA (training used FlashAttention-4), gives 3.2323 ± 0.0103.

Zero-shot benchmarks are not measured for the MDLM models.

Limitations

  • A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
  • Generated text is less fluent than an AR model's of the same size. See Sampling.
  • Context is 2048 tokens, prompt and generation together.
  • Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
  • No safety tuning and no filtering of outputs.

Provenance

Run contract mdlm-addendum--N0-D3-wu (revision a337c80e1e13). Source checkpoint sha256 8a9a4026b639cd35e561e7fa4821f3a98ce30747db3c3319eff76e095d65fd0d. See provenance.json.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Citation

Paper forthcoming. Until then, cite this repository.

Downloads last month
34
Safetensors
Model size
59.5M params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train VectaBass/vectabass-59m-5.01bt-mdlm-cooled

Collection including VectaBass/vectabass-59m-5.01bt-mdlm-cooled