Instructions to use VectaBass/vectabass-59m-5.01bt-mdlm-cooled with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use VectaBass/vectabass-59m-5.01bt-mdlm-cooled with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="VectaBass/vectabass-59m-5.01bt-mdlm-cooled", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("VectaBass/vectabass-59m-5.01bt-mdlm-cooled", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
vectabass-59m-5.01bt-mdlm-cooled
Paper cell N0-D3 of the MDLM addendum (27.3M non-embedding parameters).
A 59M-parameter masked diffusion language model (MDLM; 27.3M non-embedding), trained from scratch on 5.01B tokens. It is one cell of the Phase 1 MDLM addendum: one extra rung, N0 (27.3M non-embedding parameters), below the MDLM square's ladder of 100M to 1.6B, trained to several token budgets with the same recipe rules and data, so the scaling law can be checked at a smaller size. A matching autoregressive (AR) addendum is released separately.
This is a base model trained for a scaling study. It is not instruction tuned or safety tuned.
Use
import torch
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-59m-5.01bt-mdlm-cooled")
model = AutoModel.from_pretrained("VectaBass/vectabass-59m-5.01bt-mdlm-cooled", trust_remote_code=True, dtype=torch.bfloat16).cuda()
ids = tok("The Senedd is", return_tensors="pt")["input_ids"]
out = model.diffusion_generate(ids, order='confidence', argmax=True, presence_penalty=4.0, t_start=0.0, schedule='linear', ban_eot=True, gen_length=256, steps=256, seed=0)
print(tok.decode(out[0]))
trust_remote_code loads modeling_vectabass.py from this repo. The network is the AR models' Llama-like stack with exclusive self-attention (XSA), made bidirectional. model(ids).logits gives, at each position, the distribution of the clean token; put the mask token (id 50283) where a token is unknown. diffusion_generate appends gen_length masks to the prompt and fills them over steps forward passes. order="random" unmasks in random order, the reverse process the model was trained for. order="confidence" is MaskGIT-style: it commits the most confident positions first. argmax=True with presence_penalty=4.0 and one token per step is the setting we recommend: see Sampling. Run it in bf16, as trained.
Sampling
We compared about 30 samplers on 32 held-out document starts (32-token prompt, 256 tokens generated, one seed). Each continuation was scored by Qwen3-8B-Base in bits per UTF-8 byte (lower reads more fluently, but very low means repetition), by its share of distinct bigrams and by its share of repeated 4-grams. Numbers below are for cell N6-D5.
| Sampler | Bits per byte | Distinct bigrams | Repeated 4-grams |
|---|---|---|---|
| real continuation | 0.66 | 0.80 | 7% |
| confidence, argmax, presence penalty 4 incl. prompt (the default above) | 0.64 | 0.74 | 18% |
| same, no penalty | 0.15 | 0.16 | 80% |
| random order, exact ancestral sampling | 1.39 | 0.66 | 22% |
What we found:
- Without a penalty the confidence order loops. Subtracting 4 from the logit of every token already in the canvas or the prompt forces it to say something new. Counting the prompt matters: without it, short prompts are copied in a loop.
- Penalty 3 still repeats. Penalty 6 is less fluent than real text. 4 is the tested value.
- It is a heuristic, not exact sampling: it biases against repeated words. Lists, tables and code, where repetition is correct, were not tested, and lists still repeat.
- Keep one token per step (
steps = gen_length). 4 tokens per step costs quality (0.73 bits per byte, distinct bigrams 0.49). - Exact ancestral sampling draws from the model's own distribution and is much less fluent than our AR model of the same size. Temperature, top-k, top-p and min-p trade fluency for diversity and never reach real text.
- Facts are often wrong. One seed and 32 prompts: standard errors on bits per byte are about 0.06.
Model
| Non-embedding parameters | 27,279,360 |
| Total parameters | 59,473,920 (embeddings tied) |
| Width, layers | 640, 6 |
| Heads | 10 query, 5 key-value, dim 64 |
| Feed-forward | SwiGLU, 1728 |
| Norm | RMSNorm, pre-norm |
| Positions | RoPE, base 10000, context 2048 |
| Attention | bidirectional, GQA 2:1, XSA |
| Mask token | id 50283 |
| Vocabulary | 50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding |
Training
| Objective | MDLM: linear alpha_t = 1 - t, loss is the continuous-time negative ELBO (1/t-weighted masked cross-entropy over all positions) |
| Tokens | 5,014,290,432 (184 per non-embedding parameter) |
| Batch | 393,216 tokens |
| Optimizer | Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95) |
| Peak LR | 0.00255 |
| Warmup | 39M tokens |
| Schedule | warmup, constant, then a 1-sqrt cooldown to 0 over the last 20% |
| Weight decay | 0.136 on matrices |
| Precision | bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits |
| Hardware | 1 x NVIDIA GH200 120GB |
Warmup is pinned to 100 steps (39.3M tokens), a floor for this small size: the recipe rule gives 69.4 steps. Batch, LR and weight decay follow the rule.
The data is the phase-1-release view: 47 bins of openly licensed text, about 96B unique tokens, with UK public-sector, parliamentary and Celtic-language sources. See DATA.md for every bin, its share and its licence.
Evaluation
Negative ELBO on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens), 16 mask draws per sequence: 3.2341 ± 0.0026 nats per token (± one standard error). The negative ELBO is an upper bound on the model's negative log-likelihood, so it can be set against an AR model's cross-entropy on the same canon as a bound, not as an equal measure.
This repo's weights reproduce the training code's logits bitwise on masked canon rows. An independent one-draw estimate here, under PyTorch SDPA (training used FlashAttention-4), gives 3.2323 ± 0.0103.
Zero-shot benchmarks are not measured for the MDLM models.
Limitations
- A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
- Generated text is less fluent than an AR model's of the same size. See Sampling.
- Context is 2048 tokens, prompt and generation together.
- Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
- No safety tuning and no filtering of outputs.
Provenance
Run contract mdlm-addendum--N0-D3-wu (revision a337c80e1e13). Source checkpoint sha256 8a9a4026b639cd35e561e7fa4821f3a98ce30747db3c3319eff76e095d65fd0d. See provenance.json.
Acknowledgements
The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].
Citation
Paper forthcoming. Until then, cite this repository.
- Downloads last month
- 34