Instructions to use VectaBass/vectabass-618m-26.4bt-mdlm-converted with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use VectaBass/vectabass-618m-26.4bt-mdlm-converted with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="VectaBass/vectabass-618m-26.4bt-mdlm-converted", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("VectaBass/vectabass-618m-26.4bt-mdlm-converted", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
vectabass-618m-26.4bt-mdlm-converted
An autoregressive (AR) model converted to a masked diffusion language model (MDLM). Size N4 of the Phase 1 conversion set (527.7M non-embedding parameters).
It starts from VectaBass/vectabass-618m-26.5bt-base-cooled, cell N4-D5 of the AR square, after that model's cooldown. It was then trained as an MDLM for 26.39B tokens (T = 50 N). The conversion set has one model per size, N1 to N6. Each is the best of three cooled AR parents at that size, by final negative ELBO on our held-out canon.
This is a base model trained for a scaling study. It is not instruction tuned or safety tuned.
Use
import torch
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-618m-26.4bt-mdlm-converted")
model = AutoModel.from_pretrained("VectaBass/vectabass-618m-26.4bt-mdlm-converted", trust_remote_code=True, dtype=torch.bfloat16).cuda()
ids = tok("The Senedd is", return_tensors="pt")["input_ids"]
out = model.diffusion_generate(ids, order='confidence', argmax=True, presence_penalty=4.0, t_start=0.0, schedule='linear', ban_eot=True, gen_length=256, steps=256, seed=0)
print(tok.decode(out[0]))
trust_remote_code loads modeling_vectabass.py from this repo. The network is the parent's Llama-like stack with exclusive self-attention (XSA), made bidirectional. model(ids).logits gives, at each position, the distribution of the clean token. Put the mask token (id 50283) where a token is unknown. diffusion_generate appends gen_length masks to the prompt and fills them over steps forward passes. argmax=True with presence_penalty=4.0 and one token per step is the setting we recommend: see Sampling. Run it in bf16, as trained.
Sampling
The sampling study tested the N6 conversion of this set (VectaBass/vectabass-1.73b-80bt-mdlm-converted), which scored best of the three models tested. This size was not tested. The table below is for the N6 conversion.
We compared about 30 samplers on 32 held-out document starts (32-token prompt, 256 tokens generated, one seed). Each continuation was scored by Qwen3-8B-Base in bits per UTF-8 byte (lower reads more fluently, but very low means repetition), by its share of distinct bigrams and by its share of repeated 4-grams. Numbers below are for the N6 conversion of this set.
| Sampler | Bits per byte | Distinct bigrams | Repeated 4-grams |
|---|---|---|---|
| real continuation | 0.66 | 0.80 | 7% |
| our AR model of the same size (N6-D6), top-p 0.95 | 0.69 | 0.75 | 9% |
| confidence, argmax, presence penalty 4 incl. prompt (the default above) | 0.59 | 0.81 | 10% |
| same, no penalty | 0.13 | 0.19 | 78% |
| random order, exact ancestral sampling | 1.43 | 0.79 | 10% |
What we found:
- Without a penalty the confidence order loops. Subtracting 4 from the logit of every token already in the canvas or the prompt forces it to say something new. Counting the prompt matters: without it, short prompts are copied in a loop.
- Penalty 3 still repeats (distinct bigrams 0.63). Penalty 6 is less fluent (0.84 bits per byte). 4 is the tested value.
- It is a heuristic, not exact sampling: it biases against repeated words. Lists, tables and code, where repetition is correct, were not tested, and lists still repeat.
- Keep one token per step (
steps = gen_length). 4 tokens per step costs quality (0.93 bits per byte, distinct bigrams 0.59). - Exact ancestral sampling draws from the model's own distribution and is much less fluent than the AR model. Temperature, top-k, top-p and min-p trade fluency for diversity and never reach real text.
- Facts are often wrong. One seed and 32 prompts: standard errors on bits per byte are about 0.05 to 0.06.
Conversion recipe
| Parent | VectaBass/vectabass-618m-26.5bt-base-cooled, AR square N4-D5, cooled, 26.46B tokens, canon cross-entropy 2.0352 |
| Attention | bidirectional from the first step |
| Targets | unshifted: each position predicts its own clean token |
| Objective | MDLM: linear alpha_t = 1 - t, loss is the continuous-time negative ELBO (1/t-weighted masked cross-entropy over all positions) |
| Tokens | T = 26,390,560,768 (50 N) |
| Warmup | linear from 0 over 53M tokens (0.1 N), then constant |
| Peak LR | 0.00148 |
| Decay | fork at 0.8 T, then a 1-sqrt decay to zero |
| Weight decay | none |
| Optimizer | fresh state, the parent's is not reused. Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95) |
| Mask token row | the parent's row plus Gaussian noise of 0.1 times the embedding standard deviation, in the fp32 master weights as well as the bf16 copy |
| Data | the phase-1-release view, windows replayed in the parent's order |
| Batch | 524,288 tokens |
| Precision | bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits |
| Hardware | 1 x NVIDIA GH200 120GB |
See DATA.md for every bin, its share and its licence.
Model
| Non-embedding parameters | 527,708,160 |
| Total parameters | 617,852,928 (embeddings tied) |
| Width, layers | 1792, 15 |
| Heads | 28 query, 14 key-value, dim 64 |
| Feed-forward | SwiGLU, 4752 |
| Norm | RMSNorm, pre-norm |
| Positions | RoPE, base 10000, context 2048 |
| Attention | bidirectional, GQA 2:1, XSA |
| Mask token | id 50283 |
| Vocabulary | 50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding |
Evaluation
Negative ELBO on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens), 16 mask draws per sequence: 2.4316 ± 0.0025 nats per token (± one standard error). The negative ELBO is an upper bound on the model's negative log-likelihood, so it can be set against the parent's cross-entropy on the same canon as a bound, not as an equal measure.
This repo's weights reproduce the training code's logits bitwise on masked canon rows. An independent one-draw estimate here, under PyTorch SDPA (training used FlashAttention-4), gives 2.4306 ± 0.0099.
Zero-shot benchmarks are not measured for the MDLM models.
Limitations
- A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
- Generated text is less fluent than an AR model's of the same size without the recommended sampler. See Sampling.
- Context is 2048 tokens, prompt and generation together.
- Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
- No safety tuning and no filtering of outputs.
Provenance
Run contract conversion-v2--N4-D5-cooled-50N-mask-fix (revision 24ac85707ba9), from conversion trunk conversion-v2--N4-D5-cooled-mask-fix-trunk, from AR contract ar-square--N4-D5. Source checkpoint sha256 57a3ae11125dfbdedcd5fd6952352d8d4fe03541e14904148599fd8db4c8c5c5. See provenance.json.
Acknowledgements
The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].
Citation
Paper forthcoming. Until then, cite this repository.
- Downloads last month
- 32
Model tree for VectaBass/vectabass-618m-26.4bt-mdlm-converted
Base model
VectaBass/vectabass-618m-26.5bt-base-cooled