vectabass-618m-26.4bt-mdlm-converted

An autoregressive (AR) model converted to a masked diffusion language model (MDLM). Size N4 of the Phase 1 conversion set (527.7M non-embedding parameters).

It starts from VectaBass/vectabass-618m-26.5bt-base-cooled, cell N4-D5 of the AR square, after that model's cooldown. It was then trained as an MDLM for 26.39B tokens (T = 50 N). The conversion set has one model per size, N1 to N6. Each is the best of three cooled AR parents at that size, by final negative ELBO on our held-out canon.

This is a base model trained for a scaling study. It is not instruction tuned or safety tuned.

Use

import torch
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-618m-26.4bt-mdlm-converted")
model = AutoModel.from_pretrained("VectaBass/vectabass-618m-26.4bt-mdlm-converted", trust_remote_code=True, dtype=torch.bfloat16).cuda()
ids = tok("The Senedd is", return_tensors="pt")["input_ids"]
out = model.diffusion_generate(ids, order='confidence', argmax=True, presence_penalty=4.0, t_start=0.0, schedule='linear', ban_eot=True, gen_length=256, steps=256, seed=0)
print(tok.decode(out[0]))

trust_remote_code loads modeling_vectabass.py from this repo. The network is the parent's Llama-like stack with exclusive self-attention (XSA), made bidirectional. model(ids).logits gives, at each position, the distribution of the clean token. Put the mask token (id 50283) where a token is unknown. diffusion_generate appends gen_length masks to the prompt and fills them over steps forward passes. argmax=True with presence_penalty=4.0 and one token per step is the setting we recommend: see Sampling. Run it in bf16, as trained.

Sampling

The sampling study tested the N6 conversion of this set (VectaBass/vectabass-1.73b-80bt-mdlm-converted), which scored best of the three models tested. This size was not tested. The table below is for the N6 conversion.

We compared about 30 samplers on 32 held-out document starts (32-token prompt, 256 tokens generated, one seed). Each continuation was scored by Qwen3-8B-Base in bits per UTF-8 byte (lower reads more fluently, but very low means repetition), by its share of distinct bigrams and by its share of repeated 4-grams. Numbers below are for the N6 conversion of this set.

Sampler Bits per byte Distinct bigrams Repeated 4-grams
real continuation 0.66 0.80 7%
our AR model of the same size (N6-D6), top-p 0.95 0.69 0.75 9%
confidence, argmax, presence penalty 4 incl. prompt (the default above) 0.59 0.81 10%
same, no penalty 0.13 0.19 78%
random order, exact ancestral sampling 1.43 0.79 10%

What we found:

  • Without a penalty the confidence order loops. Subtracting 4 from the logit of every token already in the canvas or the prompt forces it to say something new. Counting the prompt matters: without it, short prompts are copied in a loop.
  • Penalty 3 still repeats (distinct bigrams 0.63). Penalty 6 is less fluent (0.84 bits per byte). 4 is the tested value.
  • It is a heuristic, not exact sampling: it biases against repeated words. Lists, tables and code, where repetition is correct, were not tested, and lists still repeat.
  • Keep one token per step (steps = gen_length). 4 tokens per step costs quality (0.93 bits per byte, distinct bigrams 0.59).
  • Exact ancestral sampling draws from the model's own distribution and is much less fluent than the AR model. Temperature, top-k, top-p and min-p trade fluency for diversity and never reach real text.
  • Facts are often wrong. One seed and 32 prompts: standard errors on bits per byte are about 0.05 to 0.06.

Conversion recipe

Parent VectaBass/vectabass-618m-26.5bt-base-cooled, AR square N4-D5, cooled, 26.46B tokens, canon cross-entropy 2.0352
Attention bidirectional from the first step
Targets unshifted: each position predicts its own clean token
Objective MDLM: linear alpha_t = 1 - t, loss is the continuous-time negative ELBO (1/t-weighted masked cross-entropy over all positions)
Tokens T = 26,390,560,768 (50 N)
Warmup linear from 0 over 53M tokens (0.1 N), then constant
Peak LR 0.00148
Decay fork at 0.8 T, then a 1-sqrt decay to zero
Weight decay none
Optimizer fresh state, the parent's is not reused. Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95)
Mask token row the parent's row plus Gaussian noise of 0.1 times the embedding standard deviation, in the fp32 master weights as well as the bf16 copy
Data the phase-1-release view, windows replayed in the parent's order
Batch 524,288 tokens
Precision bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits
Hardware 1 x NVIDIA GH200 120GB

See DATA.md for every bin, its share and its licence.

Model

Non-embedding parameters 527,708,160
Total parameters 617,852,928 (embeddings tied)
Width, layers 1792, 15
Heads 28 query, 14 key-value, dim 64
Feed-forward SwiGLU, 4752
Norm RMSNorm, pre-norm
Positions RoPE, base 10000, context 2048
Attention bidirectional, GQA 2:1, XSA
Mask token id 50283
Vocabulary 50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding

Evaluation

Negative ELBO on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens), 16 mask draws per sequence: 2.4316 ± 0.0025 nats per token (± one standard error). The negative ELBO is an upper bound on the model's negative log-likelihood, so it can be set against the parent's cross-entropy on the same canon as a bound, not as an equal measure.

This repo's weights reproduce the training code's logits bitwise on masked canon rows. An independent one-draw estimate here, under PyTorch SDPA (training used FlashAttention-4), gives 2.4306 ± 0.0099.

Zero-shot benchmarks are not measured for the MDLM models.

Limitations

  • A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
  • Generated text is less fluent than an AR model's of the same size without the recommended sampler. See Sampling.
  • Context is 2048 tokens, prompt and generation together.
  • Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
  • No safety tuning and no filtering of outputs.

Provenance

Run contract conversion-v2--N4-D5-cooled-50N-mask-fix (revision 24ac85707ba9), from conversion trunk conversion-v2--N4-D5-cooled-mask-fix-trunk, from AR contract ar-square--N4-D5. Source checkpoint sha256 57a3ae11125dfbdedcd5fd6952352d8d4fe03541e14904148599fd8db4c8c5c5. See provenance.json.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Citation

Paper forthcoming. Until then, cite this repository.

Downloads last month
32
Safetensors
Model size
0.6B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for VectaBass/vectabass-618m-26.4bt-mdlm-converted

Finetuned
(1)
this model

Datasets used to train VectaBass/vectabass-618m-26.4bt-mdlm-converted

Collection including VectaBass/vectabass-618m-26.4bt-mdlm-converted