---
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- byte-level
- pretraining
- symbolic
license: apache-2.0
datasets:
- karpathy/climbmix-400b-shuffle
- openbmb/Ultra-FineWeb-L3
- dotlabs/rewrite
- nvidia/OpenMathInstruct-2
language:
- en
---

### Updates
> - 🔹 best under 90m on sub-100m on [open slm leaderboard!](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard#dotlabs-void.1)
> - 🔸 void.2 paused to train better models focusing on stronger performance and usability.
> - ▫️ step 900,000 updated with better performance, lr decayed to zero on ~10b additional bytes.

# 👋 Meet void.1

State of the art, small language model pretrained from scratch on a diverse set of high-quality texts and an internal symbolic kernel. This is the first step into a series of models designed for fine-grained understanding of abstract/symbolic reasoning on text while being grounded on english.

<video src="https://huggingface.co/dotlabs/void.1/resolve/main/teaser.mp4" autoplay loop width="100%"></video>

## Comparison

[void.1](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard#dotlabs-void.1) (24.38 II) achieves similar and often higher performance than [palmer-006](https://huggingface.co/appvoid/palmer-006) (23.33 II) across all benchmarks and sets a new standard for future models.

<img src="https://huggingface.co/dotlabs/void.1/resolve/main/comparison.png"/>

## Details


We used the revision on step 900,000 (updated with ~10B decayed to 0) for evaluations which trained for ~12 days on a single RTX PRO 6000 gpu. It trained on 131.69 billion bytes which is between 32.92B and 43.90B standard tokens, around ~38 billion bpe tokens on average. For more details on the evals and inference, please have a look at the [official notebook](https://huggingface.co/dotlabs/void.1/blob/main/evals.ipynb). For precise training details, check the training's [config file](https://huggingface.co/dotlabs/void.1/blob/main/training_config.json) and [metrics](https://huggingface.co/dotlabs/void.1/tensorboard).
```bibtex
@misc{dotlabs,
  title  = {void: bytes is all you need},
  author = {appvoid},
  year   = {2026},
  url    = {https://huggingface.co/dotlabs/void.1}
}