BeeNara 🐝 v0.3

Put a document in the right folder, or admit that none fits.

BeeNara reads a document together with your own list of folder names and returns one of them, or "none fits". Its confidence is calibrated. When it is not sure enough, it tells you to ask the user instead of guessing. It is the category decider of PollySort, a local, bilingual document archivist.

For long folder lists (more than 9 folders) see the larger BeeNara Big, which reads the whole folder tree.

What is new in v0.3

• Much better on folder trees it has never seen. On a new benchmark with unseen folder trees, unseen names and documents written by a different model, accuracy rose from 69.2 % (v0.2) to 83.8–85.2 %.
• A stricter rule for sorting on its own. Besides the conformal rule of v0.2, v0.3 ships risk thresholds: BeeNara sorts a document automatically only if, with 90 % confidence, at most 1 % of automatically sorted documents end up in a wrong folder (Learn-then-Test). On the new benchmark it sorts 38–39 % of the tasks with 3–9 folders this way, and 0.9 % of those were wrong. v0.2 could not be certified for this target at all (0 %).
• Confidence that ranks errors. The AUROC for telling right from wrong answers rose from 0.66 to 0.91–0.93 on the v0.2 benchmark. This is what makes the risk rule possible.
• Real OCR text in training: pages were rendered, degraded (scan, phone photo, skew) and read with Tesseract.
• Long documents keep their end. The input keeps the start and the last 30 % of a long document, where signatures, totals and closing lines are.

Key points

• Any folder names, no retraining. Folder names are plain text, like "Rechnungen", "Tax 2025" or "Bank statements". You can add a short description to each.
• Knows when nothing fits. It recognizes 93.5 % of documents whose folder is missing from the list (v0.2 benchmark), and when it says "none fits", it is right 92 % of the time.
• Sorts only when it is sure. Two rules are built in, see below. With the default rule it sorts 69 % of the v0.2 benchmark on its own.
• Small and fast on a CPU. It is a 332 MB ONNX file, with no GPU and no PyTorch: about 0.08 s per document on a desktop CPU with 4 threads, about 0.2–0.3 s on a laptop.
• German and English. Documents and folder names can be in either language, or mixed.
• Private by design. It was trained only on synthetic documents, with no user data. It runs fully offline.

What it is good for, and what not

Good for	Not for
Sorting letters, invoices, contracts and statements into a user's folders	Decisions with legal, medical or financial consequences
Choosing between 3 and 9 folders, or "none"	Long lists of dozens or hundreds of folders: use BeeNara Big
Laptops and offline tools where a local LLM is too slow	Languages other than German and English
Pipelines that need an "ask the human" signal	Extracting fields such as amounts or dates

Quickstart

pip install onnxruntime tokenizers numpy huggingface_hub
hf download Kwokou/BeeNara beenara.py --local-dir .

from beenara import BeeNara

bn = BeeNara.from_pretrained("Kwokou/BeeNara")          # downloads onnx/ and tokenizer/, 366 MB

d = bn.decide(
    "Rechnung Nr. 4711\nBetrag: 119,00 EUR, zahlbar bis 30.10.2026.",
    ["Rechnungen", "Verträge", "Arztbriefe"],
    descriptions={"Rechnungen": "Rechnungen von Lieferanten"},   # optional, helps with unusual names
    rule="risk",                                                  # or "conformal" (default, as in v0.2)
)
d.category        # 'Rechnungen', or None when the user should decide
d.ask             # False: sure enough to sort automatically
d.distribution    # e.g. {'Rechnungen': 0.97, 'Verträge': 0.004, 'Arztbriefe': 0.002, '__none__': 0.024}

language="en" switches the internal question to English. The document language does not matter for the categories.

The two rules. rule="conformal" (default, unchanged from v0.2) sorts when the conformal prediction set at α = 0.1 holds exactly one folder. rule="risk" sorts only when the calibrated probability of the top folder reaches the risk threshold of its list size; this gives the "at most 1 % wrong" guarantee above. v0.3 has a risk threshold for lists of 3–9 folders. For other list sizes, rule="risk" always asks.

The previous version stays available under the tag v0.2:

from huggingface_hub import snapshot_download
bn_v02 = BeeNara(snapshot_download("Kwokou/BeeNara", revision="v0.2", allow_patterns=["onnx/*", "tokenizer/*"]))

Results

All numbers come from the same evaluation pipeline for v0.3 and v0.2, on the PyTorch weights unless noted. v0.3 was trained twice with different seeds; this repository ships seed 1.

v0.2 benchmark: the unchanged benchmark of v0.2 (byte-identical file), 1,200 decision tasks on 600 held-out documents, half of them with folder names the model never saw, 25.8 % with no fitting folder.

Metric	v0.3, seed 1 (shipped)	v0.3, seed 2	v0.2
Accuracy, folders and "none fits"	95.9 %	95.8 %	94.1 %
"None fits" recall	93.5 %	92.2 %	96.8 %
"None fits" precision	91.8 %	92.2 %	84.5 %
Wrongly said "none fits"	2.9 %	2.7 %	6.2 %
Calibration error (ECE)	0.038	0.034	0.039
AUROC, right vs. wrong answer	0.93	0.91	0.66
Sorted automatically, conformal rule	69 %	70 %	63 %

The shipped ONNX file scores 96.1 % on this benchmark (PyTorch: 95.9 %) and makes the same choice as the PyTorch weights on all 64 parity-checked tasks. It was calibrated on itself after quantization.

New benchmark (v0.3): 1,200 tasks from a held-out "world": folder trees, folder names and documents that were never used for training, written by a different teacher model (gpt-oss-120b instead of Qwen3.8-27B). The model sees the target folder together with nearby folders of the same tree.

Metric	v0.3, seed 1	v0.3, seed 2	v0.2
Accuracy	83.8 %	85.2 %	69.2 %
Calibration error (ECE)	0.074	0.073	0.275
Wrongly said "none fits"	14.1 %	9.9 %	26.6 %
3–9 folders: sorted automatically, risk rule	39.2 %	37.7 %	0 %
… of which wrong	0.9 %	0.9 %	–
"None fits" when a similar neighbour folder exists	91.5 %	92.7 %	95.1 %

Independent test, 70 documents from a different generator (PollySort's test projects), 8 folders, conformal rule. In C2 the right folder is missing; C3 uses English names in shuffled order.

	C1 German	C2 "none fits"	C3 English	Sorted automatically in C1
Names only	100 %	92.9 %	100 %	85.7 %, all correct
With descriptions	100 %	94.3 %	100 %	100 %, all correct

In C2, 4 of 70 documents were sorted automatically into a wrong folder, with and without descriptions. v0.2 had the same count with names only.

How it works

BeeNara is a cross-encoder with option markers. The question, every folder name and the document go into one sequence and through one forward pass:

[CLS] question [SEP] [MASK] folder 1 [MASK] folder 2 … [SEP] document start … document end [SEP]

• Each folder is scored at its own [MASK].
• A separate head reads [CLS] plus the shape of the folder scores and gives p("none fits").
• Both combine into one distribution: p(none) = σ(e), p(folder i) = (1 − p(none)) · softmax(s)ᵢ.

Part	Choice
Encoder	mmBERT-base, a multilingual ModernBERT: 22 layers, 768 dims, 256k vocabulary
Head	one Transformer layer, a folder scorer and a "none" scorer
Input	up to 512 tokens; 192 for question and folders; of the document, the start and the last 30 %
Shipped file	ONNX, weights in 8 bit (MatMulNBits, block 32), embeddings in INT8, 332 MB
Calibration	temperature scaling; split-conformal set at α = 0.1; risk threshold per list size (Learn-then-Test, ≤ 1 % wrong among automatic decisions, δ = 0.1), fitted on 3,000 held-out tasks

Training

• Data. Only synthetic documents, no user data:
  • 28,524 template documents from v0.1/v0.2, 20 business types, with simulated OCR noise;
  • 15,813 new documents written by Qwen3.8-27B for 3,000 synthetic folder trees, and 4,517 rewritten template documents, all checked against the facts they were built from;
  • real OCR text: about 20 % of the pages were rendered, degraded and read with Tesseract;
  • folder trees of 10–500 folders in German, English and mixed, for private households, freelancers, small firms, construction offices, associations and law offices;
  • about 20 % English mail-triage tasks from Open-Jev (CC0).
• Episodes. Every step draws a new task for each document: 3–9 folders from its tree or from the catalogue, in random order, with hard distractors and similar neighbour folders; in 25 % of the tasks the right folder is removed.
• Kept apart by tests. Three disjoint sets of names and a separate held-out world of 4,327 documents in 720 trees for selection, calibration and the new benchmark. Overlap was checked on normalized names, substrings and descriptions; near-duplicate and 13-gram checks found no leakage.
• Recipe of the shipped model. Like v0.2, but without label smoothing. We trained eight recipes with two seeds each. Some with label smoothing were up to 1.6 points more accurate on the new benchmark, but none of them could be certified for the risk target: label smoothing compresses the confidence of the surest answers, so safe and unsafe cases no longer separate. This variant is not worse in accuracy than the standard recipe (paired bootstrap, two seeds) and is the only one that sorts automatically under the risk rule.
• Setup: 3 epochs at batch 16, AdamW with encoder at 2.5e-5 and head at 1e-4, 5 % warmup, then cosine; bf16; frozen embeddings; best checkpoint by validation NLL (step 6,000).
• Hardware: one NVIDIA RTX A6000 (48 GB), about 35 minutes per run.

Environmental impact

We did not have a power meter. The numbers are an upper-bound estimate following Lacoste et al. 2019.

	Compute	Energy, at most	CO₂eq, at most
This model (one training run)	about 0.6 h on one GPU	0.2 kWh	0.1 kg
The whole v0.3 project	about 290 GPU hours on two RTX A6000 (teacher models ~70 h, training of all small and Big models ~200 h, calibration and evaluation ~15 h) over 9 days	170 kWh	65 kg

Assumptions: at most 300 W per GPU while in use, plus 350 W for the rest of the machine over the whole 9 days; grid 0.38 kg CO₂eq/kWh (German mix 2023/24). Not included: the pretraining of mmBERT and of the teacher models.

Credits and inspiration

Source	What BeeNara takes from it
mmBERT (paper), JHU CLSP, MIT	the encoder weights
ModernBERT, Warner et al. 2024	the encoder architecture behind mmBERT
laya, convaiinnovations, Apache-2.0	the idea of one [MASK] marker per option, scored in one pass
GLiClass	all labels in one sequence, one forward pass
Guo et al. 2017	temperature scaling
Angelopoulos & Bates 2021	split-conformal prediction
Angelopoulos et al. 2021, Learn then Test	the risk thresholds with a finite-sample guarantee
Xia et al., Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It	the effect that made us drop label smoothing
Qwen3.8-27B, Qwen team, Apache-2.0	wrote and rewrote training documents and proposed training folder names, run locally
gpt-oss-120b, OpenAI, Apache-2.0	wrote the held-out documents for selection, calibration and benchmark, and one family of training names, run locally
Open-Jev, ZefanCai, CC0-1.0	about 20 % of the training tasks, from mailroom-control-v1
Tesseract OCR, Apache-2.0	the OCR text in training
ONNX Runtime	weight-only 8-bit quantization (MatMulNBits)

Limitations

• Automatic sorting is certified for 3–9 folders only. For other list sizes, rule="risk" always asks. With the conformal rule BeeNara still sorts, without the 1 % guarantee.
• Not for long folder lists. A pre-selection of 9 folders by an embedding model (multilingual-e5-small) followed by BeeNara reached only about 53 % on large trees. Use BeeNara Big for those.
• Missing folder with similar neighbours. It recognizes "none fits" slightly less often than v0.2 when a similar folder is present (91.5 % vs. 95.1 %), and in the independent test 4 of 70 such documents were sorted automatically into a wrong folder.
• Synthetic training data. Real scans, letterheads and handwriting are messier. Expect a gap until the model is checked on real, consented documents.
• German and English only.
• BeeNara supports a decision; it does not make one. Use it where a wrong folder can be undone.

Files

File	Content	Size
onnx/beenara.onnx	the model	332 MB
onnx/calibration.json	temperatures, conformal threshold, risk thresholds and input truncation, fitted on this file; bound to it by a SHA-256 fingerprint	< 1 KB
tokenizer/	tokenizer of mmBERT-base	34 MB
beenara.py	inference: input format, calibration, both decision rules	8 KB

License

Apache-2.0, see LICENSE. The weights derive from mmBERT-base, which is MIT-licensed; see NOTICE. Developed by Akinara as part of PollySort.

@misc{beenara2026,
  title  = {BeeNara: a calibrated, abstaining category decider for document sorting},
  author = {Akinara},
  year   = {2026},
  url    = {https://huggingface.co/Kwokou/BeeNara}
}