README.md

metadata

title: DataInventor
emoji: 🧪
colorFrom: indigo
colorTo: pink
sdk: docker
app_port: 7860
pinned: false
hf_oauth: true
hf_oauth_scopes:
  - contribute-repos
hf_oauth_expiration_minutes: 120
short_description: Invent training rows, curate data, chart any dataset

DataInventor

DataInventor has three modes.

• Invent rows. An open-source recreation of the Invent a Dataset workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL.
• Curate a corpus. Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines.
• Visualize. Ask a question about a dataset in plain words and get a chart back, with the rows behind every bar, box or dot one click away. Use the rows you just wrote, a sample, any public Hugging Face dataset, or a CSV/JSONL file.

It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.

How it writes rows

1. Blueprint. The model turns your description into topics, personas, who answers and the shapes questions take.
2. Sampling. Unique combinations of topic, persona, shape, difficulty, length and language.
3. Writing. One message per combination, then the reply (preference pairs get two replies and a ranking).
4. Filters. Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected.

How curation works

You register Hugging Face datasets as sources to curate, and optionally describe or add targets that show what the trained model should be good at. Then four steps run in order.

1. Clean. Gopher and C4 style heuristics (FineWeb thresholds) drop broken, boilerplate and repetitive documents, with looser rules for math and code. Any document that shares a 13-word run with a test set you name (GSM8K, MATH-500, HumanEval, MMLU, ARC and others, bundled with the package) or with a held-out set of your own is removed.
2. Calibrate. Exact and MinHash near-duplicates are dropped. A model grades a sample for teaching value, clarity, accuracy and density, and for relevance to your targets. A small linear scorer learns those grades and scores every document on CPU. Each source keeps its best share, ranked by a blend of quality and relevance that you set.
3. Create. Where a source is too small for its share of a mix, its best documents are rephrased into new ones (dialogue, tutorial, questions and answers, dense notes, textbook section, worked problems, code walkthrough). These are the styles from our Sutra pedagogical pretraining work. Rewrites that add numbers or names the source lacks are dropped. You can also write new documents on a subject.
4. Compose. One prepared corpus becomes one or more mixes, each fitted to a token budget. Short sources repeat up to a cap, then get topped up with rephrased text up to a synthetic cap. The default shares come from our 1B-token mixing study: 50% textbook-style, 30% web, 20% educational web. That result was measured at 70M parameters, so treat it as a starting point.

The hosted run works on about 200 documents per source and returns the kept and created documents with a report. The runner does the same on every document:

pip install 'datainventor[full] @ https://huggingface.co/spaces/codelion/data-inventor/resolve/main/dist/datainventor-0.2.2-py3-none-any.whl'
export DI_API_KEY=...                      # key for the endpoint named in the recipe
datainventor init recipe.yaml              # or download one from the app
datainventor preview recipe.yaml           # every step on a sample
datainventor run recipe.yaml --out ./curated --procs 32 --workers 64

It writes curated/<source>/, synthetic/<source>/ and mixes/<mix>/ as JSONL shards, plus REPORT.md. Interrupted runs resume from the last finished stage or source. The endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama or a hosted API.

Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set calibrate.scorer: llm to grade every document with the model.

How Visualize works

It is built for small datasets first: the 25 to 200 rows a hosted run writes. Every row stays in memory, so each chart is an exact count over all of them.

• At a glance. As soon as data loads you see two or three charts that need no model: rows per category, and how long the texts are in each.
• Ask. A model turns your question into a chart plan. If an existing column answers it (area, language, length), the chart comes back in a couple of seconds. If it needs something no column has, such as tone, intent or whether a reply gives steps, the model reads the rows in small parallel batches and fills a new column. A regex handles literal marks like links or numbered lists without a model.
• Drill down. Click a bar, box, dot or legend entry to read the rows behind it, download them, or ask a follow-up about just those rows.
• Reuse. Columns the model made stay with the dataset, so follow-up questions reuse them instead of reading every row again.
• Honest numbers. The write-up uses only the chart's exact numbers, says when a group is too small to read much into, and the page notes which columns a model assigned.

How it compares

	Invent a Dataset (Adaption)	Curation Studio (DatologyAI)	DataInventor
Job	Training rows from a plain-language goal	Curating pretraining data	Both
Access	Hosted	Enterprise, deployed in your cloud	Open source: this Space plus a CLI
Model endpoint	Theirs	Theirs	Any OpenAI-compatible endpoint
Published quality evaluation	Yes, their own	Yes, their own	Not yet

Run the app yourself

pip install -r requirements.txt
cd app && DI_API_KEY=... uvicorn main:app --port 7860

The model endpoint is fixed by the host, so visitors never enter a key or a URL. Configure it with environment variables:

Variable	Default	What it does
DI_API_KEY	HF_TOKEN	Key for the endpoint
DI_BASE_URL	https://huggingface.co/proxy/router.huggingface.co/v1	Any OpenAI-compatible base URL
DI_MODEL	openai/gpt-oss-20b:deepinfra	Model name
DI_DISABLE_THINKING	0	Set 1 to turn thinking off for reasoning models. The default model does not think, so the setting is off
DI_WORKERS	4	Parallel calls (1 to 8)
MAX_ROWS	200	Rows per run
MAX_RUNNING	2	Runs at once

To use your own model, clone the Space and set DI_BASE_URL, DI_MODEL and DI_API_KEY as Space secrets and variables.

Not affiliated with Adaption Labs.