--- title: DataInventor emoji: 🧪 colorFrom: indigo colorTo: pink sdk: docker app_port: 7860 pinned: false hf_oauth: true hf_oauth_scopes: - contribute-repos hf_oauth_expiration_minutes: 120 short_description: Invent training rows, curate data, chart any dataset --- # DataInventor DataInventor has three modes. - **Invent rows.** An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL. - **Curate a corpus.** Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines. - **Visualize.** Ask a question about a dataset in plain words and get a chart back, with the rows behind every bar, box or dot one click away. Use the rows you just wrote, a sample, any public Hugging Face dataset, or a CSV/JSONL file. It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers. ## How it writes rows 1. **Blueprint.** The model turns your description into topics, personas, who answers and the shapes questions take. 2. **Sampling.** Unique combinations of topic, persona, shape, difficulty, length and language. 3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking). 4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected. ## How curation works You register Hugging Face datasets as **sources** to curate, and optionally describe or add **targets** that show what the trained model should be good at. Then four steps run in order. 1. **Clean.** Gopher and C4 style heuristics (FineWeb thresholds) drop broken, boilerplate and repetitive documents, with looser rules for math and code. Any document that shares a 13-word run with a test set you name (GSM8K, MATH-500, HumanEval, MMLU, ARC and others, bundled with the package) or with a held-out set of your own is removed. 2. **Calibrate.** Exact and MinHash near-duplicates are dropped. A model grades a sample for teaching value, clarity, accuracy and density, and for relevance to your targets. A small linear scorer learns those grades and scores every document on CPU. Each source keeps its best share, ranked by a blend of quality and relevance that you set. 3. **Create.** Where a source is too small for its share of a mix, its best documents are rephrased into new ones (dialogue, tutorial, questions and answers, dense notes, textbook section, worked problems, code walkthrough). These are the styles from our [Sutra pedagogical pretraining work](https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens). Rewrites that add numbers or names the source lacks are dropped. You can also write new documents on a subject. 4. **Compose.** One prepared corpus becomes one or more mixes, each fitted to a token budget. Short sources repeat up to a cap, then get topped up with rephrased text up to a synthetic cap. The default shares come from our [1B-token mixing study](https://huggingface.co/blog/codelion/optimal-dataset-mixing): 50% textbook-style, 30% web, 20% educational web. That result was measured at 70M parameters, so treat it as a starting point. The hosted run works on about 200 documents per source and returns the kept and created documents with a report. The runner does the same on every document: ```bash pip install 'datainventor[full] @ https://huggingface.co/spaces/codelion/data-inventor/resolve/main/dist/datainventor-0.2.2-py3-none-any.whl' export DI_API_KEY=... # key for the endpoint named in the recipe datainventor init recipe.yaml # or download one from the app datainventor preview recipe.yaml # every step on a sample datainventor run recipe.yaml --out ./curated --procs 32 --workers 64 ``` It writes `curated//`, `synthetic//` and `mixes//` as JSONL shards, plus `REPORT.md`. Interrupted runs resume from the last finished stage or source. The endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama or a hosted API. Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set `calibrate.scorer: llm` to grade every document with the model. ## How Visualize works It is built for small datasets first: the 25 to 200 rows a hosted run writes. Every row stays in memory, so each chart is an exact count over all of them. - **At a glance.** As soon as data loads you see two or three charts that need no model: rows per category, and how long the texts are in each. - **Ask.** A model turns your question into a chart plan. If an existing column answers it (area, language, length), the chart comes back in a couple of seconds. If it needs something no column has, such as tone, intent or whether a reply gives steps, the model reads the rows in small parallel batches and fills a new column. A regex handles literal marks like links or numbered lists without a model. - **Drill down.** Click a bar, box, dot or legend entry to read the rows behind it, download them, or ask a follow-up about just those rows. - **Reuse.** Columns the model made stay with the dataset, so follow-up questions reuse them instead of reading every row again. - **Honest numbers.** The write-up uses only the chart's exact numbers, says when a group is too small to read much into, and the page notes which columns a model assigned. ## How it compares | | [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) (Adaption) | [Curation Studio](https://www.datologyai.com/) (DatologyAI) | DataInventor | |---|---|---|---| | Job | Training rows from a plain-language goal | Curating pretraining data | Both | | Access | Hosted | Enterprise, deployed in your cloud | Open source: this Space plus a CLI | | Model endpoint | Theirs | Theirs | Any OpenAI-compatible endpoint | | Published quality evaluation | Yes, their own | Yes, their own | Not yet | ## Run the app yourself ```bash pip install -r requirements.txt cd app && DI_API_KEY=... uvicorn main:app --port 7860 ``` The model endpoint is fixed by the host, so visitors never enter a key or a URL. Configure it with environment variables: | Variable | Default | What it does | |---|---|---| | `DI_API_KEY` | `HF_TOKEN` | Key for the endpoint | | `DI_BASE_URL` | `https://router.huggingface.co/v1` | Any OpenAI-compatible base URL | | `DI_MODEL` | `openai/gpt-oss-20b:deepinfra` | Model name | | `DI_DISABLE_THINKING` | `0` | Set `1` to turn thinking off for reasoning models. The default model does not think, so the setting is off | | `DI_WORKERS` | `4` | Parallel calls (1 to 8) | | `MAX_ROWS` | `200` | Rows per run | | `MAX_RUNNING` | `2` | Runs at once | To use your own model, clone the Space and set `DI_BASE_URL`, `DI_MODEL` and `DI_API_KEY` as Space secrets and variables. Not affiliated with Adaption Labs.