Spaces:
Running
Running
| title: DataInventor | |
| emoji: 🧪 | |
| colorFrom: indigo | |
| colorTo: pink | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| hf_oauth: true | |
| hf_oauth_scopes: | |
| - contribute-repos | |
| hf_oauth_expiration_minutes: 120 | |
| short_description: Invent training rows, curate data, chart any dataset | |
| # DataInventor | |
| DataInventor has three modes. | |
| - **Invent rows.** An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL. | |
| - **Curate a corpus.** Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines. | |
| - **Visualize.** Ask a question about a dataset in plain words and get a chart back, with the rows behind every bar, box or dot one click away. Use the rows you just wrote, a sample, any public Hugging Face dataset, or a CSV/JSONL file. | |
| It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers. | |
| ## How it writes rows | |
| 1. **Blueprint.** The model turns your description into topics, personas, who answers and the shapes questions take. | |
| 2. **Sampling.** Unique combinations of topic, persona, shape, difficulty, length and language. | |
| 3. **Writing.** One message per combination, then the reply (preference pairs get two replies and a ranking). | |
| 4. **Filters.** Cut-off or degenerate text, requests for many items at once, and near-duplicate questions are rejected. | |
| ## How curation works | |
| You register Hugging Face datasets as **sources** to curate, and optionally describe or add **targets** that show what the trained model should be good at. Then four steps run in order. | |
| 1. **Clean.** Gopher and C4 style heuristics (FineWeb thresholds) drop broken, boilerplate and repetitive documents, with looser rules for math and code. Any document that shares a 13-word run with a test set you name (GSM8K, MATH-500, HumanEval, MMLU, ARC and others, bundled with the package) or with a held-out set of your own is removed. | |
| 2. **Calibrate.** Exact and MinHash near-duplicates are dropped. A model grades a sample for teaching value, clarity, accuracy and density, and for relevance to your targets. A small linear scorer learns those grades and scores every document on CPU. Each source keeps its best share, ranked by a blend of quality and relevance that you set. | |
| 3. **Create.** Where a source is too small for its share of a mix, its best documents are rephrased into new ones (dialogue, tutorial, questions and answers, dense notes, textbook section, worked problems, code walkthrough). These are the styles from our [Sutra pedagogical pretraining work](https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens). Rewrites that add numbers or names the source lacks are dropped. You can also write new documents on a subject. | |
| 4. **Compose.** One prepared corpus becomes one or more mixes, each fitted to a token budget. Short sources repeat up to a cap, then get topped up with rephrased text up to a synthetic cap. The default shares come from our [1B-token mixing study](https://huggingface.co/blog/codelion/optimal-dataset-mixing): 50% textbook-style, 30% web, 20% educational web. That result was measured at 70M parameters, so treat it as a starting point. | |
| The hosted run works on about 200 documents per source and returns the kept and created documents with a report. The runner does the same on every document: | |
| ```bash | |
| pip install 'datainventor[full] @ https://huggingface.co/spaces/codelion/data-inventor/resolve/main/dist/datainventor-0.2.2-py3-none-any.whl' | |
| export DI_API_KEY=... # key for the endpoint named in the recipe | |
| datainventor init recipe.yaml # or download one from the app | |
| datainventor preview recipe.yaml # every step on a sample | |
| datainventor run recipe.yaml --out ./curated --procs 32 --workers 64 | |
| ``` | |
| It writes `curated/<source>/`, `synthetic/<source>/` and `mixes/<mix>/` as JSONL shards, plus `REPORT.md`. Interrupted runs resume from the last finished stage or source. The endpoint is any OpenAI-compatible server: vLLM, SGLang, Ollama or a hosted API. | |
| Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set `calibrate.scorer: llm` to grade every document with the model. | |
| ## How Visualize works | |
| It is built for small datasets first: the 25 to 200 rows a hosted run writes. Every row stays in memory, so each chart is an exact count over all of them. | |
| - **At a glance.** As soon as data loads you see two or three charts that need no model: rows per category, and how long the texts are in each. | |
| - **Ask.** A model turns your question into a chart plan. If an existing column answers it (area, language, length), the chart comes back in a couple of seconds. If it needs something no column has, such as tone, intent or whether a reply gives steps, the model reads the rows in small parallel batches and fills a new column. A regex handles literal marks like links or numbered lists without a model. | |
| - **Drill down.** Click a bar, box, dot or legend entry to read the rows behind it, download them, or ask a follow-up about just those rows. | |
| - **Reuse.** Columns the model made stay with the dataset, so follow-up questions reuse them instead of reading every row again. | |
| - **Honest numbers.** The write-up uses only the chart's exact numbers, says when a group is too small to read much into, and the page notes which columns a model assigned. | |
| ## How it compares | |
| | | [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) (Adaption) | [Curation Studio](https://www.datologyai.com/) (DatologyAI) | DataInventor | | |
| |---|---|---|---| | |
| | Job | Training rows from a plain-language goal | Curating pretraining data | Both | | |
| | Access | Hosted | Enterprise, deployed in your cloud | Open source: this Space plus a CLI | | |
| | Model endpoint | Theirs | Theirs | Any OpenAI-compatible endpoint | | |
| | Published quality evaluation | Yes, their own | Yes, their own | Not yet | | |
| ## Run the app yourself | |
| ```bash | |
| pip install -r requirements.txt | |
| cd app && DI_API_KEY=... uvicorn main:app --port 7860 | |
| ``` | |
| The model endpoint is fixed by the host, so visitors never enter a key or a URL. Configure it with environment variables: | |
| | Variable | Default | What it does | | |
| |---|---|---| | |
| | `DI_API_KEY` | `HF_TOKEN` | Key for the endpoint | | |
| | `DI_BASE_URL` | `https://huggingface.co/proxy/router.huggingface.co/v1` | Any OpenAI-compatible base URL | | |
| | `DI_MODEL` | `openai/gpt-oss-20b:deepinfra` | Model name | | |
| | `DI_DISABLE_THINKING` | `0` | Set `1` to turn thinking off for reasoning models. The default model does not think, so the setting is off | | |
| | `DI_WORKERS` | `4` | Parallel calls (1 to 8) | | |
| | `MAX_ROWS` | `200` | Rows per run | | |
| | `MAX_RUNNING` | `2` | Runs at once | | |
| To use your own model, clone the Space and set `DI_BASE_URL`, `DI_MODEL` and `DI_API_KEY` as Space secrets and variables. | |
| Not affiliated with Adaption Labs. | |