# Robot-data-diff: episode-aware diffs for LeRobot datasets — no setup, no workflow **URL:** \ **Category:** Show and Tell\ **Created:** [October 16, 2026, 9:28pm UTC](https://discuss.huggingface.co/t/robot-data-diff-episode-aware-diffs-for-lerobot-datasets-no-setup-no-workflow/190685 "2026-10-16T21:28:51Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Japheth24](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@Japheth24](https://discuss.huggingface.co/u/Japheth24)\ **Post date:** [October 16, 2026, 9:28pm UTC](https://discuss.huggingface.co/t/robot-data-diff-episode-aware-diffs-for-lerobot-datasets-no-setup-no-workflow/190685/1 "2026-10-16T21:28:51Z")
Retrain a policy and it gets worse — was it the code or the data? robot-data-diff is a CLI that diffs any two revisions of a LeRobot dataset directly: local dirs or HF repo ids (v2.1 + v3.0), no tracking setup, no commit workflow — just point it at two dirs. Reports episodes added / removed / modified with per-episode stat deltas, plus an HTML report. CPU-only, no GPU, videos never downloaded. Free, Apache-2.0. Similar ideas exist (EpisodeVault), but nobody’s using them, and they make you adopt their whole track/commit workflow first. This one meets you where your data already is. Repo: [GitHub - yafetashenafi790-lgtm/robot-data-diff: Episode-aware diff for LeRobot dataset - see what changed between two dataset revisions · GitHub](https://github.com/yafetashenafi790-lgtm/robot-data-diff) pip: [Client Challenge](https://pypi.org/project/robot-data-diff/) (`pip install robot-data-diff`) Would love feedback — especially: what’s the first diff you ran, and did it surprise you? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [October 19, 2026, 6:43am UTC](https://discuss.huggingface.co/t/robot-data-diff-episode-aware-diffs-for-lerobot-datasets-no-setup-no-workflow/190685/2 "2026-10-19T06:43:14Z")
For now, I gave the public release a quick try: * * * ## The first diffs I ran I installed the published [`robot-data-diff==0.1.0` package](https://pypi.org/project/robot-data-diff/) in a Colab **CPU** runtime and tried it on local copies of two public LeRobot datasets: - **v2.1:** [`rshoff/lerobot`](https://huggingface.co/datasets/rshoff/lerobot), 2 episodes. - **v3.0:** [`lerobot/pusht`](https://huggingface.co/datasets/lerobot/pusht), 206 episodes. For each dataset, I first ran an **unchanged-copy control**. Then I made one deliberate edit in a separate copy: add `+3.0` to the first `action` dimension of every frame in **episode 0** , and recompute that episode’s stored action min/max/mean statistics. Episode IDs, lengths, tasks, schema, and the other episodes were left unchanged. | Comparison | What the CLI reported | | --- | --- | | v2.1, original vs. unchanged copy | **0 changed, 2 unchanged** | | v2.1, original vs. action-edited copy | **1 changed (episode 0), 1 unchanged** | | v3.0, original vs. unchanged copy | **0 changed, 206 unchanged** | | v3.0, original vs. action-edited copy | **1 changed (episode 0), 205 unchanged** | The action-edit comparisons both completed successfully, and the v3.0 report showed the expected `+3.0000` delta in the first dimension of action min, max, and mean. No added or removed episodes were reported. The unchanged controls also behaved as expected. **So the first concrete result is positive:** this particular episode-level, stats-based comparison worked with a _real public v2.1 dataset_ as well as v3.0. That seems worth mentioning because the [README currently calls v2.1 support beta](https://github.com/yafetashenafi790-lgtm/robot-data-diff). It is one successful v2.1 example, not a claim about every v2.1 layout. These were **controlled modifications to copies of public data** , not naturally occurring training regressions. I did not retrain a policy or inspect videos, so the test says something about the diff behavior, not about why a policy’s performance might change. ## The part that surprised me The most actionable surprise was `--no-quality`. The changed episode was still detected with that option, as expected. But I instrumented `pyarrow.parquet.read_table` in the installed loader, and **turning quality off did not eliminate the reads of action data used to compute the quality indicators** : | Dataset | `with_quality=True` | `with_quality=False` | | --- | --- | --- | | v2.1, two episode files | 2 action-Parquet reads | **2 action-Parquet reads** | | v3.0, one packed data shard | 1 metadata-Parquet + 1 action-data read | **1 metadata-Parquet + 1 action-data read** | This counts Python-level `read_table` calls on small **local** datasets. It does **not** measure total bytes read, remote download size, or a proven runtime speedup. The [loader at the tested source revision](https://github.com/yafetashenafi790-lgtm/robot-data-diff/blob/21faaf4bc2c54c2a494513a87cc1f31420d29a2d/loader.py) explains the pattern: quality values are calculated while loading and then cleared when `with_quality=False`. One fairly small improvement, consistent with the **no-setup, direct-diff** goal, would be to keep the current CLI but separate two operations: 1. **Local computation:** when quality is off, skip the action-array scan that is only needed for idle-ratio and spike-count heuristics. Keep using the existing episode metadata and action summaries for the statistical diff. 2. **Remote retrieval, if worth optimizing separately:** the current Hub path downloads `meta/ **` and `data/** `. Skipping a local scan alone would not prevent those downloads. A future metadata-focused remote mode could request only the files it actually needs, with an explicit statement of its comparison coverage. I would preserve the ordinary `diff A B` path and keep richer validation optional. The useful distinction is not “more features at all costs,” but **a quick summary diff versus additional data-quality or raw-content checks**. There was also a useful boundary check: in a separate v2.1 copy I changed `observation.state` and its stored state statistics, leaving actions untouched. The report still showed **0 changed episodes**. That matches the current [episode fingerprint](https://github.com/yafetashenafi790-lgtm/robot-data-diff/blob/21faaf4bc2c54c2a494513a87cc1f31420d29a2d/stats.py), which uses action statistics but not state statistics. I would treat this as a **scope/wording choice** , not necessarily a bug. In particular, `unchanged` means _unchanged under the selected fingerprint_, not that every frame, state value, or video byte is identical. The exact inputs and observations are below, along with a few optional follow-up branches. None requires changing the project’s basic approach. > **Probe inputs, controls, and CLI output** > > ### What was actually run > > - Runtime: Linux Colab **CPU** , Python `3.13.16`. > - Package: published `robot-data-diff==0.1.0` wheel, not an editable checkout. > - Relevant libraries: `numpy 2.1.3`, `pandas 2.2.3`, `pyarrow 23.0.1`, `huggingface_hub 0.36.2`. > - Published wheel SHA-256, also checked against [PyPI](https://pypi.org/project/robot-data-diff/): > > ```plaintext > 1e404b1cdd5dd118d1b2e869f1271c23ae94d4a29e34ed630cc183012c6ecee8 > > ``` > > The public dataset inputs were pinned to full Hub commit IDs so their meaning would not depend on a later update to `main`: > > ```plaintext > v2.1 rshoff/lerobot > 876192fd1bad1ef4992a0b2f059be0f90718d935 > > v3.0 lerobot/pusht > 7628202a2180972f291ba1bc6723834921e72c19 > > ``` > > Only the required metadata and data Parquet files were fetched. The v2.1 input had two per-episode action Parquet files and JSONL episode metadata; the v3.0 input had episode-metadata Parquet and one packed data Parquet file. **Videos were neither downloaded nor decoded** , and nothing was uploaded to the Hub. > > The probe used _separate local directories_ for A and B. It did not exercise the package’s remote two-revision comparison path. > > ### Control and edited comparisons > > In both formats, the negative control was A versus an unchanged copy of A. For the positive control, the only intentional data alteration was `action[:, 0] += 3.0` for episode 0, followed by updating the stored per-episode action summaries in the copied metadata. This matters: the test probes a **consistent edit** , not a disagreement between raw frames and stale summary statistics. > > Representative excerpts from the **actual published CLI output** : > > **v2.1 edited action:** > > ```plaintext > Episodes: 2 -> 2 (+0 added, -0 removed, ~1 changed, =1 unchanged) > > CHANGED EPISODES (1): > [ep 0] length 853 -> 853 (+0 frames) > d(action_min) [+3.0000, +0.0000, +0.0000, +0.0000, +0.0000, +0.0000] > d(action_max) [+3.0000, +0.0000, +0.0000, +0.0000, +0.0000, +0.0000] > d(action_mean) [+3.0000, -0.0007, +0.0004, +0.0000, +0.0000, -0.0001] > > ``` > > **v3.0 edited action:** > > ```plaintext > Episodes: 206 -> 206 (+0 added, -0 removed, ~1 changed, =205 unchanged) > > CHANGED EPISODES (1): > [ep 0] length 161 -> 161 (+0 frames) > d(action_min) [+3.0000, +0.0000] > d(action_max) [+3.0000, +0.0000] > d(action_mean) [+3.0000, +0.0000] > > ``` > > The very small nonzero v2.1 **mean** deltas outside the intentionally shifted dimension deserve a narrow interpretation. The edited statistics were recomputed from the action frames, whereas the original statistics came from the published metadata. Differences in stored precision or prior aggregation are plausible; I did **not** establish a bug in the differ or deliberately alter those other dimensions. > > The unchanged-copy runs reported zero changes. The v2.1 state-only test also reported zero changes (with two unchanged episodes). All six CLI comparisons completed with exit code `0`; the separate instrumentation expectations for skipping quality-only reads did **not** pass. Successful execution and meeting every behavioral expectation are different claims. > > For someone preparing the same kinds of local copies, the CLI calls are straightforward: > > ```bash > robot-data-diff diff ./v2_original ./v2_action_edited \ > --checkpoint v2_action_test --html v2_diff.html > > robot-data-diff diff ./v3_original ./v3_action_edited \ > --checkpoint v3_action_test --html v3_diff.html > > robot-data-diff diff ./v2_original ./v2_action_edited \ > --no-quality > > ``` > > These are **example commands for prepared local directories** , not a complete script for creating the edited fixtures. The public Hub datasets themselves have not been altered. > **Why the quality-off reads matter** > > The local read instrumentation captured these exact categories of calls: > > ```plaintext > v2.1, with_quality=True: > data/chunk-000/episode_000000.parquet columns=['action'] > data/chunk-000/episode_000001.parquet columns=['action'] > > v2.1, with_quality=False: > data/chunk-000/episode_000000.parquet columns=['action'] > data/chunk-000/episode_000001.parquet columns=['action'] > > v3.0, with_quality=True: > meta/episodes/chunk-000/file-000.parquet columns=all > data/chunk-000/file-000.parquet columns=['episode_index', 'action'] > > v3.0, with_quality=False: > meta/episodes/chunk-000/file-000.parquet columns=all > data/chunk-000/file-000.parquet columns=['episode_index', 'action'] > > ``` > > The v3.0 metadata read is necessary for this comparison. The possible saving is the **additional frame-data read performed only to compute quality fields that will not be returned**. In v2.1, both instrumented Parquet calls are action-data reads; the JSONL metadata is loaded separately. > > In [the pinned `loader.py`](https://github.com/yafetashenafi790-lgtm/robot-data-diff/blob/21faaf4bc2c54c2a494513a87cc1f31420d29a2d/loader.py), `_load_v2` and `_load_v3` call their respective `_add_quality_*` functions before `load_dataset` clears the quality fields if `with_quality=False`. A conceptual change would be to pass the quality flag into that stage and guard the scans, rather than computing values and discarding them afterwards. The metadata action statistics can still support the existing fingerprint comparison where those fields are present. > > This separates three things that can otherwise get mixed together: > > - **CPU-only operation:** already demonstrated by this probe. > - **Less local reading:** plausible with a small conditional path; not benchmarked here. > - **Less Hub downloading:** requires changing the retrieval policy independently of the local quality calculation. > > The [Hub download guide](https://huggingface.co/docs/huggingface_hub/en/guides/download) documents file filtering (`allow_patterns`) and distinguishes revision-aware cached snapshots from downloads into an explicit `local_dir`. That is relevant if a remote metadata-only path or revision isolation becomes useful. It is not evidence that the local read-count difference translates directly into any particular time or bandwidth saving. > > A minimal regression check after such an optimization could use the same two unchanged controls, the two action-edited copies, and an assertion that `with_quality=False` avoids reading `data/**` solely for quality computation. For Hub references, the list of files downloaded should be checked separately. > **Meaning of unchanged, and optional deeper checks** > > The current [fingerprint implementation](https://github.com/yafetashenafi790-lgtm/robot-data-diff/blob/21faaf4bc2c54c2a494513a87cc1f31420d29a2d/stats.py) compares episodes by `episode_index` and summarizes episode length, sorted task labels, action min/max/mean, and optional idle-ratio/spike-count values. It rounds numerical summaries for the comparison. Schema changes are reported separately. > > In the controlled **state-only** test, `observation.state` and its state summary were modified consistently while action data and metadata remained unchanged. The output was: > > ```plaintext > Episodes: 2 -> 2 (+0 added, -0 removed, ~0 changed, =2 unchanged) > > No changed episodes (same index + identical stats). > > ``` > > That makes the scope distinction concrete. Other differences, such as rearranging an action sequence while keeping the same summary statistics, could also escape a summary fingerprint; a video-only change is outside the loader’s scope. **I did not run those two additional tests.** > > Possible comparison levels, depending on the user’s actual question: > > | Question | Possible level | > | --- | --- | > | Did episodes or their action summary statistics change? | Keep the present fast episode-level diff. | > | Did observation/state summaries change? | Consider an optional state-statistics comparison, if useful. | > | Did the complete ordered trajectory change? | Separate frame-by-frame equality check or content hash. | > | Did camera data change or freeze? | Separate video-aware inspection. | > | Did a policy become worse _because of_ those changes? | Compare training and evaluation conditions as well; a dataset diff alone cannot establish causation. | > > This seems like a good place for a short report-level phrase such as **“unchanged under the selected statistical fingerprint”**. It would clarify the current result without changing what the tool is trying to be. > > Episode numbering is also the current identity rule. If a transformation renumbers the same trajectories, they may be reported as removed and added instead of moved. That is a reasonable, explicit design boundary for an early direct-diff tool; stable identity across renumbering would be a separate feature decision. > **Compatibility and remote-revision cases not exercised here** > > The successful v2.1 test used one public dataset with the expected JSONL and per-episode Parquet layout. It does not certify other v2.1 variants. A known-good public fixture may be useful as a small compatibility regression test if that beta path is developed further. > > The [official LeRobot v3.0 format documentation](https://huggingface.co/docs/lerobot/lerobot-dataset-v3) describes packed Parquet storage and metadata-based episode boundaries. Our successful `lerobot/pusht` test used one data shard; it did **not** test episode boundaries across multiple shards or unusual task associations. Those would be separate, compact fixtures if actual compatibility reports arose. > > Finally, this probe did **not** test `dataset@revisionA` against `dataset@revisionB` using Hub IDs. The CLI comparisons used pinned source files placed into independent local directories. If remote comparisons with an explicit `--cache-dir` become a focus, it would be worth checking that the two resolved revision contents are kept distinct. The [current resolver](https://github.com/yafetashenafi790-lgtm/robot-data-diff/blob/21faaf4bc2c54c2a494513a87cc1f31420d29a2d/loader.py) passes that option as `local_dir` to `snapshot_download`, while the [Hub documentation](https://huggingface.co/docs/huggingface_hub/en/guides/download) distinguishes local-folder downloading from the normal revision-aware snapshot cache. > > There was a [historical, now-closed LeRobot issue about revisions on shared local storage](https://github.com/huggingface/lerobot/issues/3224). It concerns a different implementation and is **not** evidence that `robot-data-diff` has the same bug. If relevant, a small known-difference, two-commit comparison using isolated destinations versus a shared explicit directory would distinguish retrieval behavior from diff behavior. This is an optional future check, not something required for the ordinary local `diff A B` workflow. The useful outcome of this first test is that the **published CLI detected a deliberately introduced episode-level action change in both tested LeRobot formats** , including a real v2.1 dataset. The one improvement I would prioritize from the observations is making `--no-quality` truly avoid **quality-only action-data reads** , while keeping the fast statistical diff distinct from deeper validation.