For now, I gave the public release a quick try:
The first diffs I ran
I installed the published robot-data-diff==0.1.0 package in a Colab CPU runtime and tried it on local copies of two public LeRobot datasets:
For each dataset, I first ran an unchanged-copy control. Then I made one deliberate edit in a separate copy: add +3.0 to the first action dimension of every frame in episode 0, and recompute that episode’s stored action min/max/mean statistics. Episode IDs, lengths, tasks, schema, and the other episodes were left unchanged.
| Comparison |
What the CLI reported |
| v2.1, original vs. unchanged copy |
0 changed, 2 unchanged |
| v2.1, original vs. action-edited copy |
1 changed (episode 0), 1 unchanged |
| v3.0, original vs. unchanged copy |
0 changed, 206 unchanged |
| v3.0, original vs. action-edited copy |
1 changed (episode 0), 205 unchanged |
The action-edit comparisons both completed successfully, and the v3.0 report showed the expected +3.0000 delta in the first dimension of action min, max, and mean. No added or removed episodes were reported. The unchanged controls also behaved as expected.
So the first concrete result is positive: this particular episode-level, stats-based comparison worked with a real public v2.1 dataset as well as v3.0. That seems worth mentioning because the README currently calls v2.1 support beta. It is one successful v2.1 example, not a claim about every v2.1 layout.
These were controlled modifications to copies of public data, not naturally occurring training regressions. I did not retrain a policy or inspect videos, so the test says something about the diff behavior, not about why a policy’s performance might change.
The part that surprised me
The most actionable surprise was --no-quality.
The changed episode was still detected with that option, as expected. But I instrumented pyarrow.parquet.read_table in the installed loader, and turning quality off did not eliminate the reads of action data used to compute the quality indicators:
| Dataset |
with_quality=True |
with_quality=False |
| v2.1, two episode files |
2 action-Parquet reads |
2 action-Parquet reads |
| v3.0, one packed data shard |
1 metadata-Parquet + 1 action-data read |
1 metadata-Parquet + 1 action-data read |
This counts Python-level read_table calls on small local datasets. It does not measure total bytes read, remote download size, or a proven runtime speedup. The loader at the tested source revision explains the pattern: quality values are calculated while loading and then cleared when with_quality=False.
One fairly small improvement, consistent with the no-setup, direct-diff goal, would be to keep the current CLI but separate two operations:
- Local computation: when quality is off, skip the action-array scan that is only needed for idle-ratio and spike-count heuristics. Keep using the existing episode metadata and action summaries for the statistical diff.
- Remote retrieval, if worth optimizing separately: the current Hub path downloads
meta/** and data/**. Skipping a local scan alone would not prevent those downloads. A future metadata-focused remote mode could request only the files it actually needs, with an explicit statement of its comparison coverage.
I would preserve the ordinary diff A B path and keep richer validation optional. The useful distinction is not “more features at all costs,” but a quick summary diff versus additional data-quality or raw-content checks.
There was also a useful boundary check: in a separate v2.1 copy I changed observation.state and its stored state statistics, leaving actions untouched. The report still showed 0 changed episodes. That matches the current episode fingerprint, which uses action statistics but not state statistics. I would treat this as a scope/wording choice, not necessarily a bug. In particular, unchanged means unchanged under the selected fingerprint, not that every frame, state value, or video byte is identical.
The exact inputs and observations are below, along with a few optional follow-up branches. None requires changing the project’s basic approach.
Probe inputs, controls, and CLI output
What was actually run
- Runtime: Linux Colab CPU, Python
3.13.16.
- Package: published
robot-data-diff==0.1.0 wheel, not an editable checkout.
- Relevant libraries:
numpy 2.1.3, pandas 2.2.3, pyarrow 23.0.1, huggingface_hub 0.36.2.
- Published wheel SHA-256, also checked against PyPI:
1e404b1cdd5dd118d1b2e869f1271c23ae94d4a29e34ed630cc183012c6ecee8
The public dataset inputs were pinned to full Hub commit IDs so their meaning would not depend on a later update to main:
v2.1 rshoff/lerobot
876192fd1bad1ef4992a0b2f059be0f90718d935
v3.0 lerobot/pusht
7628202a2180972f291ba1bc6723834921e72c19
Only the required metadata and data Parquet files were fetched. The v2.1 input had two per-episode action Parquet files and JSONL episode metadata; the v3.0 input had episode-metadata Parquet and one packed data Parquet file. Videos were neither downloaded nor decoded, and nothing was uploaded to the Hub.
The probe used separate local directories for A and B. It did not exercise the package’s remote two-revision comparison path.
Control and edited comparisons
In both formats, the negative control was A versus an unchanged copy of A. For the positive control, the only intentional data alteration was action[:, 0] += 3.0 for episode 0, followed by updating the stored per-episode action summaries in the copied metadata. This matters: the test probes a consistent edit, not a disagreement between raw frames and stale summary statistics.
Representative excerpts from the actual published CLI output:
v2.1 edited action:
Episodes: 2 -> 2 (+0 added, -0 removed, ~1 changed, =1 unchanged)
CHANGED EPISODES (1):
[ep 0] length 853 -> 853 (+0 frames)
d(action_min) [+3.0000, +0.0000, +0.0000, +0.0000, +0.0000, +0.0000]
d(action_max) [+3.0000, +0.0000, +0.0000, +0.0000, +0.0000, +0.0000]
d(action_mean) [+3.0000, -0.0007, +0.0004, +0.0000, +0.0000, -0.0001]
v3.0 edited action:
Episodes: 206 -> 206 (+0 added, -0 removed, ~1 changed, =205 unchanged)
CHANGED EPISODES (1):
[ep 0] length 161 -> 161 (+0 frames)
d(action_min) [+3.0000, +0.0000]
d(action_max) [+3.0000, +0.0000]
d(action_mean) [+3.0000, +0.0000]
The very small nonzero v2.1 mean deltas outside the intentionally shifted dimension deserve a narrow interpretation. The edited statistics were recomputed from the action frames, whereas the original statistics came from the published metadata. Differences in stored precision or prior aggregation are plausible; I did not establish a bug in the differ or deliberately alter those other dimensions.
The unchanged-copy runs reported zero changes. The v2.1 state-only test also reported zero changes (with two unchanged episodes). All six CLI comparisons completed with exit code 0; the separate instrumentation expectations for skipping quality-only reads did not pass. Successful execution and meeting every behavioral expectation are different claims.
For someone preparing the same kinds of local copies, the CLI calls are straightforward:
robot-data-diff diff ./v2_original ./v2_action_edited \
--checkpoint v2_action_test --html v2_diff.html
robot-data-diff diff ./v3_original ./v3_action_edited \
--checkpoint v3_action_test --html v3_diff.html
robot-data-diff diff ./v2_original ./v2_action_edited \
--no-quality
These are example commands for prepared local directories, not a complete script for creating the edited fixtures. The public Hub datasets themselves have not been altered.
Why the quality-off reads matter
The local read instrumentation captured these exact categories of calls:
v2.1, with_quality=True:
data/chunk-000/episode_000000.parquet columns=['action']
data/chunk-000/episode_000001.parquet columns=['action']
v2.1, with_quality=False:
data/chunk-000/episode_000000.parquet columns=['action']
data/chunk-000/episode_000001.parquet columns=['action']
v3.0, with_quality=True:
meta/episodes/chunk-000/file-000.parquet columns=all
data/chunk-000/file-000.parquet columns=['episode_index', 'action']
v3.0, with_quality=False:
meta/episodes/chunk-000/file-000.parquet columns=all
data/chunk-000/file-000.parquet columns=['episode_index', 'action']
The v3.0 metadata read is necessary for this comparison. The possible saving is the additional frame-data read performed only to compute quality fields that will not be returned. In v2.1, both instrumented Parquet calls are action-data reads; the JSONL metadata is loaded separately.
In the pinned loader.py, _load_v2 and _load_v3 call their respective _add_quality_* functions before load_dataset clears the quality fields if with_quality=False. A conceptual change would be to pass the quality flag into that stage and guard the scans, rather than computing values and discarding them afterwards. The metadata action statistics can still support the existing fingerprint comparison where those fields are present.
This separates three things that can otherwise get mixed together:
- CPU-only operation: already demonstrated by this probe.
- Less local reading: plausible with a small conditional path; not benchmarked here.
- Less Hub downloading: requires changing the retrieval policy independently of the local quality calculation.
The Hub download guide documents file filtering (allow_patterns) and distinguishes revision-aware cached snapshots from downloads into an explicit local_dir. That is relevant if a remote metadata-only path or revision isolation becomes useful. It is not evidence that the local read-count difference translates directly into any particular time or bandwidth saving.
A minimal regression check after such an optimization could use the same two unchanged controls, the two action-edited copies, and an assertion that with_quality=False avoids reading data/** solely for quality computation. For Hub references, the list of files downloaded should be checked separately.
Meaning of unchanged, and optional deeper checks
The current fingerprint implementation compares episodes by episode_index and summarizes episode length, sorted task labels, action min/max/mean, and optional idle-ratio/spike-count values. It rounds numerical summaries for the comparison. Schema changes are reported separately.
In the controlled state-only test, observation.state and its state summary were modified consistently while action data and metadata remained unchanged. The output was:
Episodes: 2 -> 2 (+0 added, -0 removed, ~0 changed, =2 unchanged)
No changed episodes (same index + identical stats).
That makes the scope distinction concrete. Other differences, such as rearranging an action sequence while keeping the same summary statistics, could also escape a summary fingerprint; a video-only change is outside the loader’s scope. I did not run those two additional tests.
Possible comparison levels, depending on the user’s actual question:
| Question |
Possible level |
| Did episodes or their action summary statistics change? |
Keep the present fast episode-level diff. |
| Did observation/state summaries change? |
Consider an optional state-statistics comparison, if useful. |
| Did the complete ordered trajectory change? |
Separate frame-by-frame equality check or content hash. |
| Did camera data change or freeze? |
Separate video-aware inspection. |
| Did a policy become worse because of those changes? |
Compare training and evaluation conditions as well; a dataset diff alone cannot establish causation. |
This seems like a good place for a short report-level phrase such as “unchanged under the selected statistical fingerprint”. It would clarify the current result without changing what the tool is trying to be.
Episode numbering is also the current identity rule. If a transformation renumbers the same trajectories, they may be reported as removed and added instead of moved. That is a reasonable, explicit design boundary for an early direct-diff tool; stable identity across renumbering would be a separate feature decision.
Compatibility and remote-revision cases not exercised here
The successful v2.1 test used one public dataset with the expected JSONL and per-episode Parquet layout. It does not certify other v2.1 variants. A known-good public fixture may be useful as a small compatibility regression test if that beta path is developed further.
The official LeRobot v3.0 format documentation describes packed Parquet storage and metadata-based episode boundaries. Our successful lerobot/pusht test used one data shard; it did not test episode boundaries across multiple shards or unusual task associations. Those would be separate, compact fixtures if actual compatibility reports arose.
Finally, this probe did not test dataset@revisionA against dataset@revisionB using Hub IDs. The CLI comparisons used pinned source files placed into independent local directories. If remote comparisons with an explicit --cache-dir become a focus, it would be worth checking that the two resolved revision contents are kept distinct. The current resolver passes that option as local_dir to snapshot_download, while the Hub documentation distinguishes local-folder downloading from the normal revision-aware snapshot cache.
There was a historical, now-closed LeRobot issue about revisions on shared local storage. It concerns a different implementation and is not evidence that robot-data-diff has the same bug. If relevant, a small known-difference, two-commit comparison using isolated destinations versus a shared explicit directory would distinguish retrieval behavior from diff behavior. This is an optional future check, not something required for the ordinary local diff A B workflow.
The useful outcome of this first test is that the published CLI detected a deliberately introduced episode-level action change in both tested LeRobot formats, including a real v2.1 dataset. The one improvement I would prioritize from the observations is making --no-quality truly avoid quality-only action-data reads, while keeping the fast statistical diff distinct from deeper validation.