shuhant/dlv-branch-h-dino-future-features

DLV Branch-H DINO future features

Gated (automatic-approval) Phase-1 collaborator delivery for the Flow-TOLD/AlpaGym collaboration. The package provides frozen, DINO-only consequence predictors that map observed driving history and candidate future ego motion to future lumos DINOv3 patch features. It is a feature provider for a downstream value head, not a policy, reward model, safety system, or end-to-end differentiable component.

Repository: shuhant/dlv-branch-h-dino-future-features (public, gated with automatic approval).

Models

public rung	Branch-H identity	examples seen	NRR @2/@4/@6 s	source checkpoint SHA-256	packaged artifact SHA-256
1b	H6-1B-cosine terminal	1,351,680	0.3798268113 / 0.5221734923 / 0.5395623551	ae172aefb16a05850f2e40b4f6504ec01c8e35f39789886317ee3d8549aecd76	2ce7a5352fc3c0380b09ae670069a990d02dce5bef637d7f8d775a0c413abba6
m	H6-M-cosine terminal	1,520,640	0.3701118527 / 0.5009018418 / 0.5092394182	5e5aabfae4ddb870cf7ed1107bb378c3469f1f24f300c2bcb0a6b1704f9b39cf	ef7be873a749a5b92098743934312a210f46a39ab41f068bd8f5a913c4d4143f

NRR is normalized regret reduction from the frozen offline realized-future retrieval evaluator. Random selection has NRR 0.0 by construction. NRR is not a value estimate, reward, safety score, success rate, or closed-loop driving metric.

The co-trained family is deliberately not included: its DINO output depends on an input-side privileged auxiliary surface constructed with future frames, so that family is not applicable to online planning. The metrics above belong only to the two DINO-only artifacts in this repository.

Frozen encoder

Do not substitute public DINOv3 weights. Both predictors are sensitive to the bundled lumos DINOv3 feature distribution:

• ViT-L/16, 24 blocks, width 1024, four storage/register tokens;
• sdpa_block_causal attention and 3-D temporal RoPE;
• LingBot-VLA2 source commit 69729b4ef24c63ec25e750915491635f4753be1d;
• teacher source blob 70fed3ab4fcc2f89642fd87c639071802108183a;
• extraction-authority NaViT source blob a1a80fc74c1c58ab6331b2cd1fece53cb6209e70;
• config SHA-256 333befe36960b094f2604084802639ab4396c1fa51e51f8d1cb160e29522ac08;
• checkpoint SHA-256 086285efd8d65bc66e96b363807c4010ee5c790b7452b765edaa23837a63705b.

The extracted wrapper removes the original OmegaConf, NumPy, LingBot package, Flex-Attention, and Flash-Attention runtime requirements. Its only numerical path is the frozen PyTorch SDPA path. Original notices and the upstream repository license are included under lumos_dinov3/.

Installation

Use Python 3.12, PyTorch from the validated Alpamayo runtime, and huggingface_hub. Clone or snapshot-download the whole gated repository so that predictor.py and lumos_dinov3/ remain adjacent.

pip install torch huggingface_hub
python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="shuhant/dlv-branch-h-dino-future-features",
    local_dir="./dlv-branch-h-dino-future-features",
)
PY

An authenticated Hugging Face account with access through this repository's automatic-approval gate is required.

Quick start

import sys
import torch

sys.path.insert(0, "./dlv-branch-h-dino-future-features")
from predictor import (
    DLVPredictor,
    PhysicalFutureActions,
    feature_history_from_physical,
)

model = DLVPredictor.from_pretrained(
    "shuhant/dlv-branch-h-dino-future-features",
    rung="m",  # or "1b"
)

history = feature_history_from_physical(
    tokens=history_tokens,            # float32 [B,4,261,1024]
    physical_motion=history_motion,   # float [B,3,5,4]
    dt=history_dt,                    # float32 [B,3], interval durations
)
future = PhysicalFutureActions(future_motion)  # float [B,K,12,5,4]
features = model.predict(history, future, future_dt, pool="none")
assert features.shape == (B, K, 12, 256, 1024)

Four complete scripts are in examples/: RGB input, feature input, history waypoint conversion, and a 64-candidate CEM-style batch.

API contract

The normative review-candidate contract is spec/dlv-api-v0.9.md. The executable transport manifest schema is spec/dlv-api.schema.json. Important frozen points:

• H1-H4 are four observed frames at a nominal 0.5 s cadence.
• Each history/future interval contains five equal-duration substeps. Every substep is (dx,dy,dyaw,dv) and is flattened substep then axis to 20 values.
• dx/dy use the interval-start ego body frame; dyaw is wrapped to [-pi,pi); dv is signed longitudinal-speed change.
• q01/q99 normalization is fp32 and unclamped.
• Future actions are [B,K,12,20]; all K candidates run in one call without cross-candidate mixing.
• Exact waypoint input requires (x,y,yaw,longitudinal_speed) at 61 substep boundaries. The (x,y,yaw)-only compatibility mode estimates speed and is explicitly outside bitwise acceptance.
• Unpooled output is float32 [B,K,12,256,1024], with fp32 per-token L2 canonicalization after the predictor head. pool="frame" and pool="global" are post-canonicalization arithmetic means.

predict is decorated with torch.no_grad(), calls the predictor in eval() mode, and disables every parameter gradient. Phase 1 does not support gradient flow. End-to-end fine-tuning is outside this delivery.

RGB preprocessing and temporal rate

RGB input is [B,4,3,H,W], uint8 [0,255] or float in [0,1]. Frames are stretched to 256x256 without cropping using bilinear interpolation, align_corners=False, antialias=True, then ImageNet-normalized and arranged as BCTHW. Each frame emits CLS + 4 storage + 256 row-major patch tokens. The history surface includes all 261 post-final-LayerNorm tokens and does not apply per-token L2 normalization.

Temporal RoPE takes an explicit positive float32 effective_fps[B]. Exact historical replay used float32(7/(camera_pts_h8-camera_pts_h1)) from its eight-frame extraction clip and reused that scalar for the causal H1-H4 prefix. Online history-only use must not inspect future frames; use the observed camera rate or nominal 2.0 Hz for an exact 0.5 s grid. Jitter relative to historical replay is an acknowledged distribution shift.

Training data and evaluation scope

Training data is NVIDIA-internal PAI/wave0 real-driving data: 215,327 exact training rows and 1,534 validation rows. The frozen held-out metric population contains 817 eligible anchor parents. The data and evaluation targets are not included in this repository.

The model is trained for real-driving imagery and motion. CarRacing is only an integration prototype for the collaborator and is outside the trained visual domain. The nominal prediction grid is 12 x 0.5 s (six seconds); other timing is accepted as continuous input but is out of training distribution.

Validated latency

Single-GPU latency, B=1, feature-entry, unpooled output, CUDA bf16 autocast, after warm-up:

rung	K=1	K=16	K=64	GPU/runtime
1b	36.79 ms	316.31 ms	1272.79 ms	H100 80GB; torch 2.8.0+cu128; CUDA 12.8
m	20.09 ms	110.17 ms	417.58 ms	H100 80GB; torch 2.8.0+cu128; CUDA 12.8

Each number is the median of three synchronized calls after one warm-up. Exact samples, p90, peak allocated memory, and cold-load time are in acceptance/predictor-{1b,m}.json. RGB encoder latency is reported separately because history encoding is parent-level and reusable across CEM iterations.

Acceptance evidence

gate	1b	m
packaged cold load	PASS (5.604 s)	PASS (1.370 s)
source checkpoint identity	PASS	PASS
history-only public signature	PASS	PASS
original evaluator same-input bitwise output	PASS	PASS
K=1/16/64 shape and fixed-K candidate isolation	PASS	PASS

Candidate isolation means that, at a fixed K, perturbing every unrelated candidate leaves candidate zero bitwise unchanged. Cross-K batch-shape comparisons are diagnostic rather than a bitwise gate because SDPA can select different numerical kernels when K changes; the observed maximum absolute cross-K differences were 0.0035141 (1b) and 0.0025113 (m).

The standalone encoder also passed cold load (4.226 s), RGB preprocessing bitwise equality, official-vs-packaged history-token bitwise equality, and the causal future-perturbation prefix gate. Its four-frame H100 median latency was 23.862 ms over three synchronized calls after one warm-up. Exact evidence is in acceptance/encoder.json.

The artifact manifest is checkpoint-manifest.json. from_pretrained checks artifact size/SHA before deserialization, then verifies export schema, original terminal SHA, canonical checkpoint identity, predictor config fingerprint, Branch-H arm, loss, and example count.

Limitations and prohibited interpretations

• Predictions are features, not calibrated probabilities or physical states.
• No collision, legality, comfort, safety, reward, or value guarantee is made.
• NRR uses realized future only in the offline evaluator; realized future is prohibited from all deployment inputs.
• No map or auxiliary encoder is used.
• Pose-only waypoint conversion cannot reproduce native velocity-derived dv.
• Six-second open-loop feature prediction may compound error and has not been validated as a closed-loop controller.
• Inputs outside the PAI/wave0 real-driving distribution, including CarRacing, require downstream validation.

Distribution and license

This is a gated (automatic-approval) NVIDIA/Kesai collaboration artifact containing internal-data-trained weights. Access is provided to authenticated collaborators through the repository gate; do not redistribute the packaged artifacts outside that access channel. Bundled third-party source notices and license text are under lumos_dinov3/; access to the weights does not grant access to the underlying training data.