HTT — Heterogeneous Tactile Transformer

A multimodal tactile representation model. One shared transformer backbone encodes four different tactile sensors into a common 192-dimensional embedding space, pretrained with masked-autoencoder reconstruction and cross-modal alignment. Feed a raw sensor reading, get a feature vector for any downstream head (classification, force / slip estimation, policy learning).

Modality	Type	Raw input
gsmini	vision (GelSight Mini)	uint8 image [224, 224, 3]
9dtact	vision (9DTact)	uint8 image [224, 224, 3]
xela	taxel array	float [T, 72]
tac02	taxel array	float [T, 66]

Checkpoint

File	htt_4sensors_best.pth (~69 MB)
Contents	model_state_dict = 4 encoders + shared 9-layer trunk + 4 decoders (17.1 M params)
Embedding dim	192
SHA-256	024f4c3a067168197d0a6996bbca5c03e744ed5abd1d35a666dbf78e7ac673f0

Slim inference/finetune checkpoint (optimizer / predictor states dropped).

Usage

Use it with the HTT package (contains the architecture, preprocessing, and examples). Download the weights into checkpoints/:

hf download AllenBi21/HTT htt_4sensors_best.pth --local-dir checkpoints

import numpy as np
from htt import HTT

model = HTT(modality="gsmini")                       # loads checkpoints/htt_4sensors_best.pth
frame = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8)   # your sensor frame
emb = model(frame)                                   # -> torch.Tensor [1, 192]

Read the raw-input contract before feeding your own data — the model returns garbage on out-of-distribution inputs.

License

Released under CC BY-NC 4.0 (non-commercial). Portions are derived from Meta's V-JEPA / DINOv2 (Apache-2.0 and CC-BY-NC-4.0).