SuhaoYu1020/Uni-Sign-YTASL-stage1-v2

Uni-Sign stage-1 (pose-only pre-training) on YouTube-ASL — independent reproduction (v2)

Stage-1 weights of Uni-Sign (pose-only pre-training), trained from scratch (mT5-base initialisation) on YouTube-ASL with the official code.

The official release (ZechengLi19/Uni-Sign) provides stage-1/2 pre-training weights only for CSL-News (csl_stage1_weight.pth, csl_stage2_weight.pth); for ASL it releases fine-tuned downstream checkpoints (How2Sign, OpenASL, WLASL) but no YouTube-ASL pre-training weights.

Files

file	what
best_checkpoint.pth	epoch 16 (epochs are 0-indexed, 0–19), selected by pre-training validation BLEU-4 (see the caveat below)
checkpoint_19.pth	final epoch (epoch 19, 0-indexed)
log.txt	per-epoch train loss / LR / validation metrics (JSON lines)

Each .pth is {"model": state_dict} (587.7M parameters, no optimizer state):

sd = torch.load("best_checkpoint.pth", map_location="cpu")["model"]
missing, unexpected = model.load_state_dict(sd, strict=False)   # model = Uni-Sign's models.Uni_Sign

Use it the way the official stage-1 weights are used: --finetune best_checkpoint.pth for stage 2 (RGB-pose) or stage 3 (downstream fine-tuning).

Training

• Data: YouTube-ASL, 566,928 caption-aligned clips from 10,165 videos (poses extracted with Uni-Sign's pipeline). The pre-training loader splits news_labels.json by order: 561,258 train / 5,670 validation clips.
• Recipe: official stage-1 settings — 4 × B200, batch 128 per GPU (effective 512), LR 3e-4, 20 epochs, pose only. ~70 min per epoch.
• Code: upstream Uni-Sign plus fixes that do not change stage-1 training: ROUGE crash on empty strings in evaluation, deterministic frame sampling at evaluation, dataset read retries, and the PGF fusion-gate initialisation fix (RGB branch; stage 2 only).

Results

Pre-training validation BLEU-4: 12.21 (best, epoch 16) / 12.20 (final). Not a held-out number: the validation clips are a random 1% of clips, and 99.94% of their videos also appear in the training split.

Downstream: this checkpoint → stage-3 pose-only fine-tuning (4 GPUs × batch 32, LR 3e-4, 20 epochs, stage 2 skipped) → test set, deterministic frame sampling. The official row is the official fine-tuned checkpoint re-evaluated with the same harness.

test set	BLEU-4 / ROUGE-L, last epoch	BLEU-4 / ROUGE-L, best epoch	official checkpoint
How2Sign (N=2,349)	13.45 / 33.26	13.89 / 33.91 (test-selected: How2Sign has no dev split)	14.71 / 34.67
OpenASL (N=976)	21.71 / 41.99	21.36 / 41.91 (dev-selected)	23.06 / 43.04

OpenASL contamination: OpenASL and YouTube-ASL share source videos. 226 of the 456 OpenASL test videos (49.6%) are in this pre-training set with English translations, so OpenASL test numbers are not a clean generalisation measure (this applies to the official Uni-Sign checkpoint as well).

Citation

@article{li2025uni,
  title={Uni-Sign: Toward Unified Sign Language Understanding at Scale},
  author={Li, Zecheng and Zhou, Wengang and Zhao, Weichao and Wu, Kepeng and Hu, Hezhen and Li, Houqiang},
  journal={arXiv preprint arXiv:2501.15187},
  year={2025}
}
@inproceedings{uthus2023youtubeasl,
  title={YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus},
  author={Uthus, David and Tanzer, Garrett and Georg, Manfred},
  booktitle={NeurIPS Datasets and Benchmarks},
  year={2023}
}