PyraFuse: skin, fabric, and background segmentation

PyraFuse is a DINOv3-based semantic-segmentation framework for three classes: 0 = background, 1 = fabric, and 2 = skin. The associated paper has been accepted at AIMLSystems 2026.

Available checkpoints

Variant	Encoder	Folder
small	DINOv3 ViT-S/16	small/
small_plus	DINOv3 ViT-S+/16	small_plus/
base	DINOv3 ViT-B/16	base/
large	DINOv3 ViT-L/16	large/

Each folder is self-contained: config.json, decoder.pt, ema.pt, and a backbone/ directory with the DINOv3 configuration, model.safetensors, and adapter feature-normalisation weights. Use EMA weights for evaluation.

Usage

Install the PyraFuse code repository, then load any variant. Pin a revision for reproducible experiments.

from pyrafuse import load_pretrained

model = load_pretrained("base", revision="v1.0.0", device="cuda")

Inputs must be RGB, ImageNet-normalised, and have height and width divisible by 16. The default checkpoint configuration uses 448 × 448 inputs.

Mobile and edge: ExecuTorch programs

The mobile/ folder contains ready-to-run ExecuTorch programs for Android, iOS, React Native, Flutter, and embedded Linux. The same .pte file runs in every framework that embeds the ExecuTorch runtime.

mobile/
├── manifest.json                       # sizes, SHA-256, delegation, held-out mIoU
├── small/pyrafuse_small_<target>.pte
├── small_plus/pyrafuse_small_plus_<target>.pte
└── base/pyrafuse_base_<target>.pte

Target	Runs on	Precision
xnnpack_fp32	CPU: Android, iOS, macOS, Linux/ARM boards	FP32
xnnpack_int8	same CPUs, about 4× smaller	INT8 weights, dynamic INT8 activations
coreml_fp32	iOS 17+ / macOS 14+ GPU and CPU via Core ML	FP32
vulkan_fp32	Android GPU via Vulkan	FP32

Variant	Program	Size	mIoU	Agreement with PyTorch
small	PyTorch reference		0.907
	xnnpack_fp32	91 MB	0.907	100.00%
	xnnpack_int8	23 MB	0.906	99.78%
	coreml_fp32	92 MB	on device	on device
	vulkan_fp32	91 MB	on device	on device
small_plus	PyTorch reference		0.908
	xnnpack_fp32	119 MB	0.908	100.00%
	xnnpack_int8	31 MB	0.907	99.74%
	coreml_fp32	121 MB	on device	on device
	vulkan_fp32	119 MB	on device	on device
base	PyTorch reference		0.912
	xnnpack_fp32	348 MB	0.912	100.00%
	xnnpack_int8	88 MB	0.912	99.80%
	coreml_fp32	349 MB	on device	on device
	vulkan_fp32	348 MB	on device	on device

mIoU is measured on the first 200 images of the held-out test split at 448 px by executing each program with the ExecuTorch runtime; the Core ML and Vulkan programs need Apple or Android hardware and are not yet scored.

Contract. Input float32 [1, 3, 448, 448]: RGB, stretch-resized, divided by 255 (ImageNet normalisation is inside the program). Output float32 [1, 3, 448, 448] logits for background, fabric, skin; take argmax over dim 1.

from executorch.runtime import Runtime
from pyrafuse import download_mobile_model  # pip install "pyrafuse[mobile]"

forward = Runtime.get().load_program(
    download_mobile_model("small", "xnnpack_int8", revision="v1.2.0")).load_method("forward")
logits = forward.execute([image])[0]  # image: float32 [1, 3, 448, 448] in [0, 1]

React Native (react-native-executorch 0.10+) loads these files directly with useSemanticSegmenter using labels ['background', 'fabric', 'skin'], resizeMode: 'stretch', and normalizeOpts: { alpha: 1 / 255, beta: 0 }. Copy-paste snippets for React Native, Flutter, Kotlin, Swift, and C++ are in the code repository README.

There are no FP16 programs: DINOv3's first block produces attention logits up to about 2.5 × 10⁶, beyond the FP16 range, so FP16 backends (including the Core ML Neural Engine) output NaNs. Core ML and Vulkan programs are validated at export but have not yet been benchmarked on devices.

Release policy

These eager PyTorch checkpoint bundles are the portable source of truth; the mobile/ ExecuTorch programs are exported from them with the repository's scripts/export_mobile.py. TensorRT engines are deliberately not distributed here because they are tied to the target GPU, CUDA, TensorRT version, precision, and optimization profile. Build and validate TensorRT engines on the deployment host from the PyTorch checkpoint using the repository's scripts/export_trt.py command.

Licence

The PyraFuse decoder and repository code are Apache-2.0. Each checkpoint bundle includes Meta DINOv3 backbone material; use and redistribution of those bundles are additionally subject to the DINOv3 License. A verbatim copy is provided at LICENSES/DINOv3-LICENSE.md in this model repository. The underlying Fashionpedia and visuAAL datasets are not redistributed and retain their own terms.

Data, limitations, and responsible use

Training labels fuse Fashionpedia annotations with visuAAL skin masks. The source data is not redistributed in this repository; users must comply with the source datasets' terms. Model performance can vary by image quality, lighting, occlusion, garment appearance, and representation in source data. This model is for research and engineering use, not a medical or biometric decision system.

Citation

The accepted manuscript is included in the code repository. Formal citation metadata will be added after the AIMLSystems 2026 camera-ready publication.