thenukegun10x/PlantDetect-V2-Preview

PlantDetect V2 Preview

Built with DINOv3. This is the first preview of the completed V2 mean-pooling model for 2,991 plant species. Supply one photo, or 2–4 distinct photos from the same plant observation. This release uses the epoch-12 EMA weights from the expanded-data BF16 training run. The CLI uses self-contained TorchScript graphs (encoder.ts and classifier.ts) and imports no project modules. FP32 model.safetensors is also included for research use. Optimizer state and training data are excluded.

Measured accuracy

Evaluation	Photos / observations	Top-1	Top-5
Internal validation, individual photos	42,944 photos	86.25%	93.64%
Internal validation, one-view fused	18,848 observations	90.15%	96.16%
Internal validation, multi-photo fused	10,491 observations	93.24%	98.16%
Separate WA single-photo benchmark	1,822 observations, 271 species	86.33%	93.47%

The external benchmark was selected before predictions and audited for known training overlap. It samples species across three training-count bands and covers 271 of 2,991 classes. It is not a natural-use-frequency estimate or an all-class accuracy measurement. Its single-photo metric is directly comparable with the internal individual photos row, not the one-view fused cohort. Photos mostly come through the GBIF/iNaturalist source family; community labels and incomplete historical upstream observation IDs limit the audit. See evaluation.json for the benchmark manifest checksum and breakdown. These rows were measured from the source EMA checkpoint with GPU BF16 autocast. The FP32 export contains exactly the same weights; CPU FP32 predictions may differ on close calls.

Run locally

Requires Python 3.11+, PyTorch, Pillow, and NumPy. The CLI works on Windows and Linux; it does not require timm, safetensors, or a checkout of this project.

pip install -r requirements.txt
python plant_v2_cli.py identify leaf.jpg flower.jpg --device cpu --topk 5

The CLI also accepts --images leaf.jpg flower.jpg, and info prints model metadata. Inputs are RGB, bilinear resized to 576 x 576, centre cropped to 448 x 448, normalized with ImageNet mean/std, and EXIF orientation is applied (phone photos routinely carry an orientation tag; pass --no-exif to disable). The encoder ships as bf16 for both CPU and GPU (encoder.ts / encoder_gpu.ts); bf16 is ~35x faster than fp32 for this model on CPUs with AVX512-BF16 (oneDNN: 1124 vs 32 GFLOP/s) and matches the bf16 autocast the accuracy numbers above were measured with.

Output is a short human-readable report by default; --format json emits the full structured result (strict JSON, so non-finite values become null). Exit codes make it scriptable: 0 identified, 1 not confidently known, 2 usage or input error (missing or corrupt image, checksum mismatch, unavailable device). Scores are uncalibrated softmax; do not interpret them as measured probabilities.

Unknown species

The classifier can only ever return one of the known taxa, so for a plant outside the taxonomy the honest answer is "not confidently a known species". The CLI requires several independent signals before it names a species, and otherwise reports rejected_unknown with the failing checks - it never silently snaps a guess to the nearest class.

• Taxonomic agreement (always available). The species head's pick must sit in the same genus and family that the genus and family heads predict, and with 2-4 photos every photo's genus must match the fused pick. This is the strongest signal and needs no extra download.
• Nearest-centroid distance (optional). Each known species has an embedding centroid built from training photos; the nearest-centroid cosine must exceed a threshold calibrated on held-out photos of known species, so the false-rejection rate on knowns is a chosen quantile rather than a guess.

The centroid bundle (species_centroids.npz + centroid_report.json, ~5.4 MB) is optional. Without it the CLI still runs and still applies the taxonomic checks, and says so in the output. It is never downloaded implicitly: when it is missing and you are at a terminal the CLI offers to fetch it, or you can run it explicitly.

python plant_v2_cli.py fetch-centroids          # cache it, sha256-verified
python plant_v2_cli.py fetch-centroids --force  # re-download

The cache defaults to tempfile.gettempdir()/plantdetect; override with --cache-dir or PLANTDETECT_CACHE_DIR. --yes skips the prompt, --no-download never offers.

--relaxed keeps only the calibrated distance gate (the taxonomic checks become advisory); --no-reject disables rejection entirely (unsafe - a species is always named); --min-centroid-cosine moves the threshold.

Measured both ways with a grouped 5-fold cross-validation (threshold chosen on 4 folds, scored on the held-out fold), against 1,822 known photos (external WA benchmark, 271 species) and 6,318 photos of 819 species the model was never trained on (image_expansion rows absent from the taxonomy):

threshold	known accepted	unknown rejected
0.60 (first release)	89.0%	52.1%
0.84 (default, max Youden J)	73.0%	83.6%
0.90 (more protective)	62.7%	89.8%
0.95 (most protective)	46.3%	95.5%

Cross-validated: 72.6% +/- 4.8 known accepted and 83.8% +/- 0.8 unknown rejected at threshold 0.842 +/- 0.004 - essentially the same as in-sample, so the operating point is not overfit.

The distance gate cannot be both protective and usable: reaching ~95% unknown rejection costs ~54% of known species. Two earlier figures were wrong and are corrected here - a knowns-only calibration put the threshold at 0.26 (52% unknown rejection), and a 4-species unknown set (103 of 110 photos one seagrass) suggested 93.6% at 0.60. Neither survives contact with 819 real species. centroid_report.json records the full protection curve and the selection rule; --min-centroid-cosine moves the line. The always-on taxonomic checks reject only ~31% of unknowns on their own, so they corroborate rather than carry.

Native ExecuTorch PTE and FP8 weights

executorch/PlantDetect-WA-V2-INT8.pte is a W8A8 export of this same mean-pooling checkpoint for CPU inference without PyTorch. It uses custom plant_quant::* kernels (INT8 weights with per-output-channel FP32 scales, FP16 activations, FP32 accumulation) and ships for the Arm I8MM, SDOT and plain FP16-FML paths. On a Galaxy S24 (Exynos 2400) one 448x448 encode runs in roughly 3 seconds at 5 threads; on an Allwinner A733 (2x Cortex-A76 + 6x Cortex-A55) it runs in roughly 25 seconds at 2 threads. The W8A8 path measures 86.11% top-1 on the separate WA single-photo benchmark above, 0.22 points below the FP32 export. The PTE is single-photo (one image per encode); multi-photo fusion uses classifier.ts with one embedding per photo.

fp8/model.safetensors is the AdaRound E4M3FN per-channel weight-only export of the same checkpoint. Adaptive rounding improved 107 of 108 layer reconstructions (0.875% aggregate MSE reduction over nearest rounding); see fp8/adaround_report.json.

Running the PTE. It does not load on a stock ExecuTorch runtime — it requires the four custom plant_quant::* operators. The operator sources, build instructions and a prebuilt Windows x64 runner are included:

executorch/windows-x64/plant_runner.exe executorch/PlantDetect-WA-V2-INT8.pte encode image.bin embedding.bin
executorch/windows-x64/plant_runner.exe executorch/PlantDetect-WA-V2-INT8.pte classify slots.bin out 4

image.bin is float32 ImageNet-normalised RGB 448x448 (3*448*448 floats); slots.bin is 4*512 floats. Set OMP_NUM_THREADS (5 on an Exynos 2400, 2 on an Allwinner A733). Required ExecuTorch version and commit, ISA tiers (PLANTDETECT_MARCH), activation modes (PLANTDETECT_ACT) and profiling switches are documented in extension/moe/README.md.

Attribution and terms

Built with DINOv3 (vit_base_patch16_dinov3.lvd1689m). DINO-derived weights remain subject to the included DINOv3 agreement. Project-specific code is Apache-2.0. No training or benchmark photos are redistributed. See NOTICE.md.