Document packed runtime and expose optional AR offload

README.md

@@ -1,9 +1,70 @@
 ---
+library_name: transformers
 license: cc-by-nc-4.0
 base_model: m-a-p/YuE2-3B
-pipeline_tag: text-to-audio
-inference: false
+tags:
+- music-generation
+- orbitquant
+- quantization
+- 4-bit
+- custom-code
+language:
+- en
+- zh
+- ru
 ---
-# YuE2-3B OrbitQuant W4A4
+# YuE2-3B · OrbitQuant W4A4
 
-Private release candidate. Publication and final model card follow independent clean-load verification.
+Packed 4-bit weights and activations for all 392 transformer projections in YuE2-3B, with a directly loadable runtime and a dedicated CUDA GEMV kernel. The default example is **«До утра»**, an original Russian synth-pop song. Russian is an experiment: the upstream model advertises Chinese and English, and this example does not establish general Russian-language reliability.
+
+This is a **partial-model W4A4 release**. Embeddings, output heads, normalization, auxiliary projections and the separate FP32 audio VAE retain their original precision. No fine-tuning or distillation was performed.
+
+## Architecture
+
+YuE2 has 28 mixture-of-transformers layers with fixed AR/NAR token-type routing, hidden width 2048, MLP width 6144, 16 query heads and 8 KV heads. Each branch has Q/K/V/O attention and gate/up/down SwiGLU projections. RMSNorm and rotary position embeddings remain unchanged. This is not learned sparse MoE routing.
+
+The autoregressive branch plans an ABC score and generates semantic audio tokens. The non-autoregressive transformer uses flow matching with 32 midpoint integration steps to produce 64-dimensional latents at 25 Hz. The separate Oobleck-style VAE decodes these to 48 kHz stereo using convolutions, transposed convolutions and SnakeBeta activations; it is not a transformer. The upstream semantic encoder is not included in the released generation checkpoint.
+
+## Storage and execution
+
+2,818,572,288 transformer weights occupy 1,409,286,144 packed bytes. The complete model checkpoint is **3,035,926,840 bytes**, versus **7,261,441,640 bytes** upstream: 2.39× smaller (58.2% reduction). The separate 530,512,720-byte VAE is unchanged and downloaded at a pinned revision.
+
+Compatible Q/K/V and gate/up projections are grouped: 224 physical packed modules represent 392 logical projections without duplicated packed storage. A dedicated bias-free CUDA W4A4 GEMV handles 1–8 rows; larger inputs retain OrbitQuant's packed path. No persistent decoded BF16 or INT8 weight caches are used. Original AR CUDA graphs and prefix caching were already present upstream and are not credited as new optimizations.
+
+The default VAE core is 512 frames with the original 16-frame halo. On the recorded Russian song this was waveform-identical to 1024 frames, while lowering the decoder's isolated memory peak. This is a tested example, not a universal bitwise-equivalence guarantee.
+
+## Supported runtime
+
+**Validated only on NVIDIA RTX 5090 (SM120), Linux x86_64, glibc ≥ 2.34, Python 3.12, PyTorch 2.10.0 / CUDA 12.8.** Packaged kernels use Python ABI3. Other GPUs and Torch/CUDA combinations require rebuilding and validation; the runner rejects unsupported GPU capability. This is not a generic `AutoModel.from_pretrained` integration.
+
+```bash
+hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
+cd YuE2-W4A4
+python3.12 -m venv .venv
+source .venv/bin/activate
+pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
+pip install -r requirements.txt
+python run.py --output song
+```
+
+The output includes `audio.wav` and generation artifacts. `--prompt` accepts a JSON file containing `style`, `lyrics`, `seed`, and `cot`. `--low-memory` disables grouped projections; it is an alternative profile, not a guaranteed speedup. `--offload-ar` enables upstream AR offload: our same-song test was slower (44.33 vs 42.93 seconds) with unchanged overall peak before the VAE chunk-size change, so it is not the default. Acoustic CUDA graph capture was tested but not selected as a default due to its small benefit and additional memory.
+
+## Measurement protocol
+
+Measurements use one RTX 5090 rented at $0.69/hour. Reported generation time excludes checkpoint loading unless explicitly named otherwise. First-request and warm same-process results are separate. NVML peak is device memory sampled during generation; PyTorch allocated/reserved values are different metrics. Full songs may have different semantic sequences and durations even with the same prompt and seed. The controlled 60-second acoustic comparison replays identical 1500 semantic tokens and initial noise.
+
+See `evaluation/` for exact metrics, hashes, source pins and the clean-load proof. The published runtime was downloaded into a separate HF cache and executed with the original checkpoint directory unavailable. All packed module counts and absence of decoded caches were asserted. Model checkpoint SHA256: `e169774b8e79f423e6e8586221558acd28a35bacc67aae912d9a8a905231134a`.
+
+## Quality and limitations
+
+For the controlled Russian acoustic pair, latent cosine similarity was 0.97822, waveform correlation 0.90434, and spectral convergence 0.16642. On 128 teacher-forced AR positions, mean KL was 0.03131 nats and top-1 agreement 82.81%. These are diagnostics, not perceptual scores.
+
+Whisper-large-v3-turbo measured WER 13.87% for the original Russian song and 5.84% for the quantized song against the supplied lyrics. This is one example with different generated music and an imperfect automatic recognizer; it does not show a general quality improvement. Listen to both samples. MP3 previews are lossy encodes without loudness normalization or other postprocessing.
+
+Earlier unoptimized packed execution was substantially slower than BF16, illustrating why packing alone is insufficient. The included kernel/runtime changes are required for the reported profile. Low-memory hardware feasibility is not established solely by the measured peak; loading and graph capture can have separate requirements.
+
+## Provenance and license
+
+Source pins are in `source-lock.json`; runtime changes are in `runtime/OPTIMIZATIONS.md`; standalone kernel sources and tests are in `kernel-source/`. OrbitQuant 0.9.2 is included as a wheel built from its pinned source revision. Original licenses and third-party notices are retained.
+
+Model weights are **CC BY-NC 4.0**, inherited from YuE2. This release does not grant commercial rights. Runtime and kernel components retain their respective code licenses. Credit M-A-P / YuE2 and OrbitQuant when using this derivative.

evaluation/ar-fused-fixed.json

@@ -0,0 +1,33 @@
+{
+  "model": "models/YuE2-OrbitQuant-W4A4",
+  "gemv": true,
+  "fuse_packed": true,
+  "replay": "benchmark/ru-original-full/run-00",
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "results": [
+    {
+      "repeat": 0,
+      "prefill_seconds": 1.1080292710103095,
+      "wall_seconds": 2.194309923099354,
+      "cuda_ms": 2194.27587890625,
+      "tokens": 512,
+      "tps": 233.33075907382553
+    },
+    {
+      "repeat": 1,
+      "prefill_seconds": 0.14538272097706795,
+      "wall_seconds": 2.1817037297878414,
+      "cuda_ms": 2181.6689453125,
+      "tokens": 512,
+      "tps": 234.67897726415362
+    },
+    {
+      "repeat": 2,
+      "prefill_seconds": 0.14446253678761423,
+      "wall_seconds": 2.199943969026208,
+      "cuda_ms": 2199.907958984375,
+      "tokens": 512,
+      "tps": 232.7332001217439
+    }
+  ]
+}

evaluation/ar-gemv-fixed.json

@@ -0,0 +1,33 @@
+{
+  "model": "models/YuE2-OrbitQuant-W4A4",
+  "gemv": true,
+  "fuse_packed": false,
+  "replay": "benchmark/ru-original-full/run-00",
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "results": [
+    {
+      "repeat": 0,
+      "prefill_seconds": 1.698816369054839,
+      "wall_seconds": 2.863719446817413,
+      "cuda_ms": 2863.684326171875,
+      "tokens": 512,
+      "tps": 178.78846357278812
+    },
+    {
+      "repeat": 1,
+      "prefill_seconds": 0.17642259295098484,
+      "wall_seconds": 2.8788871839642525,
+      "cuda_ms": 2878.85400390625,
+      "tokens": 512,
+      "tps": 177.84649667826565
+    },
+    {
+      "repeat": 2,
+      "prefill_seconds": 0.17390872887335718,
+      "wall_seconds": 2.814904622035101,
+      "cuda_ms": 2814.87451171875,
+      "tokens": 512,
+      "tps": 181.88893363990346
+    }
+  ]
+}

evaluation/ar-original-fixed.json

@@ -0,0 +1,33 @@
+{
+  "model": "models/YuE2-3B",
+  "gemv": false,
+  "fuse_packed": false,
+  "replay": "benchmark/ru-original-full/run-00",
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "results": [
+    {
+      "repeat": 0,
+      "prefill_seconds": 0.4288525169249624,
+      "wall_seconds": 2.5710276649333537,
+      "cuda_ms": 2570.99365234375,
+      "tokens": 512,
+      "tps": 199.14215898305866
+    },
+    {
+      "repeat": 1,
+      "prefill_seconds": 0.09946558880619705,
+      "wall_seconds": 2.5770173692144454,
+      "cuda_ms": 2576.986328125,
+      "tokens": 512,
+      "tps": 198.6792972823747
+    },
+    {
+      "repeat": 2,
+      "prefill_seconds": 0.0979860769584775,
+      "wall_seconds": 2.5706858979538083,
+      "cuda_ms": 2570.656005859375,
+      "tokens": 512,
+      "tps": 199.16863449071596
+    }
+  ]
+}

evaluation/asr-source.json

@@ -0,0 +1,4 @@
+{
+  "repo": "openai/whisper-large-v3-turbo",
+  "revision": "41f01f3fe87f28c78e2fbf8b568835947dd65ed9"
+}

evaluation/harness/ar_bench.py

@@ -0,0 +1,37 @@
+"""Identical prefix and forced token trace for fair AR runtime measurements."""
+import argparse,json,time,gc
+from pathlib import Path
+import numpy as np
+import torch
+from yue2.modeling_yue2 import YuE2ForCausalLM
+from yue2.cuda_graph import GraphAR
+from yue2.protocol import CODEC_OFFSET
+from yue2_orbit import load_packed
+
+@torch.inference_mode()
+def main():
+    p=argparse.ArgumentParser();p.add_argument('--model',default='models/YuE2-3B');p.add_argument('--replay',default='benchmark/ru-original-full/run-00');p.add_argument('--output',required=True);p.add_argument('--gemv',action='store_true');p.add_argument('--fuse-packed',action='store_true');p.add_argument('--tokens',type=int,default=512);a=p.parse_args()
+    if a.gemv:
+        from enable_gemv import enable
+        enable()
+    if (Path(a.model)/'orbitquant_manifest.json').exists():
+        model,manifest=load_packed(a.model)
+        if a.fuse_packed:
+            from fuse_packed import fuse_model
+            fuse_model(model,manifest)
+    else:model=YuE2ForCausalLM.from_pretrained(a.model,torch_dtype=torch.bfloat16).cuda().eval()
+    prefix=np.load(Path(a.replay)/'prefix.npy').tolist();tokens=np.load(Path(a.replay)/'semantic.npy').tolist()[:a.tokens]
+    results=[]
+    for repeat in range(3):
+        g=GraphAR(model,[prefix],len(tokens)+1,capture=True)
+        t=time.perf_counter();g.prefill();torch.cuda.synchronize();prefill=time.perf_counter()-t
+        start,end=torch.cuda.Event(enable_timing=True),torch.cuda.Event(enable_timing=True)
+        torch.cuda.synchronize();t=time.perf_counter();start.record()
+        for token in tokens:g.step(int(token)+CODEC_OFFSET)
+        end.record();end.synchronize();wall=time.perf_counter()-t
+        results.append(dict(repeat=repeat,prefill_seconds=prefill,wall_seconds=wall,cuda_ms=start.elapsed_time(end),tokens=len(tokens),tps=len(tokens)/wall))
+        g.close()
+    value=dict(model=a.model,gemv=a.gemv,fuse_packed=a.fuse_packed,replay=a.replay,gpu=torch.cuda.get_device_name(),results=results)
+    Path(a.output).write_text(json.dumps(value,indent=2));print(json.dumps(value),flush=True)
+
+if __name__=='__main__':main()

evaluation/harness/ar_quality.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+"""Teacher-forced next-token KL on a saved baseline sequence, never sampled paths."""
+import argparse
+import gc
+import json
+from pathlib import Path
+import numpy as np
+import torch
+from yue2.modeling_yue2 import YuE2ForCausalLM
+from yue2.protocol import CODEC_OFFSET,CODEC_SIZE,MUSIC_END
+from yue2_orbit import load_packed
+
+
+@torch.inference_mode()
+def logits(model,prefix,tokens):
+    # Position 0 predicts first semantic token; every later input is from baseline.
+    ids=torch.tensor([prefix+[t+CODEC_OFFSET for t in tokens[:127]]],device='cuda')
+    result=model(ids,logits_to_keep=128,use_cache=False).logits[0].float()
+    return torch.cat([result[:,MUSIC_END:MUSIC_END+1],result[:,CODEC_OFFSET:CODEC_OFFSET+CODEC_SIZE]],dim=1).cpu()
+
+
+@torch.inference_mode()
+def main():
+    p=argparse.ArgumentParser(); p.add_argument('--replay',default='benchmark/original-full/run-00'); p.add_argument('--candidate',default='models/YuE2-OrbitQuant-W4A4'); p.add_argument('--output',default='reports/ar-quality.json'); p.add_argument('--activation-bits',type=int); a=p.parse_args()
+    root=Path(a.replay); prefix=np.load(root/'prefix.npy').tolist(); tokens=np.load(root/'semantic.npy').tolist()
+    if len(tokens)<128: raise ValueError('At least 128 baseline semantic tokens required')
+    model=YuE2ForCausalLM.from_pretrained('models/YuE2-3B',torch_dtype=torch.bfloat16,local_files_only=True).cuda().eval()
+    ref=logits(model,prefix,tokens); del model; gc.collect(); torch.cuda.empty_cache()
+    model,manifest=load_packed(a.candidate,activation_bits=a.activation_bits)
+    quant=logits(model,prefix,tokens)
+    lr=ref.log_softmax(-1); lq=quant.log_softmax(-1)
+    kl=(lr.exp()*(lr-lq)).sum(-1)
+    targets=torch.tensor(tokens[:128])+1
+    result=dict(candidate=a.candidate,activation_bits=a.activation_bits or manifest['config']['activation_bits'],positions=128,
+                teacher_forced_kl_mean=float(kl.mean()),teacher_forced_kl_max=float(kl.max()),
+                top1_agreement=float((ref.argmax(-1)==quant.argmax(-1)).float().mean()),
+                source_nll=float(-lr.gather(1,targets[:,None]).mean()),candidate_nll=float(-lq.gather(1,targets[:,None]).mean()),
+                interpretation='128 positions from one held baseline prefix; numerical diagnostic, not a general quality benchmark')
+    Path(a.output).write_text(json.dumps(result,indent=2)); print(json.dumps(result),flush=True)
+
+if __name__=='__main__': main()

evaluation/harness/asr_evaluate.py

@@ -0,0 +1,42 @@
+"""Whisper lyric transcription diagnostic; no perceptual-quality claims."""
+import argparse,json,re,time
+from pathlib import Path
+import numpy as np
+import soundfile as sf
+from scipy.signal import resample_poly
+import torch
+from transformers import AutoModelForSpeechSeq2Seq,AutoProcessor
+
+def words(text):
+    text=re.sub(r'\[[^\]]*\]',' ',text).lower().replace('ё','е')
+    return re.findall(r'[a-zа-я0-9]+',text)
+
+def distance(a,b):
+    prev=list(range(len(b)+1))
+    for i,x in enumerate(a,1):
+        row=[i]
+        for j,y in enumerate(b,1):row.append(min(row[-1]+1,prev[j]+1,prev[j-1]+(x!=y)))
+        prev=row
+    return prev[-1]
+
+@torch.inference_mode()
+def main():
+    p=argparse.ArgumentParser();p.add_argument('runs',nargs='+');p.add_argument('--output',required=True);a=p.parse_args()
+    model=AutoModelForSpeechSeq2Seq.from_pretrained('models/whisper-large-v3-turbo',torch_dtype=torch.float16,attn_implementation='sdpa').cuda().eval()
+    processor=AutoProcessor.from_pretrained('models/whisper-large-v3-turbo')
+    reference=words(json.loads(Path('prompts/default-ru.json').read_text())['lyrics']);report=[]
+    for run in a.runs:
+        signal,sr=sf.read(Path(run)/'audio.wav',dtype='float32');signal=signal.mean(axis=1)
+        from math import gcd
+        g=gcd(sr,16000);signal=resample_poly(signal,16000//g,sr//g)
+        inputs=processor(signal,sampling_rate=16000,return_tensors='pt',return_attention_mask=True,truncation=False,padding='longest')
+        inputs={k:v.cuda().to(torch.float16) if k=='input_features' else v.cuda() for k,v in inputs.items()}
+        start=time.perf_counter()
+        ids=model.generate(**inputs,language='ru',task='transcribe',return_timestamps=True)
+        text=processor.batch_decode(ids,skip_special_tokens=True)[0]
+        hypothesis=words(text)
+        item=dict(run=run,transcript=text,reference_words=len(reference),transcript_words=len(hypothesis),word_error_rate=distance(reference,hypothesis)/len(reference),seconds=time.perf_counter()-start)
+        report.append(item);print(json.dumps(item,ensure_ascii=False),flush=True)
+        Path(a.output).write_text(json.dumps(dict(results=report,interpretation='Whisper ASR is imperfect on singing; word error rate is a lyric intelligibility diagnostic, not a music-quality score.'),ensure_ascii=False,indent=2))
+
+if __name__=='__main__':main()

evaluation/harness/benchmark.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Original/packed YuE2 measurements with replayable semantic and noise inputs."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import threading
+import time
+
+import numpy as np
+import torch
+import pynvml
+
+from yue2 import YuE2Pipeline
+from yue2.pipeline import SemanticResult, SymbolicPlan
+
+
+class Monitor:
+    def __init__(self):
+        pynvml.nvmlInit()
+        self.handle=pynvml.nvmlDeviceGetHandleByIndex(0)
+        self.stop=threading.Event()
+        self.peak=0
+        self.thread=threading.Thread(target=self.run,daemon=True)
+    def run(self):
+        while not self.stop.is_set():
+            self.peak=max(self.peak,pynvml.nvmlDeviceGetMemoryInfo(self.handle).used)
+            self.stop.wait(.05)
+    def __enter__(self):
+        torch.cuda.reset_peak_memory_stats()
+        self.thread.start()
+        return self
+    def __exit__(self,*args):
+        self.stop.set(); self.thread.join()
+
+
+def digest(array):
+    return hashlib.sha256(np.ascontiguousarray(array).tobytes()).hexdigest()
+
+
+def main():
+    p=argparse.ArgumentParser()
+    p.add_argument('--model',default='models/YuE2-3B')
+    p.add_argument('--vae',default='models/YuE2-Vae')
+    p.add_argument('--prompt',default='prompts/default-ru.json')
+    p.add_argument('--output',required=True)
+    p.add_argument('--replay')
+    p.add_argument('--frames',type=int)
+    p.add_argument('--repeat',type=int,default=1)
+    p.add_argument('--activation-bits',type=int)
+    p.add_argument('--gemv',action='store_true')
+    p.add_argument('--fuse-packed',action='store_true')
+    p.add_argument('--nar-graph',action='store_true')
+    p.add_argument('--trim-cache',action='store_true')
+    p.add_argument('--offload-ar',action='store_true')
+    a=p.parse_args()
+    if a.nar_graph:
+        from nar_graph import enable as enable_nar
+        enable_nar()
+    if a.gemv:
+        from enable_gemv import enable
+        enable()
+    out=Path(a.output); out.mkdir(parents=True,exist_ok=True)
+    if (Path(a.model)/'orbitquant_manifest.json').exists():
+        from yue2_orbit import OrbitPipeline
+        pipe=OrbitPipeline.from_pretrained(a.model,vae=a.vae,device='cuda',memory_budget_gib=30,progress=False)
+        pipe.fused=a.fuse_packed
+        pipe.trim_cache=a.trim_cache
+        if a.activation_bits is not None:
+            pipe.activation_bits=a.activation_bits
+    else:
+        pipe=YuE2Pipeline.from_pretrained(a.model,vae=a.vae,device='cuda',memory_budget_gib=30,progress=False)
+    pipe.offload_ar=a.offload_ar
+    prompt=json.loads(Path(a.prompt).read_text())
+    request={k:prompt[k] for k in ('style','lyrics','seed','cot') if k in prompt}
+    print('loading model',flush=True)
+    t=time.perf_counter(); pipe._load_model(); torch.cuda.synchronize(); load=time.perf_counter()-t
+    for index in range(a.repeat):
+        dest=out/f'run-{index:02d}'; dest.mkdir(exist_ok=True)
+        print(f'generation {index} start replay={a.replay} frames={a.frames}',flush=True)
+        with Monitor() as monitor:
+            torch.cuda.synchronize(); start=time.perf_counter()
+            if a.replay:
+                original=Path(a.replay)
+                plan=SymbolicPlan.load(original)
+                tokens=np.load(original/'semantic.npy').tolist()
+                if a.frames: tokens=tokens[:a.frames]
+                semantic=SemanticResult(plan,tokens,{},False)
+                nar_start=time.perf_counter(); latents=pipe.synthesize(semantic); torch.cuda.synchronize()
+                nar_seconds=time.perf_counter()-nar_start
+                vae_start=time.perf_counter(); audio=pipe.decode(latents); torch.cuda.synchronize()
+                timing=dict(nar_seconds=nar_seconds,vae_seconds=time.perf_counter()-vae_start,
+                            e2e_seconds=time.perf_counter()-start)
+                from yue2.pipeline import SongResult
+                song=SongResult(audio,48000,semantic,latents,pipe.effective_config(plan.request),pipe.weights,timing,'controlled-replay')
+            else:
+                song=pipe(**request)
+                torch.cuda.synchronize()
+            elapsed=time.perf_counter()-start
+        song.save_artifacts(dest)
+        song.save(dest/'audio.wav')
+        noise=torch.randn((len(song.semantic.tokens),64),generator=torch.Generator(device='cpu').manual_seed(song.semantic.plan.request.seed),dtype=torch.float32).numpy()
+        np.save(dest/'initial_noise.npy',noise)
+        metrics=dict(run=index,wall_seconds=elapsed,load_seconds=load,audio_seconds=len(song.audio)/48000,
+                     rtf=elapsed/(len(song.audio)/48000),nvml_peak_bytes=monitor.peak,
+                     torch_peak_allocated_bytes=torch.cuda.max_memory_allocated(),torch_peak_reserved_bytes=torch.cuda.max_memory_reserved(),
+                     gpu=torch.cuda.get_device_name(),torch=torch.__version__,cuda=torch.version.cuda,
+                     timing=song.timing,truncated=song.truncated,finite=bool(np.isfinite(song.audio).all()),
+                     audio_peak=float(np.abs(song.audio).max()),audio_rms=float(np.sqrt(np.mean(song.audio**2))),
+                     clipped_fraction=float(np.mean(np.abs(song.audio)>=1)),semantic_sha256=digest(np.asarray(song.semantic.tokens,dtype=np.int32)),
+                     noise_sha256=digest(noise),latent_sha256=digest(song.latents),waveform_sha256=digest(song.audio),
+                     source_or_quantized_model=str(a.model),replay=a.replay,frames=a.frames,gemv=a.gemv,fuse_packed=a.fuse_packed,nar_graph=a.nar_graph,trim_cache=a.trim_cache,offload_ar=a.offload_ar,
+                     measurement='first process request' if index==0 else 'warm same-process request',pid=os.getpid())
+        if hasattr(pipe,'runtime_report'): metrics['packed_runtime']=pipe.runtime_report()
+        (dest/'metrics.json').write_text(json.dumps(metrics,indent=2))
+        print(json.dumps(metrics),flush=True)
+        del song
+    pipe.close()
+
+if __name__=='__main__': main()

evaluation/harness/clean_validate.py

@@ -0,0 +1,59 @@
+"""Run the downloaded release with the original checkpoint path unavailable."""
+from pathlib import Path
+import json,os,sys,time,importlib.util,hashlib
+from huggingface_hub import snapshot_download
+import numpy as np
+import torch
+
+root=Path('/workspace/yue2')
+info=json.loads((root/'reports/candidate-hub-final-runtime.json').read_text())
+folder=Path(snapshot_download(info['repo'],revision=info['revision'],cache_dir=str(root/'clean-hf-cache')))
+manifest=json.loads((folder/'weights_manifest.json').read_text())
+def sha(path):
+    h=hashlib.sha256()
+    with path.open('rb') as f:
+        for block in iter(lambda:f.read(8*1024*1024),b''):h.update(block)
+    return h.hexdigest()
+assert sha(folder/'model.safetensors')==manifest['files']['model.safetensors']['sha256']
+original=root/'models/YuE2-3B';hidden=root/'models/YuE2-3B-clean-test-unavailable'
+assert original.exists() and not hidden.exists()
+original.rename(hidden)
+try:
+    spec=importlib.util.spec_from_file_location('downloaded_release',folder/'run.py')
+    release=importlib.util.module_from_spec(spec);spec.loader.exec_module(release)
+    # VAE is also resolved by the released entrypoint at its pinned Hub revision.
+    pipe=release.load_pipeline(progress=False)
+    from benchmark import Monitor,digest
+    request=json.loads((folder/'prompts/default-ru.json').read_text())
+    start=time.perf_counter();pipe._load_model();torch.cuda.synchronize();load_seconds=time.perf_counter()-start
+    expected=json.loads((root/'benchmark/ru-final-w4a4-full/run-01/metrics.json').read_text())
+    results=[]
+    for i in range(2):
+        out=root/f'benchmark/clean-release/run-{i:02d}';out.mkdir(parents=True,exist_ok=True)
+        with Monitor() as monitor:
+            torch.cuda.synchronize();start=time.perf_counter()
+            song=pipe(**{k:request[k] for k in ['style','lyrics','cot','seed']})
+            torch.cuda.synchronize();elapsed=time.perf_counter()-start
+        song.save_artifacts(out);song.save(out/'audio.wav')
+        semantic=digest(np.asarray(song.semantic.tokens,dtype=np.int32));wave=digest(song.audio)
+        assert semantic==expected['semantic_sha256'],'Published runtime changed semantic generation'
+        assert wave==expected['waveform_sha256'],'Published runtime changed waveform'
+        runtime=pipe.runtime_report()
+        assert runtime['modules']==224 and runtime['logical_projections']==392
+        assert runtime['decoded_weight_caches']==0 and runtime['int8_weight_caches']==0
+        assert runtime['effective_modes']=={'native_packed_matmul':224}
+        report=dict(run=i,repo=info['repo'],revision=info['revision'],model_sha256=sha(folder/'model.safetensors'),
+            source_checkpoint_path_unavailable=not original.exists(),module_file=sys.modules['yue2'].__file__,
+            wall_seconds=elapsed,load_seconds=load_seconds,audio_seconds=len(song.audio)/48000,
+            rtf=elapsed/(len(song.audio)/48000),nvml_peak_bytes=monitor.peak,
+            torch_peak_allocated_bytes=torch.cuda.max_memory_allocated(),torch_peak_reserved_bytes=torch.cuda.max_memory_reserved(),
+            semantic_sha256=semantic,waveform_sha256=wave,packed_runtime=runtime,config=song.config,timing=song.timing,
+            finite=bool(np.isfinite(song.audio).all()),truncated=song.truncated,
+            measurement='first process request' if i==0 else 'warm same-process request')
+        (out/'metrics.json').write_text(json.dumps(report,indent=2))
+        results.append(report);print(json.dumps(report),flush=True)
+        del song
+    pipe.close()
+    (root/'reports/clean-load-proof.json').write_text(json.dumps(dict(status='passed',runs=results),indent=2))
+finally:
+    hidden.rename(original)

evaluation/harness/satellite.py

@@ -0,0 +1,29 @@
+#!/usr/bin/env python3
+"""A one-shot <60 second remote job/liveness snapshot, never a daemon."""
+import argparse
+import json
+from pathlib import Path
+import subprocess
+import time
+
+p=argparse.ArgumentParser()
+p.add_argument("job")
+a=p.parse_args()
+job=Path(a.job)
+start=time.monotonic()
+for offset in (0,15,30,45):
+    time.sleep(max(0,start+offset-time.monotonic()))
+    value=json.loads((job/'status.json').read_text()) if (job/'status.json').exists() else {"status":"missing"}
+    alive=False
+    if value.get('pid'):
+        result=subprocess.run(['ps','-p',str(value['pid']),'-o','pid=,stat=,etime=,pcpu=,pmem=,comm='],capture_output=True,text=True)
+        alive=result.returncode==0
+        value['process']=result.stdout.strip()
+    log=job/'output.log'
+    value['log_age_seconds']=round(time.time()-log.stat().st_mtime,1) if log.exists() else None
+    value['log_tail']=subprocess.run(['tail','-n','8' if value['status']!='running' else '1',str(log)],capture_output=True,text=True).stdout.strip()
+    value['gpu']=subprocess.run(['nvidia-smi','--query-gpu=name,utilization.gpu,memory.used,power.draw','--format=csv,noheader'],capture_output=True,text=True,timeout=5).stdout.strip()
+    value['alive']=alive
+    print(json.dumps(value),flush=True)
+    if value['status']!='running' or not alive:
+        break

evaluation/harness/supervise.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Detached child runner with unbuffered minute heartbeats and atomic status."""
+import argparse
+import json
+import os
+from pathlib import Path
+import subprocess
+import time
+
+p = argparse.ArgumentParser()
+p.add_argument("--job", required=True)
+p.add_argument("command", nargs=argparse.REMAINDER)
+a = p.parse_args()
+job = Path(a.job)
+job.mkdir(parents=True, exist_ok=True)
+command = a.command[1:] if a.command[:1] == ["--"] else a.command
+started = time.time()
+with (job / "output.log").open("a", buffering=1) as log:
+    child = subprocess.Popen(command, stdout=log, stderr=subprocess.STDOUT,
+                             env={**os.environ, "PYTHONUNBUFFERED": "1"})
+    def status():
+        code = child.poll()
+        value = dict(status="running" if code is None else "complete" if code == 0 else "failed",
+                     pid=child.pid, supervisor_pid=os.getpid(), started=started,
+                     updated=time.time(), elapsed_seconds=time.time()-started, exit_code=code)
+        temporary = job / "status.tmp"
+        temporary.write_text(json.dumps(value, indent=2))
+        temporary.replace(job / "status.json")
+        print(f"heartbeat status={value['status']} pid={child.pid} elapsed={value['elapsed_seconds']:.0f}s exit={code}", flush=True)
+        return code
+    while status() is None:
+        try:
+            child.wait(timeout=60)
+        except subprocess.TimeoutExpired:
+            pass
+    raise SystemExit(child.returncode)

evaluation/harness/test_fused.py

@@ -0,0 +1,27 @@
+import torch
+from orbitquant import OrbitQuantConfig
+from orbitquant.layers import OrbitQuantLinear
+from test_packed_graph import small_model
+from fuse_packed import fuse_model
+from yue2.cuda_graph import GraphAR
+from enable_gemv import enable
+
+def test_fused_graph_and_storage():
+    enable()
+    model=small_model()
+    before=sum(m.packed_weight_indices.numel() for m in model.modules() if isinstance(m,OrbitQuantLinear))
+    g=GraphAR(model,[[1,2,3]],8,capture=True)
+    expected=[g.prefill().clone()]+[g.step(t).clone() for t in (4,5,6,7)]
+    g.close()
+    fuse_model(model,{'config':OrbitQuantConfig().to_dict()})
+    after=sum(m.packed_weight_indices.numel() for m in model.modules() if isinstance(m,OrbitQuantLinear))
+    assert before==after
+    g=GraphAR(model,[[1,2,3]],8,capture=True)
+    actual=[g.prefill().clone()]+[g.step(t).clone() for t in (4,5,6,7)]
+    for a,b in zip(actual,expected):torch.testing.assert_close(a,b,rtol=0,atol=0)
+    g.close()
+    # Both device transfers retain one owner and correct compatibility views.
+    model.cpu().cuda()
+    for layer in model.model.layers:
+        assert layer.self_attn.q_proj._group is layer.self_attn.packed_qkv
+        assert layer.mlp.up_proj._group is layer.mlp.packed_gate_up

evaluation/harness/test_gemv.py

@@ -0,0 +1,35 @@
+import json
+from pathlib import Path
+import pytest
+import torch
+from kernels import get_local_kernel
+from orbitquant.kernels.native_packed_matmul import matmul_packed_w4a4_int8_with_native_kernel as original
+
+kernel=get_local_kernel(Path('/workspace/yue2/orbitquant-gemv/build'), 'orbitquant_gemv')
+
+@pytest.mark.parametrize('rows,n,k',[(1,2048,2048),(2,6144,2048),(8,2048,6144),(1,129,128),(2,1024,2048)])
+@pytest.mark.parametrize('dtype',[torch.bfloat16,torch.float16])
+def test_exact_integer_gemv(rows,n,k,dtype):
+    torch.manual_seed(123)
+    x=torch.randint(0,256,(rows,k//2),device='cuda',dtype=torch.uint8)
+    w=torch.randint(0,256,(n*k//2,),device='cuda',dtype=torch.uint8)
+    xn=torch.rand(rows,device='cuda');wn=torch.rand(n,device='cuda',dtype=torch.bfloat16)
+    ac=torch.arange(-8,8,device='cuda',dtype=torch.int8);wc=ac.flip(0).contiguous()
+    bias=None
+    kw=dict(activation_scale=.03125,weight_scale=.0625,bias=bias,output_dtype=dtype)
+    ref=original(x,w,xn,wn,ac,wc,out_features=n,in_features=k,**kw)
+    out=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
+    torch.testing.assert_close(out,ref,rtol=0,atol=0)
+    stream=torch.cuda.Stream();stream.wait_stream(torch.cuda.current_stream())
+    with torch.cuda.stream(stream):
+        for _ in range(3):kernel.gemv(x,w,xn,wn,ac,wc,**kw)
+    torch.cuda.current_stream().wait_stream(stream)
+    graph=torch.cuda.CUDAGraph()
+    with torch.cuda.graph(graph): captured=kernel.gemv(x,w,xn,wn,ac,wc,**kw)
+    graph.replay();torch.cuda.synchronize()
+    torch.testing.assert_close(captured,ref,rtol=0,atol=0)
+
+def test_rejects_bad_weight_shape():
+    x=torch.zeros(1,64,device='cuda',dtype=torch.uint8)
+    with pytest.raises(RuntimeError,match='packed weight size'):
+        kernel.gemv(x,x.flatten(),torch.ones(1,device='cuda'),torch.ones(2,device='cuda',dtype=torch.bfloat16),torch.zeros(16,device='cuda',dtype=torch.int8),torch.zeros(16,device='cuda',dtype=torch.int8),activation_scale=1,weight_scale=1)

evaluation/harness/test_packed_graph.py

@@ -0,0 +1,54 @@
+"""The packed graph must match its eager path across KV updates and reload."""
+import torch
+from yue2.modeling_yue2 import YuE2Config, YuE2ForCausalLM
+from yue2.cuda_graph import GraphAR
+from orbitquant import OrbitQuantConfig
+from orbitquant.layers import OrbitQuantLinear
+
+
+def small_model():
+    torch.manual_seed(42)
+    config=YuE2Config(hidden_size=128,intermediate_size=256,num_hidden_layers=2,
+                      num_attention_heads=4,num_key_value_heads=2,head_dim=32,
+                      vocab_size=256,max_position_embeddings=128,max_latent_frames=128)
+    model=YuE2ForCausalLM(config).to(device='cuda',dtype=torch.bfloat16).eval()
+    for name,module in list(model.named_modules()):
+        if isinstance(module,torch.nn.Linear) and name.startswith('model.layers.'):
+            parent,leaf=name.rsplit('.',1)
+            model.get_submodule(parent)._modules[leaf]=OrbitQuantLinear.from_linear(module,config=OrbitQuantConfig(),module_name=name)
+    return model
+
+
+def test_packed_graph_matches_eager_fixed_tokens():
+    model=small_model()
+    graph=GraphAR(model,[[1,2,3]],8,capture=True)
+    eager=GraphAR(model,[[1,2,3]],8,capture=False)
+    try:
+        torch.testing.assert_close(graph.prefill(),eager.prefill(),rtol=0,atol=0)
+        for token in (4,5,6,7):
+            torch.testing.assert_close(graph.step(token).clone(),eager.step(token),rtol=0,atol=0)
+    finally:
+        graph.close(); eager.close()
+
+
+def test_packed_graph_rejects_bf16_fusion():
+    import pytest
+    with pytest.raises(ValueError,match='packed'):
+        GraphAR(small_model(),[[1,2]],4,capture=False,fuse_projections=True)
+
+
+def test_packed_reload_preserves_logits(tmp_path):
+    import json
+    from safetensors.torch import save_file
+    from yue2_orbit import load_packed
+    model=small_model()
+    modules=[dict(name=name,in_features=m.in_features,out_features=m.out_features,bias=m.bias is not None)
+             for name,m in model.named_modules() if isinstance(m,OrbitQuantLinear)]
+    (tmp_path/'config.json').write_text(json.dumps(model.config.to_dict()))
+    (tmp_path/'orbitquant_manifest.json').write_text(json.dumps(dict(config=OrbitQuantConfig().to_dict(),modules=modules)))
+    save_file({k:v.detach().cpu().contiguous() for k,v in model.state_dict().items()},str(tmp_path/'model.safetensors'))
+    restored,_=load_packed(tmp_path)
+    ids=torch.tensor([[1,2,3]],device='cuda')
+    with torch.inference_mode():
+        torch.testing.assert_close(model(ids).logits,restored(ids).logits,rtol=0,atol=0)
+    assert all(m._dequantized_weight_cache is None for m in restored.modules() if isinstance(m,OrbitQuantLinear))

evaluation/harness/vae_bench.py

@@ -0,0 +1,36 @@
+"""Measure the existing VAE chunk-size knob independently of transformer changes."""
+import json,time
+from pathlib import Path
+import numpy as np
+import torch
+import soundfile as sf
+from yue2.modeling_vae import YuE2VAE
+from benchmark import Monitor
+
+@torch.inference_mode()
+def main():
+    # Match the pinned pipeline's numerical settings, including convolution TF32.
+    torch.backends.cudnn.benchmark=False
+    torch.backends.cudnn.deterministic=True
+    torch.backends.cuda.matmul.allow_tf32=False
+    torch.backends.cudnn.allow_tf32=False
+    torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction=False
+    torch.set_float32_matmul_precision('highest')
+    model=YuE2VAE.from_pretrained('models/YuE2-Vae',decoder_only=True,device='cuda',local_files_only=True)
+    root=Path('benchmark/ru-final-w4a4-full/run-01')
+    z=torch.from_numpy(np.load(root/'latent.npy')).T.unsqueeze(0)
+    reference,sr=sf.read(root/'audio.wav',dtype='float32')
+    results=[]
+    for frames in (1024,512,256):
+        for repeat in range(3):
+            torch.cuda.empty_cache()
+            with Monitor() as monitor:
+                torch.cuda.synchronize();start=time.perf_counter()
+                audio=model.decode_tiled(z,core_frames=frames,halo_frames=16,output_device='cpu')[0].float().clamp(-1,1).T.contiguous().numpy()
+                torch.cuda.synchronize();elapsed=time.perf_counter()-start
+            diff=audio-reference
+            row=dict(pipeline_numerics=True,core_frames=frames,repeat=repeat,seconds=elapsed,nvml_peak_bytes=monitor.peak,torch_peak_allocated_bytes=torch.cuda.max_memory_allocated(),max_abs_difference=float(np.abs(diff).max()),relative_l2=float(np.linalg.norm(diff)/(np.linalg.norm(reference)+1e-12)),waveform_correlation=float(np.corrcoef(audio.ravel(),reference.ravel())[0,1]))
+            results.append(row);print(json.dumps(row),flush=True)
+    Path('reports/vae-chunk-benchmark.json').write_text(json.dumps(results,indent=2))
+
+if __name__=='__main__': main()

evaluation/ru-ar-quality.json

@@ -0,0 +1,11 @@
+{
+  "candidate": "models/YuE2-OrbitQuant-W4A4",
+  "activation_bits": 4,
+  "positions": 128,
+  "teacher_forced_kl_mean": 0.031312376260757446,
+  "teacher_forced_kl_max": 0.12084035575389862,
+  "top1_agreement": 0.828125,
+  "source_nll": 3.8141632080078125,
+  "candidate_nll": 3.8379945755004883,
+  "interpretation": "128 positions from one held baseline prefix; numerical diagnostic, not a general quality benchmark"
+}

evaluation/ru-asr.json

@@ -0,0 +1,21 @@
+{
+  "results": [
+    {
+      "run": "benchmark/ru-original-full/run-01",
+      "transcript": " Ночной трамвай Стираю с улиц тишину И этот город слышит нас одних Пускай часы торопятся вперед Мы не считаем пройденных шагов Пока над крышами рассвет встает Мне хватит самых обыкновенных слов Останьтесь со мной до утра Пока не погасли огни, нам эта короткая ночь Подарит другие дни. Останься со мной до утра и просто за руку держи. Когда просыпается город, мы снова учимся жить. На остановке мокрая сканья И теплый ветер трогает руках Я столько раз искал тебя во снах Но ты стоишь и смотришь на меня Останься со мной до утра, пока не потасли огни Нам эта короткая ночь подарит другие дни Останься со мной до утра и просто за руку держи Когда просыпается город Пока просыпается город Останьте со мной до утра",
+      "reference_words": 137,
+      "transcript_words": 124,
+      "word_error_rate": 0.1386861313868613,
+      "seconds": 1.5337588579859585
+    },
+    {
+      "run": "benchmark/ru-final-w4a4-full/run-01",
+      "transcript": " Ночной трамвай уходят за мосты В стекле дрожат последние дни Я собираю с улиц тишину И этот город слышит нас одни Пускай часы торопятся вперед Мы не считаем пройденных шагов Пока над крышами рассвет встает Мне хватит самых обыкновенных слов Останьте со мной до утра, пока не погасли огни Нам эта короткая ночь подарит другие дни Останьте со мной до утра и просто за руку держи Когда просыпается город, мы снова учимся жить На остановке мокрая сканья И теплый ветер трогает рукав Я столько раз искал тебя во снах Но ты стоишь и смотришь на меня Останься со мной до утра, пока не погасли огни Нам эта короткая ночь подарит другие дни Останься со мной до утра и просто за руку держи Когда просыпается город, мы снова учимся жить Пока просыпается город Останьте за мной до утра",
+      "reference_words": 137,
+      "transcript_words": 137,
+      "word_error_rate": 0.058394160583941604,
+      "seconds": 0.9663123651407659
+    }
+  ],
+  "interpretation": "Whisper ASR is imperfect on singing; word error rate is a lyric intelligibility diagnostic, not a music-quality score."
+}

evaluation/ru-final-w4a4-full/run-00/metrics.json

@@ -0,0 +1,82 @@
+{
+  "run": 0,
+  "wall_seconds": 45.04905349877663,
+  "load_seconds": 1.029712200164795,
+  "audio_seconds": 163.83866666666665,
+  "rtf": 0.2749598395501467,
+  "nvml_peak_bytes": 9756803072,
+  "torch_peak_allocated_bytes": 5495350272,
+  "torch_peak_reserved_bytes": 8495562752,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "abc": {
+      "seconds": 10.528724248055369,
+      "prefill_seconds": 0.8370891781523824,
+      "ttft_seconds": 0.9959756580647081,
+      "output_tokens": 1772,
+      "content_tokens": 1771,
+      "output_tps": 168.30149201858757,
+      "prefix_tokens": 423,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "semantic": {
+      "seconds": 23.20693418988958,
+      "prefill_seconds": 0.3349543798249215,
+      "ttft_seconds": 0.33607721398584545,
+      "output_tokens": 4097,
+      "content_tokens": 4096,
+      "output_tps": 176.54206137167893,
+      "prefix_tokens": 2196,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "nar_seconds": 7.658700210042298,
+    "vae_seconds": 3.6495362292043865,
+    "load": {
+      "resolve_and_integrity_seconds": 2.4894183888100088,
+      "mot_load_seconds": 1.0296437400393188
+    },
+    "e2e_seconds": 45.04794106609188
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 0.9706120491027832,
+  "audio_rms": 0.14440932869911194,
+  "clipped_fraction": 0.0,
+  "semantic_sha256": "45d407d6c04fb19e54f29a17b622a89060240e5b8d8af4b0f3eee1fc338e5251",
+  "noise_sha256": "0f1e184cfce6b99c4e2b2f5dd27a470c894fc6a1dcbe6bd0cab6ec635d343f29",
+  "latent_sha256": "e371f6a61d2d2d13a9ff1003aeae8bf6355a62569c7707988c49f0053bd1e76f",
+  "waveform_sha256": "eceb3ad260035470803e5827bc88192819a5ca08f01536dd3d580abb4b190d54",
+  "source_or_quantized_model": "models/YuE2-OrbitQuant-W4A4",
+  "replay": null,
+  "frames": null,
+  "gemv": true,
+  "fuse_packed": true,
+  "nar_graph": false,
+  "trim_cache": true,
+  "measurement": "first process request",
+  "pid": 10795,
+  "packed_runtime": {
+    "modules": 224,
+    "logical_projections": 392,
+    "fused": true,
+    "effective_modes": {
+      "native_packed_matmul": 224
+    },
+    "activation_backends": {
+      "native_cuda_int8_surrogate": 168,
+      "triton_cuda_packed_w4": 56
+    },
+    "decoded_weight_caches": 0,
+    "int8_weight_caches": 0,
+    "packed_weight_bytes": 1409286144
+  }
+}

evaluation/ru-final-w4a4-full/run-01/metrics.json

@@ -0,0 +1,82 @@
+{
+  "run": 1,
+  "wall_seconds": 42.934636591002345,
+  "load_seconds": 1.029712200164795,
+  "audio_seconds": 163.83866666666665,
+  "rtf": 0.2620543578907035,
+  "nvml_peak_bytes": 9857466368,
+  "torch_peak_allocated_bytes": 5782598144,
+  "torch_peak_reserved_bytes": 8596226048,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "abc": {
+      "seconds": 9.711376916151494,
+      "prefill_seconds": 0.15963919600471854,
+      "ttft_seconds": 0.16055710101500154,
+      "output_tokens": 1772,
+      "content_tokens": 1771,
+      "output_tps": 182.46640155145198,
+      "prefix_tokens": 423,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "semantic": {
+      "seconds": 23.016886444063857,
+      "prefill_seconds": 0.15774337714537978,
+      "ttft_seconds": 0.1586433621123433,
+      "output_tokens": 4097,
+      "content_tokens": 4096,
+      "output_tps": 177.9997485740141,
+      "prefix_tokens": 2196,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "nar_seconds": 7.616418123943731,
+    "vae_seconds": 2.2683653261046857,
+    "load": {
+      "resolve_and_integrity_seconds": 2.4894183888100088,
+      "mot_load_seconds": 1.0296437400393188
+    },
+    "e2e_seconds": 42.93380210502073
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 0.9706120491027832,
+  "audio_rms": 0.14440932869911194,
+  "clipped_fraction": 0.0,
+  "semantic_sha256": "45d407d6c04fb19e54f29a17b622a89060240e5b8d8af4b0f3eee1fc338e5251",
+  "noise_sha256": "0f1e184cfce6b99c4e2b2f5dd27a470c894fc6a1dcbe6bd0cab6ec635d343f29",
+  "latent_sha256": "e371f6a61d2d2d13a9ff1003aeae8bf6355a62569c7707988c49f0053bd1e76f",
+  "waveform_sha256": "eceb3ad260035470803e5827bc88192819a5ca08f01536dd3d580abb4b190d54",
+  "source_or_quantized_model": "models/YuE2-OrbitQuant-W4A4",
+  "replay": null,
+  "frames": null,
+  "gemv": true,
+  "fuse_packed": true,
+  "nar_graph": false,
+  "trim_cache": true,
+  "measurement": "warm same-process request",
+  "pid": 10795,
+  "packed_runtime": {
+    "modules": 224,
+    "logical_projections": 392,
+    "fused": true,
+    "effective_modes": {
+      "native_packed_matmul": 224
+    },
+    "activation_backends": {
+      "native_cuda_int8_surrogate": 168,
+      "triton_cuda_packed_w4": 56
+    },
+    "decoded_weight_caches": 0,
+    "int8_weight_caches": 0,
+    "packed_weight_bytes": 1409286144
+  }
+}

evaluation/ru-offload-w4a4-full/run-00/metrics.json

@@ -0,0 +1,83 @@
+{
+  "run": 0,
+  "wall_seconds": 46.23842817801051,
+  "load_seconds": 1.14848616393283,
+  "audio_seconds": 163.83866666666665,
+  "rtf": 0.2822192655661902,
+  "nvml_peak_bytes": 9756803072,
+  "torch_peak_allocated_bytes": 5495350272,
+  "torch_peak_reserved_bytes": 8495562752,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "abc": {
+      "seconds": 10.56912293494679,
+      "prefill_seconds": 0.8523490249644965,
+      "ttft_seconds": 1.038547317031771,
+      "output_tokens": 1772,
+      "content_tokens": 1771,
+      "output_tps": 167.65818799787866,
+      "prefix_tokens": 423,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "semantic": {
+      "seconds": 23.106563664972782,
+      "prefill_seconds": 0.28357866406440735,
+      "ttft_seconds": 0.2846065699122846,
+      "output_tokens": 4097,
+      "content_tokens": 4096,
+      "output_tps": 177.30892656317556,
+      "prefix_tokens": 2196,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "nar_seconds": 9.069145092042163,
+    "vae_seconds": 3.4875931590795517,
+    "load": {
+      "resolve_and_integrity_seconds": 2.487326374044642,
+      "mot_load_seconds": 1.1484210339840502
+    },
+    "e2e_seconds": 46.23729805299081
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 0.9706120491027832,
+  "audio_rms": 0.14440932869911194,
+  "clipped_fraction": 0.0,
+  "semantic_sha256": "45d407d6c04fb19e54f29a17b622a89060240e5b8d8af4b0f3eee1fc338e5251",
+  "noise_sha256": "0f1e184cfce6b99c4e2b2f5dd27a470c894fc6a1dcbe6bd0cab6ec635d343f29",
+  "latent_sha256": "e371f6a61d2d2d13a9ff1003aeae8bf6355a62569c7707988c49f0053bd1e76f",
+  "waveform_sha256": "eceb3ad260035470803e5827bc88192819a5ca08f01536dd3d580abb4b190d54",
+  "source_or_quantized_model": "models/YuE2-OrbitQuant-W4A4",
+  "replay": null,
+  "frames": null,
+  "gemv": true,
+  "fuse_packed": true,
+  "nar_graph": false,
+  "trim_cache": true,
+  "offload_ar": true,
+  "measurement": "first process request",
+  "pid": 11579,
+  "packed_runtime": {
+    "modules": 224,
+    "logical_projections": 392,
+    "fused": true,
+    "effective_modes": {
+      "native_packed_matmul": 224
+    },
+    "activation_backends": {
+      "native_cuda_int8_surrogate": 168,
+      "triton_cuda_packed_w4": 56
+    },
+    "decoded_weight_caches": 0,
+    "int8_weight_caches": 0,
+    "packed_weight_bytes": 1409286144
+  }
+}

evaluation/ru-offload-w4a4-full/run-01/metrics.json

@@ -0,0 +1,83 @@
+{
+  "run": 1,
+  "wall_seconds": 44.32611982291564,
+  "load_seconds": 1.14848616393283,
+  "audio_seconds": 163.83866666666665,
+  "rtf": 0.27054736665489415,
+  "nvml_peak_bytes": 9857466368,
+  "torch_peak_allocated_bytes": 5782598144,
+  "torch_peak_reserved_bytes": 8596226048,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "abc": {
+      "seconds": 9.711860368028283,
+      "prefill_seconds": 0.15409997710958123,
+      "ttft_seconds": 0.15508739417418838,
+      "output_tokens": 1772,
+      "content_tokens": 1771,
+      "output_tps": 182.4573184591362,
+      "prefix_tokens": 423,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "semantic": {
+      "seconds": 23.071978967869654,
+      "prefill_seconds": 0.15794157283380628,
+      "ttft_seconds": 0.15890546003356576,
+      "output_tokens": 4097,
+      "content_tokens": 4096,
+      "output_tps": 177.57471111193092,
+      "prefix_tokens": 2196,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "nar_seconds": 8.86763684405014,
+    "vae_seconds": 2.319226206978783,
+    "load": {
+      "resolve_and_integrity_seconds": 2.487326374044642,
+      "mot_load_seconds": 1.1484210339840502
+    },
+    "e2e_seconds": 44.32531815697439
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 0.9706120491027832,
+  "audio_rms": 0.14440932869911194,
+  "clipped_fraction": 0.0,
+  "semantic_sha256": "45d407d6c04fb19e54f29a17b622a89060240e5b8d8af4b0f3eee1fc338e5251",
+  "noise_sha256": "0f1e184cfce6b99c4e2b2f5dd27a470c894fc6a1dcbe6bd0cab6ec635d343f29",
+  "latent_sha256": "e371f6a61d2d2d13a9ff1003aeae8bf6355a62569c7707988c49f0053bd1e76f",
+  "waveform_sha256": "eceb3ad260035470803e5827bc88192819a5ca08f01536dd3d580abb4b190d54",
+  "source_or_quantized_model": "models/YuE2-OrbitQuant-W4A4",
+  "replay": null,
+  "frames": null,
+  "gemv": true,
+  "fuse_packed": true,
+  "nar_graph": false,
+  "trim_cache": true,
+  "offload_ar": true,
+  "measurement": "warm same-process request",
+  "pid": 11579,
+  "packed_runtime": {
+    "modules": 224,
+    "logical_projections": 392,
+    "fused": true,
+    "effective_modes": {
+      "native_packed_matmul": 224
+    },
+    "activation_backends": {
+      "native_cuda_int8_surrogate": 168,
+      "triton_cuda_packed_w4": 56
+    },
+    "decoded_weight_caches": 0,
+    "int8_weight_caches": 0,
+    "packed_weight_bytes": 1409286144
+  }
+}

evaluation/ru-original-full/run-00/metrics.json

@@ -0,0 +1,64 @@
+{
+  "run": 0,
+  "wall_seconds": 51.78475882997736,
+  "load_seconds": 3.197218642104417,
+  "audio_seconds": 164.63866666666667,
+  "rtf": 0.3145358248972135,
+  "nvml_peak_bytes": 10583080960,
+  "torch_peak_allocated_bytes": 8692496896,
+  "torch_peak_reserved_bytes": 9332326400,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "abc": {
+      "seconds": 10.894125336082652,
+      "prefill_seconds": 0.48003934998996556,
+      "ttft_seconds": 0.6661999989300966,
+      "output_tokens": 1669,
+      "content_tokens": 1668,
+      "output_tps": 153.20183571526127,
+      "prefix_tokens": 423,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "semantic": {
+      "seconds": 26.13979333708994,
+      "prefill_seconds": 0.11473918193951249,
+      "ttft_seconds": 0.11574821802787483,
+      "output_tokens": 4117,
+      "content_tokens": 4116,
+      "output_tps": 157.49933241279146,
+      "prefix_tokens": 2093,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "nar_seconds": 8.817161875078455,
+    "vae_seconds": 5.922186557902023,
+    "load": {
+      "resolve_and_integrity_seconds": 4.985906707821414,
+      "mot_load_seconds": 0.2366083210799843
+    },
+    "e2e_seconds": 51.78419973095879
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 1.0,
+  "audio_rms": 0.15364815294742584,
+  "clipped_fraction": 1.3286672227666243e-06,
+  "semantic_sha256": "f37633f5006e9b80ae72a0321d663e1623e939d5963dd9493a1b592126db122e",
+  "noise_sha256": "c8aa36bd1c94b4d8d0f311f823e2e948ddc876c9951d98bc8a47e8c5b5bffb1c",
+  "latent_sha256": "752ebcaea4ffae13bfeb169655d6c84381b1db186871feffc0c0105670f16b73",
+  "waveform_sha256": "a8a6ea78c6b5dc4478b4e54d538625282ee87f3a10b4127323cc737f881e0338",
+  "source_or_quantized_model": "models/YuE2-3B",
+  "replay": null,
+  "frames": null,
+  "gemv": false,
+  "measurement": "first process request",
+  "pid": 5444
+}

evaluation/ru-original-full/run-01/metrics.json

@@ -0,0 +1,64 @@
+{
+  "run": 1,
+  "wall_seconds": 50.16780533106066,
+  "load_seconds": 3.197218642104417,
+  "audio_seconds": 164.63866666666667,
+  "rtf": 0.3047145992297921,
+  "nvml_peak_bytes": 10908139520,
+  "torch_peak_allocated_bytes": 8977385472,
+  "torch_peak_reserved_bytes": 9646899200,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "abc": {
+      "seconds": 10.320648127002642,
+      "prefill_seconds": 0.08523392281495035,
+      "ttft_seconds": 0.0860717708710581,
+      "output_tokens": 1669,
+      "content_tokens": 1668,
+      "output_tps": 161.714650035716,
+      "prefix_tokens": 423,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "semantic": {
+      "seconds": 26.154879739973694,
+      "prefill_seconds": 0.11253222916275263,
+      "ttft_seconds": 0.11347067612223327,
+      "output_tokens": 4117,
+      "content_tokens": 4116,
+      "output_tps": 157.40848518251076,
+      "prefix_tokens": 2093,
+      "cfg_branches": 1,
+      "execution": "cuda_graph",
+      "attention": "flash"
+    },
+    "nar_seconds": 8.794583374867216,
+    "vae_seconds": 4.3119754288345575,
+    "load": {
+      "resolve_and_integrity_seconds": 4.985906707821414,
+      "mot_load_seconds": 0.2366083210799843
+    },
+    "e2e_seconds": 50.16745935194194
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 1.0,
+  "audio_rms": 0.15364815294742584,
+  "clipped_fraction": 1.3286672227666243e-06,
+  "semantic_sha256": "f37633f5006e9b80ae72a0321d663e1623e939d5963dd9493a1b592126db122e",
+  "noise_sha256": "c8aa36bd1c94b4d8d0f311f823e2e948ddc876c9951d98bc8a47e8c5b5bffb1c",
+  "latent_sha256": "752ebcaea4ffae13bfeb169655d6c84381b1db186871feffc0c0105670f16b73",
+  "waveform_sha256": "a8a6ea78c6b5dc4478b4e54d538625282ee87f3a10b4127323cc737f881e0338",
+  "source_or_quantized_model": "models/YuE2-3B",
+  "replay": null,
+  "frames": null,
+  "gemv": false,
+  "measurement": "warm same-process request",
+  "pid": 5444
+}

evaluation/ru-original-replay1500/run-00/metrics.json

@@ -0,0 +1,36 @@
+{
+  "run": 0,
+  "wall_seconds": 9.195442344993353,
+  "load_seconds": 3.375524572096765,
+  "audio_seconds": 59.998666666666665,
+  "rtf": 0.15326077821162726,
+  "nvml_peak_bytes": 9165406208,
+  "torch_peak_allocated_bytes": 7884326912,
+  "torch_peak_reserved_bytes": 8009023488,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "nar_seconds": 3.627831295831129,
+    "vae_seconds": 5.565822640201077,
+    "e2e_seconds": 9.195178817026317
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 0.8817682862281799,
+  "audio_rms": 0.13392212986946106,
+  "clipped_fraction": 0.0,
+  "semantic_sha256": "105e53ffbbe671c28587ce52c917f57698087ed47bf6b63792a377e357236c6b",
+  "noise_sha256": "304490172c3b67d1dd3984d61d39e3cfc0d311ad6df79200674f815bc1081eae",
+  "latent_sha256": "b9461dadf0c0a16941fa0178ea5f3845f32d1f6bda70de4fe17541b2e81faabf",
+  "waveform_sha256": "9a80ae8267c22cd066aaf7cd1f7e5a12c7fcaedfb3a2e0b5f80b95ee2a47e482",
+  "source_or_quantized_model": "models/YuE2-3B",
+  "replay": "benchmark/ru-original-full/run-00",
+  "frames": 1500,
+  "gemv": false,
+  "measurement": "first process request",
+  "pid": 6449
+}

evaluation/ru-original-replay1500/run-01/metrics.json

@@ -0,0 +1,36 @@
+{
+  "run": 1,
+  "wall_seconds": 7.551422053948045,
+  "load_seconds": 3.375524572096765,
+  "audio_seconds": 59.998666666666665,
+  "rtf": 0.12585983111760338,
+  "nvml_peak_bytes": 9469493248,
+  "torch_peak_allocated_bytes": 8150668800,
+  "torch_peak_reserved_bytes": 8304721920,
+  "gpu": "NVIDIA GeForce RTX 5090",
+  "torch": "2.10.0+cu128",
+  "cuda": "12.8",
+  "timing": {
+    "nar_seconds": 4.007696468848735,
+    "vae_seconds": 3.542188974097371,
+    "e2e_seconds": 7.551166985882446
+  },
+  "truncated": {
+    "abc": false,
+    "semantic": false
+  },
+  "finite": true,
+  "audio_peak": 0.8817682862281799,
+  "audio_rms": 0.13392212986946106,
+  "clipped_fraction": 0.0,
+  "semantic_sha256": "105e53ffbbe671c28587ce52c917f57698087ed47bf6b63792a377e357236c6b",
+  "noise_sha256": "304490172c3b67d1dd3984d61d39e3cfc0d311ad6df79200674f815bc1081eae",
+  "latent_sha256": "b9461dadf0c0a16941fa0178ea5f3845f32d1f6bda70de4fe17541b2e81faabf",
+  "waveform_sha256": "9a80ae8267c22cd066aaf7cd1f7e5a12c7fcaedfb3a2e0b5f80b95ee2a47e482",
+  "source_or_quantized_model": "models/YuE2-3B",
+  "replay": "benchmark/ru-original-full/run-00",
+  "frames": 1500,
+  "gemv": false,
+  "measurement": "warm same-process request",
+  "pid": 6449
+}

evaluation/vae-chunk-benchmark.json

@@ -0,0 +1,101 @@
+[
+  {
+    "pipeline_numerics": true,
+    "core_frames": 1024,
+    "repeat": 0,
+    "seconds": 1.1320553519763052,
+    "nvml_peak_bytes": 8204910592,
+    "torch_peak_allocated_bytes": 2871368192,
+    "max_abs_difference": 0.0,
+    "relative_l2": 0.0,
+    "waveform_correlation": 1.0
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 1024,
+    "repeat": 1,
+    "seconds": 0.9049297058954835,
+    "nvml_peak_bytes": 8286699520,
+    "torch_peak_allocated_bytes": 2870057472,
+    "max_abs_difference": 0.0,
+    "relative_l2": 0.0,
+    "waveform_correlation": 1.0
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 1024,
+    "repeat": 2,
+    "seconds": 0.9072453379631042,
+    "nvml_peak_bytes": 8286699520,
+    "torch_peak_allocated_bytes": 2870581760,
+    "max_abs_difference": 0.0,
+    "relative_l2": 0.0,
+    "waveform_correlation": 1.0
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 512,
+    "repeat": 0,
+    "seconds": 0.9228434218093753,
+    "nvml_peak_bytes": 5040308224,
+    "torch_peak_allocated_bytes": 1737859584,
+    "max_abs_difference": 0.0,
+    "relative_l2": 0.0,
+    "waveform_correlation": 1.0
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 512,
+    "repeat": 1,
+    "seconds": 0.9203053209930658,
+    "nvml_peak_bytes": 5010948096,
+    "torch_peak_allocated_bytes": 1737193984,
+    "max_abs_difference": 0.0,
+    "relative_l2": 0.0,
+    "waveform_correlation": 1.0
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 512,
+    "repeat": 2,
+    "seconds": 0.9190280558541417,
+    "nvml_peak_bytes": 5021433856,
+    "torch_peak_allocated_bytes": 1738250752,
+    "max_abs_difference": 0.0,
+    "relative_l2": 0.0,
+    "waveform_correlation": 1.0
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 256,
+    "repeat": 0,
+    "seconds": 1.0967565891332924,
+    "nvml_peak_bytes": 3282894848,
+    "torch_peak_allocated_bytes": 1172341248,
+    "max_abs_difference": 3.874301910400391e-06,
+    "relative_l2": 1.3670459111381206e-06,
+    "waveform_correlation": 0.9999999999990744
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 256,
+    "repeat": 1,
+    "seconds": 1.0739446009974927,
+    "nvml_peak_bytes": 3282894848,
+    "torch_peak_allocated_bytes": 1172406784,
+    "max_abs_difference": 3.874301910400391e-06,
+    "relative_l2": 1.3670459111381206e-06,
+    "waveform_correlation": 0.9999999999990744
+  },
+  {
+    "pipeline_numerics": true,
+    "core_frames": 256,
+    "repeat": 2,
+    "seconds": 1.1024859179742634,
+    "nvml_peak_bytes": 3316449280,
+    "torch_peak_allocated_bytes": 1172470272,
+    "max_abs_difference": 3.874301910400391e-06,
+    "relative_l2": 1.3670459111381206e-06,
+    "waveform_correlation": 0.9999999999990744
+  }
+]

run.py

@@ -6,7 +6,7 @@ ROOT=Path(__file__).absolute().parent
 sys.path.insert(0,str(ROOT/'runtime/kernels/base'))
 sys.path.insert(0,str(ROOT/'runtime'))
 
-def load_pipeline(*,vae=None,low_memory=False,progress=True):
+def load_pipeline(*,vae=None,low_memory=False,offload_ar=False,progress=True):
     import torch
     if not torch.cuda.is_available():raise RuntimeError('This release requires an NVIDIA CUDA GPU.')
     if torch.cuda.get_device_capability()!=(12,0):
@@ -19,15 +19,15 @@ def load_pipeline(*,vae=None,low_memory=False,progress=True):
     from enable_gemv import enable
     from yue2_orbit import OrbitPipeline
     enable()
-    pipe=OrbitPipeline.from_pretrained(str(ROOT),vae=vae,device='cuda',memory_budget_gib=30,vae_core_frames=512,progress=progress)
+    pipe=OrbitPipeline.from_pretrained(str(ROOT),vae=vae,device='cuda',memory_budget_gib=30,vae_core_frames=512,offload_ar=offload_ar,progress=progress)
     pipe.fused=not low_memory
     pipe.trim_cache=True
     return pipe
 
 def main():
-    p=argparse.ArgumentParser();p.add_argument('--prompt',default=str(ROOT/'prompts/default-ru.json'));p.add_argument('--output',default='song');p.add_argument('--vae');p.add_argument('--low-memory',action='store_true');a=p.parse_args()
+    p=argparse.ArgumentParser();p.add_argument('--prompt',default=str(ROOT/'prompts/default-ru.json'));p.add_argument('--output',default='song');p.add_argument('--vae');p.add_argument('--low-memory',action='store_true');p.add_argument('--offload-ar',action='store_true');a=p.parse_args()
     request=json.loads(Path(a.prompt).read_text());out=Path(a.output);out.mkdir(parents=True,exist_ok=True)
-    with load_pipeline(vae=a.vae,low_memory=a.low_memory) as pipe:
+    with load_pipeline(vae=a.vae,low_memory=a.low_memory,offload_ar=a.offload_ar) as pipe:
         start=time.perf_counter()
         song=pipe(**{k:request[k] for k in ('style','lyrics','seed','cot') if k in request})
         song.save_artifacts(out);song.save(out/'audio.wav')