Tencent-Hunyuan-Multimodal-RL
/

SD3.5-GenEval2-Single-Reward

@@ -1,197 +1,78 @@
 ---
 base_model: stabilityai/stable-diffusion-3.5-medium
 library_name: peft
 tags:
 - lora
 ---
-# Model Card for Model ID
-<!-- This model card has been generated automatically according to the information provided by the PEFT library. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper:** [Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models](https://huggingface.co/papers/2606.11025)
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section provides recommendations for the use of the model. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is useful to do so. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in Lacoste et al. (2019).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- Clearly defined terms go in this section. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
 ### Framework versions
 - PEFT 0.19.1

 ---
 base_model: stabilityai/stable-diffusion-3.5-medium
 library_name: peft
+pipeline_tag: text-to-image
 tags:
 - lora
+- text-to-image
+- diffusers
+- flow-matching
+- reinforcement-learning
+- flow-dppo
+- geneval2
 ---
+# SD3.5 GenEval2 Single-Reward (Flow-DPPO)
+A LoRA adapter for [`stabilityai/stable-diffusion-3.5-medium`](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium), fine-tuned with **Flow-DPPO** on GenEval2 in the **single-reward** setting (optimizing the GenEval2 reward only).
+## Flow-DPPO
+[**Flow-DPPO**](https://huggingface.co/papers/2606.11025) (Flow Divergence Proximal Policy Optimization) is an online reinforcement learning method for flow-matching image/video generators. Methods such as Flow-GRPO and Flow-CPS cast the denoising process as a Markov Decision Process and apply PPO-style ratio clipping to enforce a trust region. Flow-DPPO argues that ratio clipping is a noisy, single-sample proxy for the true policy divergence, which over-constrains some parts of the trajectory and under-constrains others.
+Because the per-step policy of a flow model is Gaussian, the KL divergence between the old and new policies can be computed exactly and cheaply. Flow-DPPO replaces ratio clipping with a divergence-proximal constraint, implemented as an **asymmetric divergence mask**: a gradient update is blocked only when (1) the advantage and ratio indicate the update is moving the policy *away* from the old policy, and (2) the exact KL already exceeds a threshold. Updates that move *back toward* the old policy are never blocked, accelerating recovery from overshooting.
+This yields higher reward, better KL-proximal efficiency, stronger robustness to catastrophic forgetting, balanced multi-objective optimization, and stable multi-epoch training.
+- Paper: https://huggingface.co/papers/2606.11025
+- Code: https://github.com/Tencent-Hunyuan/UniRL/tree/main/FlowDPPO
+- Trained with the [Flow-Factory](https://github.com/Jayce-Ping/Flow-Factory) RL framework.
+## Usage
+```python
+import torch
+from diffusers import StableDiffusion3Pipeline
+from peft import PeftModel
+pipe = StableDiffusion3Pipeline.from_pretrained(
+    "stabilityai/stable-diffusion-3.5-medium",
+    torch_dtype=torch.bfloat16,
+)
+# Load the Flow-DPPO LoRA adapter
+pipe.transformer = PeftModel.from_pretrained(
+    pipe.transformer,
+    "Tencent-Hunyuan-Multimodal-RL/SD3.5-GenEval2-Single-Reward",
+    torch_dtype=torch.bfloat16,
+)
+pipe.enable_model_cpu_offload()  # remove and call pipe.to("cuda") if you have enough VRAM
+prompt = "four white cats are behind a red bagel"
+image = pipe(
+    prompt,
+    height=1024,
+    width=1024,
+    guidance_scale=4.5,
+    num_inference_steps=40,
+    max_sequence_length=512,
+    generator=torch.Generator("cpu").manual_seed(0),
+).images[0]
+image.save("output.png")
+```
+## Citation
+```bibtex
+@article{ping2026flowdppo,
+  title={Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models},
+  author={Ping, Bowen and Zhou, Xiangxin and Qi, Penghui and Luo, Minnan and Bo, Liefeng and Pang, Tianyu},
+  journal={arXiv preprint arXiv:2606.11025},
+  year={2026}
+}
+```
 ### Framework versions
 - PEFT 0.19.1