Psy-Qwen-DPO-LoRA

DPO-aligned LoRA adapter for a Chinese psychological counseling LLM, trained on top of an SFT version of Qwen3.5-0.8B. Aligns the model to be more empathetic, mirror the client's exact words, and avoid premature advice or leading questions.

Result: 74.88% win rate vs SFT baseline on 202 held-out prompts, with near-zero length bias (+1.3 chars), evaluated using DeepSeek V4-Flash as judge with 2-way position-bias mitigation.

🔗 Full project (training pipeline, evaluation scripts, debug log, case studies): github.com/ChenLingD/Psy-Qwen-DPO 🔗 Stage 1 (SFT): github.com/ChenLingD/Psy-Qwen-SFT

---

Headline Numbers

Stage	Metric	Result
Scorer iteration	chosen-better agreement	43% → 60% (+17 pp)
DPO training	eval reward accuracy	78.95% (15/19 val pairs)
DPO training	DPO loss (start → end)	0.693 → 0.378 (−45%)
DPO training	trainable params	5.4M / 0.63% (LoRA)
Phase 3 eval	overall win rate vs SFT	74.88% (202 prompts)
Phase 3 eval	DPO consistent wins	134 / 202 (66.3%)
Phase 3 eval	length bias (DPO − SFT)	+1.3 chars

---

How to Use

This is a LoRA adapter only — you need the base SFT model from the companion project to load it.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# 1. Load the SFT base (from the companion repo)
BASE_SFT = "path/to/your/sft-checkpoint"   # see Psy-Qwen-SFT
tokenizer = AutoTokenizer.from_pretrained(BASE_SFT, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    BASE_SFT,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# 2. Apply this DPO LoRA adapter on top
model = PeftModel.from_pretrained(model, "ChenLingD/Psy-Qwen-DPO-LoRA")
model.eval()

# 3. Generate (use the PsyDTCorpus REBT system prompt for best results)
messages = [
    {"role": "system", "content": "你是一位精通理性情绪行为疗法（REBT）的心理咨询师..."},
    {"role": "user", "content": "我最近压力很大，工作上感觉做什么都不对..."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.05,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

⚠️ Always pass eos_token_id=tokenizer.eos_token_id — Qwen3.5 uses <|im_end|> as the chat terminator, which differs from pad_token_id. Without this, generation will not stop properly.

---

Training Setup

Base	Qwen3.5-0.8B (SFT-fine-tuned counselor model)
Framework	ms-swift 4.1.0.dev0
Adapter	LoRA r=8, α=32, target qkv_proj + o_proj + mlp.{up,down,gate}
Reference adapter	Same SFT adapter (standard DPO setup)
Hyperparameters	β=0.1, lr=5e-6, 3 epochs, bf16, gradient checkpointing
Hardware	1 × NVIDIA A10 (24 GiB)
Train time	40m 46s
Peak GPU memory	9.66 / 24 GiB

---

Preference Data

431 train + 19 val preference pairs auto-generated from a single SFT model with K=4 candidate sampling, scored by an iteratively-refined 7-dimensional rule-based scorer:

• empathy_lead: empathic openers
• good_question: open-ended counseling questions
• cited_user_words: mirroring the client's exact phrasing
• length_penalty, repetition_penalty
• closing_question, safety_floor

The scorer was iterated through 2 rounds of error analysis on 60 manually-reviewed sample pairs, raising chosen-better agreement from 43% → 60% and reducing length bias from +12.7 → +6.4 chars in the training pairs.

Stratified across 12 psychological topics (romance, family, emotion, growth, …). 782 prompts kept fully held-out for Phase 3 evaluation.

---

Evaluation Methodology

200 held-out prompts (actual 202 due to stratified rounding), each judged twice by DeepSeek V4-Flash with swapped A/B order to mitigate position bias. Outcomes:

Outcome	Count	%
DPO consistent win (both runs)	134	66.3%
DPO win + tie	2	1.0%
Both tie	1	0.5%
Position bias (1-1 split)	29	14.4%
SFT win + tie	3	1.5%
SFT consistent win (both runs)	33	16.3%
Weighted win rate		74.88%

All 12 psychological topics show ≥62.5% DPO win rate; no scenario where DPO regressed.

See the GitHub repo for full training scripts, judge prompt, scoring rule, and case studies.

---

Files

• adapter_model.safetensors — LoRA weights (21 MB)
• adapter_config.json — PEFT configuration
• additional_config.json — ms-swift training config
• args.json — full training arguments (transparency)
• trainer_state.json — training history with loss/accuracy curves

---

Limitations

• In-distribution evaluation only. All eval prompts come from the same PsyDTCorpus distribution as training. Out-of-distribution performance not measured.
• REBT-flavored. Scorer rewards REBT-style replies; a different therapeutic framework would need a different scorer and yield a different aligned model.
• Single judge. DeepSeek V4-Flash is one judge; stronger validation would use a panel.
• Crisis intervention is out of scope. Not designed or evaluated for crisis intervention; production-grade safety routing is required for any real deployment.

---

Citation

@misc{ling2026psyqwendpo,
  author = {Chen Ling},
  title  = {Psy-Qwen-DPO: DPO Alignment of a Chinese Psychological Counselor LLM},
  year   = {2026},
  url    = {https://github.com/ChenLingD/Psy-Qwen-DPO}
}

---

Author

Chen Ling (Dawn) — LinkedIn — chenlingdawn@gmail.com