# Error while training LORA in KOHYA\_SS (stabilityai/stable-diffusion-xl-base-1.0)

**URL:** <https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069>\
**Category:** Beginners\
**Created:** [January 30, 2025, 9:21am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069 "2025-01-30T09:21:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [January 30, 2025, 9:21am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/1 "2025-01-30T09:21:28Z")

</div>

Hello

I have provided 1 image for training. It’s a simple JPG.

 ![Sample1](https://us1.discourse-cdn.com/hellohellohello/original/3X/8/4/84007d0150a84bdc1d884be0b1a01ca12a0933b2.jpeg)

I am getting the error below:

```auto
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\sdxl_train_network.py", line 185, in <module>
    trainer.train(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\train_network.py", line 272, in train
    train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 2325, in cache_latents
    dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 1146, in cache_latents
    cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.alpha_mask, subset.random_crop)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 2775, in cache_batch_latents
    raise RuntimeError(f"NaN detected in latents: {info.absolute_path}")
RuntimeError: NaN detected in latents: D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\trained-model\img\40_A Event of a Pharma Client A Event of a Pharma Client\Sample1.jpg
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\Scripts\accelerate.EXE\ __main__.py", line 7, in <module>
    sys.exit(main())
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
    args.func(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model/config_lora-20250130-144011.toml']' returned non-zero exit status 1.
14:43:08-514022 INFO Training has ended.

```

The config file is as below:

```auto
bucket_no_upscale = true
bucket_reso_steps = 64
cache_latents = true
caption_extension = ".txt"
clip_skip = 1
dynamo_backend = "no"
enable_bucket = true
epoch = 1
gradient_accumulation_steps = 1
huber_c = 0.1
huber_schedule = "snr"
learning_rate = 0.0001
logging_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\log"
loss_type = "l2"
lr_scheduler = "cosine"
lr_scheduler_args = []
lr_scheduler_num_cycles = 1
lr_scheduler_power = 1
lr_warmup_steps = 160
max_bucket_reso = 2048
max_data_loader_n_workers = 0
max_grad_norm = 1
max_timestep = 1000
max_token_length = 75
max_train_steps = 1600
min_bucket_reso = 256
mixed_precision = "fp16"
multires_noise_discount = 0.3
network_alpha = 1
network_args = []
network_dim = 8
network_module = "networks.lora"
noise_offset_type = "Original"
optimizer_args = []
optimizer_type = "AdamW8bit"
output_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model"
output_name = "last"
pretrained_model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0"
prior_loss_weight = 1
resolution = "512,512"
sample_prompts = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model\\prompt.txt"
sample_sampler = "euler_a"
save_every_n_epochs = 1
save_model_as = "safetensors"
save_precision = "fp16"
text_encoder_lr = 0.0001
train_batch_size = 1
train_data_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\img"
unet_lr = 0.0001
xformers = true

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 30, 2025, 9:40am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/2 "2025-01-30T09:40:38Z")

</div>

Maybe this is it. There is a famous bug in the VAE of stabilityai/stable-diffusion-xl-base-1.0, and it seems that the VAE does not work properly with fp16 precision…

> <https://github.com/bmaltais/kohya_ss/issues/2674>
>
> I am trying to create an SDXL LoRA using Runpod. My dataset consists of 25 image…s of women, each with a size of 1024x1024 pixels. I keep encountering error logs indicating issues with the images. All images are 1024x1024 pixels, and I've tried using both PNG and JPG formats, but the issue persists. Does anyone know how to fix this?
> 
> \*\*\[Error log\]\*\*
> File "/workspace/kohya\_ss/sd-scripts/sdxl\_train\_network.py", line 185, in \<module\>
> trainer.train(args)
> File "/workspace/kohya\_ss/sd-scripts/train\_network.py", line 272, in train
> train\_dataset\_group.cache\_latents(vae, args.vae\_batch\_size, args.cache\_latents\_to\_disk, accelerator.is\_main\_process)
> File "/workspace/kohya\_ss/sd-scripts/library/train\_util.py", line 2324, in cache\_latents
> dataset.cache\_latents(vae, vae\_batch\_size, cache\_to\_disk, is\_main\_process, file\_suffix)
> File "/workspace/kohya\_ss/sd-scripts/library/train\_util.py", line 1146, in cache\_latents
> cache\_batch\_latents(vae, cache\_to\_disk, batch, subset.flip\_aug, subset.alpha\_mask, subset.random\_crop)
> File "/workspace/kohya\_ss/sd-scripts/library/train\_util.py", line 2772, in cache\_batch\_latents
> raise RuntimeError(f"NaN detected in latents: {info.absolute\_path}")
> RuntimeError: NaN detected in latents: /workspace/data/img\_xyzminji/40\_xyzminji woman/xyzminji\_(1).jpg
> Traceback (most recent call last):
> File "/workspace/kohya\_ss/venv/bin/accelerate", line 8, in \<module\>
> sys.exit(main())
> File "/workspace/kohya\_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate\_cli.py", line 47, in main
> args.func(args)
> File "/workspace/kohya\_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch\_command
> simple\_launcher(args)
> File "/workspace/kohya\_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple\_launcher
> raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
> subprocess.CalledProcessError: Command '\['/workspace/kohya\_ss/venv/bin/python', '/workspace/kohya\_ss/sd-scripts/sdxl\_train\_network.py', '--config\_file', '/workspace/data/model\_lora/config\_lora-20240729-035252.toml'\]' returned non-zero exit status 1.
> 03:53:26-826393 INFO Training has ended.

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 4, 2025, 7:13am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/3 "2025-02-04T07:13:38Z")

</div>

Hello

I successfully trained the model. Your help resolved it.

Now when I run the model to generate a new image, I get the following error:

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 7.55it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
Load the LoRA weights
Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 34, in <module>
    raise e
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 31, in <module>
    pipeline.load_lora_weights(lora_weights_path)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\lora_pipeline.py", line 129, in load_lora_weights
    self.load_lora_into_unet(
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\lora_pipeline.py", line 305, in load_lora_into_unet
    unet.load_lora_adapter(
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\peft.py", line 301, in load_lora_adapter
    inject_adapter_in_model(lora_config, self, adapter_name=adapter_name, **peft_kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\mapping.py", line 260, in inject_adapter_in_model
    peft_model = tuner_cls(model, peft_config, adapter_name=adapter_name, low_cpu_mem_usage=low_cpu_mem_usage)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\tuners\lora\model.py", line 141, in __init__
    super(). __init__ (model, config, adapter_name, low_cpu_mem_usage=low_cpu_mem_usage)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\tuners\tuners_utils.py", line 184, in __init__
    self.inject_adapter(self.model, adapter_name, low_cpu_mem_usage=low_cpu_mem_usage)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\tuners\tuners_utils.py", line 520, in inject_adapter
    raise ValueError(error_msg)
ValueError: Target modules {'8.1.transformer_blocks.5.attn2.to_v', '8.1.transformer_blocks.8.attn1.to_out.0', 'mid_block.1.transformer_blocks.8.attn1.to_k', '1.1.transformer_blocks.0.attn2.to_q', '8.1.transformer_blocks.9.attn1.to_q', 'mid_block.1.proj_out', '7.1.transformer_blocks.5.attn2.to_out.0', 'mid_block.1

```

My program is as follows:

```auto
from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
#lora_weights_path = os.path.join(model_dir, "last.safetensors")
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
hf_token = "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"
pipeline = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float32,
).to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path)
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]

# Convert the generated image to a NumPy array
generated_image_np = np.array(generated_image)

# Check the generated image properties
print("Generated image properties:")
print("Shape:", generated_image_np.shape)
print("Data type:", generated_image_np.dtype)
print("Value range:", generated_image_np.min(), "-", generated_image_np.max())

# Handle NaN or infinite values and ensure the range is valid
print("Handle NaN or infinite values and ensure the range is valid")
generated_image_np = np.nan_to_num(generated_image_np, nan=0.0, posinf=255.0, neginf=0.0)
generated_image_np = np.clip(generated_image_np, 0, 255)
generated_image_np = generated_image_np.astype(np.uint8)

# Save or display the generated image
print("Save or display the generated image")

# Convert the NumPy array back to a PIL Image and save or display the generated image
pil_image = Image.fromarray(generated_image_np)
pil_image.save("generated_image.jpg")
pil_image.show()

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 4, 2025, 7:19am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/4 "2025-02-04T07:19:14Z")

</div>

Hello. It seems to be a compatibility issue with the Diffusers.😅 It is said to work after converting.

> <https://github.com/G-U-N/Phased-Consistency-Model/issues/2>
>
> Hey!
> 
> Congrats on you work, and thanks a lot of sharing it 🤗
> When trying to u…se the sd1.5 and sdxl checkpoints on the hub for inference with \`diffusers\`, I got this following error when calling \`load\_lora\_weights:\` 
> 
> 
> \`\`\`
> from diffusers import AutoPipelineForText2Image
> 
> model\_id = "stabilityai/stable-diffusion-xl-base-1.0"
> adapter\_id = "wangfuyun/PCM\_SDXL\_LoRAs"
> 
> pipe = AutoPipelineForText2Image.from\_pretrained(model\_id, torch\_dtype=torch.float16, variant="fp16")
> pipe.load\_lora\_weights(adapter\_id, weight\_name="pcm\_sdxl\_normalcfg\_16step.safetensors")
> \`\`\`
> 
> \`\`\`
> 
> ValueError: Target modules {'base\_model.model.up\_blocks.1.attentions.0.transformer\_blocks.0.attn2.to\_out.0', 'base\_model.model.up\_blocks.3.attentions.2.transformer\_blocks.0.attn2.to\_out.0', 'base\_model.model.down\_blocks.0.attentions.1.proj\_in', 'base\_model.model.up\_blocks.1.attentions.1.proj\_in', 'base\_model.model.down\_blocks.0.attentions.0.transformer\_blocks.0.attn1.to\_out.0', 'base\_model.model.up\_blocks.3.resnets.0.conv\_shortcut', 'base\_model.model.down\_blocks.3.resnets.0.conv1', 'base\_model.model.down\_blocks.3.resnets.0.time\_emb\_proj', 'base\_model.model.up\_blocks.1.attentions.2.transformer\_blocks.0.attn2.to\_out.0', 'base\_model.model.up\_blocks.3.attentions.1.transformer\_blocks.0.ff.net.0.proj', 'base\_model.model.down\_blocks.3.resnets.0.conv2', '
> ....
> , 'base\_model.model.down\_blocks.0.attentions.0.transformer\_blocks.0.attn1.to\_v', 'base\_model.model.up\_blocks.1.attentions.1.transformer\_blocks.0.attn2.to\_q', 'base\_model.model.up\_blocks.2.attentions.1.proj\_out', 'base\_model.model.up\_blocks.2.attentions.2.transformer\_blocks.0.attn1.to\_v', 'base\_model.model.up\_blocks.3.attentions.0.proj\_out', 'base\_model.model.up\_blocks.3.attentions.1.transformer\_blocks.0.attn2.to\_v', , 'base\_model.model.down\_blocks.0.resnets.1.time\_emb\_proj', 'base\_model.model.down\_blocks.0.attentions.1.transformer\_blocks.0.attn1.to\_v'} not found in the base model. Please check the target modules and try again.
> 
> \`\`\`

> <https://github.com/huggingface/diffusers/blob/main/scripts/convert_lora_safetensor_to_diffusers.py>

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 5, 2025, 5:23am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/5 "2025-02-05T05:23:25Z")

</div>

Hello

I ran the below program and getting a blank black image.

The code

```auto
from diffusers import AutoPipelineForText2Image
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
#lora_weights_path = os.path.join(model_dir, "last.safetensors")
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]

# Convert the image data to a NumPy array
image_data = np.array(generated_image)

# Handle NaN or infinite values
image_data = np.nan_to_num(image_data, nan=0.0, posinf=255.0, neginf=0.0)

# Ensure the values are within the range [0, 255]
image_data = np.clip(image_data, 0, 255)

# Convert to uint8
image_data = image_data.astype(np.uint8)

# Create a PIL image from the NumPy array
final_image = Image.fromarray(image_data)

# Save and display the final image
final_image.save("generated_image.png")
final_image.show()

```

The output

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00, 3.23it/s]
Load the LoRA weights
Generate an image from a text prompt
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [45:56<00:00, 55.14s/it]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

```

The image is below

 ![kohya_ss_output_stable-diffusion-xl-base-1.0](https://us1.discourse-cdn.com/hellohellohello/original/3X/1/c/1c9135ff1e1c6cf85a3ab578588258e095cf4fe3.png)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 7:09am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/6 "2025-02-05T07:09:26Z")

</div>

I generated this image by like this

```py
from diffusers import AutoPipelineForText2Image
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
#lora_weights_path = os.path.join(model_dir, "last.safetensors")
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]
generated_image.save("generated_image.png")

```

 ![generated_image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/2/d23f5d670a14b5aa6fab7c71fb8880731251f9e2.jpeg)

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 7, 2025, 8:43am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/7 "2025-02-07T08:43:26Z")

</div>

Strange, I used your code, but still not works with me (I updated all libraries too)

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00, 2.57it/s]
Load the LoRA weights
Generate an image from a text prompt
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [41:52<00:00, 50.24s/it]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 7, 2025, 9:01am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/8 "2025-02-07T09:01:19Z")

</div>

There is a problem with float16 when used with older GeForce cards, and the image may turn black, I wonder if that’s it.

```py
#, torch_dtype=torch.float16
, torch_dtype=torch.float32

```

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 7, 2025, 9:17am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/9 "2025-02-07T09:17:05Z")

</div>

This might not concern you, but getting the following error:

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:43<00:00, 6.22s/it]
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 26, in <module>
    pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16").to("cuda")
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 461, in to
    module.to(device, dtype)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 1077, in to
    return super().to(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1340, in to
    return self._apply(convert)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply
    module._apply(fn)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply
    module._apply(fn)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 927, in _apply
    param_applied = fn(param)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1326, in convert
    return t.to(
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 4.00 GiB of which 0 bytes is free. Of the allocated memory 10.67 GiB is allocated by PyTorch, and 229.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 7, 2025, 9:24am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/10 "2025-02-07T09:24:13Z")

</div>

This way, the shortage in VRAM is made up for with RAM. It’s slow, but…

```py
#pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16", device_map="auto")

```

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 7, 2025, 9:30am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/11 "2025-02-07T09:30:19Z")

</div>

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 26, in <module>
    pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16", device_map="auto")
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\auto_pipeline.py", line 428, in from_pretrained
    return text_2_image_cls.from_pretrained(pretrained_model_or_path, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 710, in from_pretrained
    raise NotImplementedError(
NotImplementedError: auto not supported. Supported strategies are: balanced

```

And the “balanced” strategy is giving me the following error

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 71%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 5/7 [00:37<00:12, 6.27s/it]Some parameters are on the meta device because they were offloaded to the cpu.
Some parameters are on the meta device because they were offloaded to the cpu.
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:38<00:00, 5.54s/it]
Load the LoRA weights
You shouldn't move a model that is dispatched using accelerate hooks.
You shouldn't move a model that is dispatched using accelerate hooks.
Generate an image from a text prompt
  0%| | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 42, in <module>
    generated_image = pipeline(prompt=text_prompt).images[0]
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py", line 1208, in __call__
    noise_pred = self.unet(
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\models\unets\unet_2d_condition.py", line 1143, in forward
    emb = self.time_embedding(t_emb, timestep_cond)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\models\embeddings.py", line 1304, in forward
    sample = self.linear_1(sample)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\linear.py", line 125, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 7, 2025, 10:14am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/12 "2025-02-07T10:14:18Z")

</div>

If the problem can’t be solved by upgrading to a newer version of accelerate, this method is going to be difficult…

```auto
pip install -U accelerate diffusers

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 8, 2025, 2:51pm UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/13 "2025-02-08T14:51:52Z")

</div>

I remembered something important. There is a bug in the official SDXL 1.0 VAE with 16-bit precision, so this may be improved.

```py
from diffusers import AutoPipelineForText2Image, AutoencoderKL
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, vae=vae, torch_dtype=torch.float16, variant="fp16").to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]
generated_image.save("generated_image.png")

```

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 10, 2025, 5:01am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/14 "2025-02-10T05:01:20Z")

</div>

Getting same error:

```auto
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 631/631 [00:00<00:00, 630kB/s]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\file_download.py:140: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\ADMIN\.cache\huggingface\hub\models--madebyollin--sdxl-vae-fp16-fix. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
diffusion_pytorch_model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 335M/335M [00:45<00:00, 7.41MB/s]
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00, 2.72it/s]
Load the LoRA weights
Generate an image from a text prompt
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [43:16<00:00, 51.93s/it]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 10, 2025, 5:05am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/15 "2025-02-10T05:05:04Z")

</div>

If it comes to that, it has to be this…

> <https://github.com/huggingface/diffusers/issues/2153>
>
> Newbie here. I am interested in stable diffusion and wanted to practice and see …how it performs. I came across https://github.com/huggingface/diffusers/tree/main#new--stable-diffusion-is-now-fully-compatible-with-diffusers which I followed the code:
> \`\`\`
> import torch
> from diffusers import StableDiffusionPipeline
> 
> pipe = StableDiffusionPipeline.from\_pretrained("CompVis/stable-diffusion-v1-4", revision="fp16", torch\_dtype=torch.float16, safety\_checker=None)
> pipe = pipe.to("cuda")
> 
> prompt = "a photo of an astronaut riding a horse on mars"
> image = pipe(prompt).images\[0\]
> 
> image.save("astronaut\_rides\_horse.png")
> \`\`\`
> My issue is I am having this error in terminal and the file "astronaut\_rides\_horse.png" is only a black image:
> Python\\Python310\\lib\\site-packages\\diffusers\\pipelines\\pipeline\_utils.py:818: RuntimeWarning: invalid value encountered in cast
> images = (images \* 255).round().astype("uint8")

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 10, 2025, 6:22am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/17 "2025-02-10T06:22:34Z")

</div>

John6666,

The following code gives the same error:

```auto
from diffusers import AutoPipelineForText2Image, AutoencoderKL
import torch
import os
import numpy as np
from PIL import Image

print("vae")

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, vae=vae, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipeline.enable_sequential_cpu_offload()
pipeline.enable_attention_slicing("max")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]
generated_image.save("generated_image.png")

```

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 10, 2025, 6:47am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/18 "2025-02-10T06:47:33Z")

</div>

I am not training, but generating images from a trained model (The training is successfully completed)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 10, 2025, 9:29am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/19 "2025-02-10T09:29:59Z")

</div>

In that case, it seems likely that the problem is not due to a bug in VAE, but rather a limitation of GeForce 10x0 and/or torch.float16. I think it would work if you let the CPU do the inference, but the speed is too slow…

```py
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16").to("cpu")

```

or

```py
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16")
pipeline.enable_model_cpu_offload(device="cuda")

```

?

If you quantize it, you can save VRAM, but actually, there is a bug where LoRA will not work if you do that…

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 10, 2025, 10:32am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/21 "2025-02-10T10:32:08Z")

</div>

Works. Thank You.

---

<div class="post-metadata">

**Author:** ![deicool](https://avatars.discourse-cdn.com/v4/letter/d/9f8e36/32.png) [@deicool](https://discuss.huggingface.co/u/deicool)\
**Post date:** [February 11, 2025, 5:21am UTC](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069/22 "2025-02-11T05:21:10Z")

</div>

Hello John6666,

A different question:

The output is coming great for the prompt “A Beautiful Woman” (I have trained for 10 images).

 ![ABeautifulWoman-11thFeb2025-2](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/4/34959a7ed69a97c5176a82b048ba82f6bc1be6cb.png)

However the output for the prompt “A Event of a Pharma Client” is not that all good ( I have trained for 10 images).

 ![PharmaEvent-11thFeb2025](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/5/d5d4b6ca2ab143db644eda21051d8c330dedaf22.jpeg)

Why would that be?

Thank You

[Next page](https://discuss.huggingface.co/t/error-while-training-lora-in-kohya-ss-stabilityai-stable-diffusion-xl-base-1-0/138069.md?page=2)