Train Your Own Image Lora Model: Easy Tutorial

Training AI models used to be expensive, slow, and inaccessible to most. But with Lora (Low-Rank Adaptation), that’s no longer the case. It enables you to customize powerful models with a fraction of the data and compute power.

In this tutorial, you’ll learn how to create your own Lora from scratch using Kohya.

What is Lora?

Lora is a method for efficiently fine-tuning large AI models. Instead of retraining the entire model, it introduces a small set of additional trainable weights, allowing you to teach the model new behaviors while leaving its core parameters unchanged. This approach makes training faster and more accessible, requiring less data and compute power. It also produces lightweight files that are easy to share and reuse.

Loras have many advantages but also some disadvantages, here are the pros and cons:

• Pros
  • Uses significantly less GPU memory than full fine-tuning.
  • Much faster to train.
  • Can run on modest consumer GPUs.
  • Great for small datasets.
  • Produces lightweight files that are easy to store and share.
• Cons
  • Generally less powerful and flexible than full fine-tuning.
  • Poor training can lead to overfitting or visual artifacts.
  • Often requires careful parameter tuning for best results.
  • May struggle with tasks that differ significantly from the base model’s original training.

Install Kohya

Kohya is a user-friendly tool that lets you fine-tune a Lora model in just a few straightforward steps. We installed it on Windows using pip and Conda. If you’d like to follow the same setup, you can refer to the official installation instructions here: https://github.com/bmaltais/kohya_ss/blob/master/docs/Installation/pip_windows.md . If you prefer a different setup or are using another platform, you can explore all available installation options here: https://github.com/bmaltais/kohya_ss#installation-options.

Preparing The Dataset

For this tutorial, we’ll fine-tune a model to generate images of mugs featuring animal logos. Here’s an example of the kind of result we’re aiming for:

First, download the dataset by clicking this link mugs.zip.

The dataset consists of 10 images of white mugs, each featuring a different black animal logo. All images are in 512×512 resolution and saved in .png format. Each image is paired with a corresponding .txt file containing a caption that describes the image content.

Now extract the archive and place the root folder (the mugs folder) inside the datasets directory within the kohya_ss folder.

Note that the images and captions are stored inside a folder named 10_images. The number prefix (e.g. 10_) controls how many times each image is repeated during training. It is not required, but it affects how strongly the model learns from the dataset. Your final folder structure should look something like this:

kohya_ss

——> dataset

——> mugs

——> 10_images

——> bear.png

——> bear.txt

——> cat.png

——> cat.txt

——> ………..

——> dog.png

——> dog.txt

Training The Lora

Open Kohya by running gui.bat on Windows or gui.sh on Linux. Once it starts, it will display a URL that you can open in your browser. Next, go to the Lora tab. Set the base model to runwayml/stable-diffusion-v1-5, and point the Image folder to your mugs dataset directory. Finally, set the trained model output name to mugs.

Under Folders, set the Output directory for trained model to the outputs folder included with Kohya. This is where your trained model will be saved.

Under Parameters, set Epoch to 30. This means the model will iterate over your dataset 30 times during training. Set Max train steps to 0, otherwise it will override your epoch setting and be used instead.

Leave all the rest of the settings as default, unless you are familiar with tuning Lora parameters.

Now it’s time to start training. Click the Start Training button at the bottom of the page. Once the process is complete, a mugs.safetensors file will be generated in the outputs folder. Notice that the size of the file is small (for us it is only around 7 MB) highlighting one of the key advantages of Loras compared to full fine-tuning. Also, it only took around 20 minutes to finish, which is impressive considering we tested it on a modest 8 GB VRAM GPU.

Testing The Lora

We’ll show you how to test your Lora using the WebUI from https://github.com/automatic1111/stable-diffusion-webui. If you don’t already have it installed, follow the instructions in that repository to set it up first.

Copy the mugs.safetensors file from the kohya_ss\outputs folder and paste it into the models\Lora directory inside your Automatic1111 WebUI installation folder. Next, launch the WebUI and open it in your browser. Make sure you’re using a Stable Diffusion 1.5 model, since the Lora must match the base model version it was trained on.

Then enter the following positive prompt:

Logo of a black lion on a white mug

For the negative prompt, you can use something like:

low resolution

To apply the Lora, open the Lora tab and select the mugs Lora. You’ll notice it automatically inserts a <lora:mugs:1> tag into your prompt.

The number in <lora:mugs:1> represents the Lora weight. Keeping it at 1 usually works well, but you can adjust it to see how it affects the results. However, increasing it too much may lead to distorted or lower-quality outputs.

Use the following generation parameters:

• Sampling method: DPM++ 2M SDE
• Schedule type: Karras
• Sampling steps: 30
• Width and Height: 512
• CFG Scale: 7

We generated six samples using the prompt above, here are the results:

As you can see, the model generates a lion icon even though no lion mug exists in the dataset. This suggests it is effectively generalizing to unseen inputs, indicating that the Lora is functioning as intended. It also captures key visual characteristics from the training data, for example, the mugs are consistently placed on a desk with a depth-of-field effect, showing that the Lora has successfully learned underlying patterns from the dataset.

If you’re not satisfied with the results, you can experiment with Kohya’s training parameters. You can increase the number of epochs, but keep in mind that setting it too high may lead to overfitting. You can also adjust the learning rate, while being careful not to push it too far in either direction, as this can also affect stability and lead to overfitting. Another important factor is your dataset quality, if your images are blurry or low-resolution, improving them can significantly enhance the final results. You can also experiment with different captions in your dataset and see what works best.

Conclusion

This tutorial demonstrates that with a small dataset and modest hardware, you can train a Lora that effectively captures the style you’re aiming for and produces impressive results. This offers a significant advantage over full fine-tuning pipelines, which often require much longer training times and substantially more compute power, sometimes taking hours or even longer, while delivering relatively similar results.

Tired of generating images and ready to step into video? Check out our blog to find out if your computer can handle the switch: Can You Run AI Video Generation Locally on 8GB of VRAM?.