Train a Custom AI Model: LoRA Photo Generation Guide

This article may contain affiliate links. We may earn a small commission if you purchase through our links, at no extra cost to you. We only recommend tools we’ve personally tested.

A complete guide to building an automated LoRA training pipeline — from thousands of raw photos to a working AI model, using hardware you probably already own.

What This Guide Covers

Most AI image generators create generic faces. But what if you want to generate photorealistic images of a specific real person — in any setting, any pose, any lighting? That requires training a custom model on their photos.

The problem is that most people do this badly. They dump 20 random photos into a training script, get blurry results that look nothing like the person, and conclude it doesn’t work.

This guide shows you how to do it right — with an automated pipeline that:

• Pulls thousands of source photos from cloud storage
• Uses computer vision to find and score every photo of the target person
• Detects professional color calibration charts for accurate skin tone reproduction
• Selects the optimal 30-50 training images with the right mix of angles and expressions
• Produces a training-ready dataset with unique per-image captions
• Trains a LoRA model that nails the person’s identity

The entire pipeline runs on a home PC. No cloud subscriptions, no expensive hardware.

The Two Approaches: LoRA vs Full Fine-Tuning

Before we start, you need to understand there are two paths to custom AI photo generation. Which one you choose depends on your hardware and budget.

Path 1: LoRA (Low-Rank Adaptation) — Home PC Friendly

LoRA works by only modifying a tiny fraction of the base model’s parameters (~2-4 million out of billions). This means it trains fast, needs little data, and produces a small portable file. The catch is that it has a ceiling — throw too many images at it (100+) and the identity gets diluted rather than improved.

Path 2: Full Fine-Tuning — Cloud GPU Required

Full fine-tuning rewrites the entire model to know this person. It produces better results for complex scenarios but requires serious compute. You don’t need to own the hardware — rent a multi-GPU server for 24-48 hours, upload your photos, train, download the result, and delete the server.

Cloud GPU Providers:

This guide focuses on Path 1 (LoRA) since it’s accessible to everyone with a gaming PC. The photo curation pipeline is the same for both paths — only the training step differs.

Phase 1: Photo Acquisition

What You Need

• Source photos: The more the better. We started with 3,000+ and selected 40. Having a large pool lets you be extremely picky about quality.
• A mix of shots: Face close-ups (70%), three-quarter angles (20%), and full body (10%)
• Multiple lighting conditions: Indoor, outdoor, warm light, cool light
• Multiple expressions: Neutral, smiling, laughing, serious
• A Calibrite ColorChecker Passport Photo (optional but highly recommended): Photograph it once per lighting setup for accurate color calibration

Source Photo Guidelines

The Download Pipeline

If your photos are on Google Drive, you can automate the download using the Google Drive API:

# Install dependencies
pip install gdown requests

# Or use the Drive API directly with an API key
# (works for "Anyone with the link" shared folders)

import urllib.request, urllib.parse, json

API_KEY = "your-google-drive-api-key"
FOLDER_ID = "your-folder-id-from-url"

# List files in a shared folder
params = {
"q": f"'{FOLDER_ID}' in parents and trashed=false",
"fields": "files(id,name,mimeType,size)",
"pageSize": "1000",
"key": API_KEY,
}
url = "https://www.googleapis.com/drive/v3/files?" + urllib.parse.urlencode(params)
response = urllib.request.urlopen(url)
files = json.loads(response.read())["files"]

# Filter for images only
IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".arw", ".cr2", ".nef", ".dng", ".raw"}
images = [f for f in files if any(f["name"].lower().endswith(ext) for ext in IMAGE_EXTS)]

# Download each image
for img in images:
download_url = f"https://www.googleapis.com/drive/v3/files/{img['id']}?alt=media&key={API_KEY}"
urllib.request.urlretrieve(download_url, f"downloads/{img['name']}")

Storage tip: Use a NAS or external drive. RAW photos from a Sony A7RV are ~60MB each. 3,000 of them = ~180GB. Don’t try to process these on your boot drive.

Phase 2: Intelligent Photo Curation

This is where the magic happens. Instead of manually scrolling through thousands of photos, we use computer vision to automate the entire selection process.

Step 1: Install Dependencies

pip install opencv-python-headless numpy rawpy

# Download face detection models
mkdir -p models && cd models
# DNN face detector (more accurate than Haar cascades)
curl -LO "https://raw.githubusercontent.com/opencv/opencv/master/samples/dnn/face_detector/deploy.prototxt"
curl -LO "https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel"
# Face recognition model (for identity matching)
curl -LO "https://github.com/opencv/opencv_zoo/raw/main/models/face_recognition_sface/face_recognition_sface_2021dec.onnx"

Step 2: Build a Reference Identity

You need 10-20 photos where you’re certain the target person is the main subject. These become the “identity anchor” that the pipeline matches against.

import cv2
import numpy as np

# Load models
face_net = cv2.dnn.readNetFromCaffe("models/deploy.prototxt",
"models/res10_300x300_ssd_iter_140000.caffemodel")
face_recognizer = cv2.FaceRecognizerSF.create(
"models/face_recognition_sface_2021dec.onnx", "")

def detect_faces(img, confidence_threshold=0.6):
"""Detect faces using DNN detector."""
h, w = img.shape[:2]
blob = cv2.dnn.blobFromImage(img, 1.0, (300, 300), (104, 177, 123))
face_net.setInput(blob)
detections = face_net.forward()

faces = []
for i in range(detections.shape[2]):
conf = detections[0, 0, i, 2]
if conf < confidence_threshold:
continue
x1 = int(detections[0, 0, i, 3] * w)
y1 = int(detections[0, 0, i, 4] * h)
x2 = int(detections[0, 0, i, 5] * w)
y2 = int(detections[0, 0, i, 6] * h)
faces.append((x1, y1, x2-x1, y2-y1, float(conf)))
return faces

def get_embedding(img, face_bbox):
"""Extract 128-dim face embedding."""
roi = np.array(face_bbox[:4], dtype=np.int32)
aligned = face_recognizer.alignCrop(img, roi)
return face_recognizer.feature(aligned)

# Build reference embeddings from known photos
reference_embeddings = []
for photo_path in known_photos:
img = cv2.imread(photo_path)
faces = detect_faces(img)
if faces:
largest = max(faces, key=lambda f: f[2] * f[3])
emb = get_embedding(img, largest)
reference_embeddings.append(emb)

Step 3: Color Checker Detection

If you shoot with a Calibrite ColorChecker Passport Photo, the pipeline can detect it and compute a color correction matrix for each lighting setup.

The ColorChecker has 24 standard color patches with known sRGB values. By comparing what the camera captured versus the known values, we compute a 3×3 matrix that corrects color across the entire shoot.

# Known sRGB reference values for the 24 classic patches
COLORCHECKER_SRGB = np.array([
[115, 82, 68],    # Dark Skin
[194, 150, 130],  # Light Skin
[98, 122, 157],   # Blue Sky
[87, 108, 67],    # Foliage
# ... (all 24 patches)
[243, 243, 242],  # White
[52, 52, 52],     # Black
], dtype=np.float64)

def compute_color_correction(measured_patches, reference_patches):
"""Compute 3x3 CCM using least squares."""
ccm, _, _, _ = np.linalg.lstsq(measured_patches, reference_patches, rcond=None)
return ccm

def apply_correction(image_rgb, ccm):
"""Apply color correction to an image."""
pixels = image_rgb.reshape(-1, 3).astype(np.float64)
corrected = np.clip(pixels @ ccm, 0, 255).astype(np.uint8)
return corrected.reshape(image_rgb.shape)

Why this matters for AI training: Without color correction, the model learns the lighting artifacts (orange skin from tungsten, blue tint from shade) instead of the person’s actual appearance. With correction, skin tones are accurate and consistent across all training images.

Step 4: Score and Rank Every Photo

For each photo that matches the target person, we compute a quality score based on:

def score_image(img, face_bbox, match_score):
h, w = img.shape[:2]
x, y, fw, fh = face_bbox[:4]

# 1. Sharpness — Laplacian variance (higher = sharper)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
sharpness = cv2.Laplacian(gray, cv2.CV_64F).var()

# 2. Lighting — histogram spread (wider = better exposed)
hist = cv2.calcHist([gray], [0], None, [256], [0, 256]).flatten()
hist = hist / hist.sum()
std = np.sqrt(np.average((np.arange(256) - np.average(range(256), weights=hist))**2, weights=hist))
lighting = min(1.0, std / 80.0)

# 3. Face size — larger face = more detail for training
face_ratio = (fw  fh) / (w  h)

# 4. Face angle — estimated from aspect ratio
aspect = fw / max(fh, 1)
angle = "front" if aspect > 0.85 else "three_quarter" if aspect > 0.65 else "profile"

# Composite score (weighted blend)
composite = (
min(sharpness / 500, 1.0) * 0.3 +
max(0, lighting) * 0.2 +
min(face_ratio  20, 1.0)  0.3 +
(1.0 - match_score) * 0.2
)

return composite, angle

Step 5: Select with Diversity

Don’t just take the top 40 by score — you need angle diversity. A model trained entirely on front-facing photos can only generate front-facing images.

# Target distribution
TARGET = {"front": 15, "three_quarter": 10, "profile": 5}  # = 30 face shots
BODY_SHOTS = 8  # Full body shots
# Total: 38 training images

selected = []
angle_counts = {"front": 0, "three_quarter": 0, "profile": 0}

# Select face shots with diversity
for candidate in sorted(all_candidates, key=lambda x: x["score"], reverse=True):
angle = candidate["angle"]
if angle_counts[angle] < TARGET[angle]:
selected.append(candidate)
angle_counts[angle] += 1
if sum(angle_counts.values()) >= 30:
break

# Add body shots (face is small relative to image)
body_candidates = [c for c in all_candidates if c["face_ratio"] < 0.05]
selected.extend(sorted(body_candidates, key=lambda x: x["score"], reverse=True)[:BODY_SHOTS])

Step 6: Crop, Convert, and Caption

Each selected image gets:

1. Cropped to head+shoulders (face shots) or full-body framing (body shots)
2. Resized to 1024×1024 pixels (FLUX native resolution)
3. Color corrected using the shoot’s ColorChecker data (if available)
4. Saved as lossless PNG
5. Captioned with a unique text file describing the image

training-set/
├── subject_001_cc.png
├── subject_001_cc.txt    → "subject_name, front-facing portrait, well-lit"
├── subject_002.png
├── subject_002.txt       → "subject_name, three-quarter view, natural lighting"
├── subject_038.png
├── subject_038.txt       → "subject_name, full body, casual outfit, outdoor"
└── ...

The captions are critical. This is the #1 reason most LoRA training fails. If every image has the same caption (or no caption), the model can’t learn what makes each photo unique. Unique per-image captions tell the model: “this trigger word + front-facing = this look, this trigger word + profile = this look.”

Ethnicity anchoring matters. If your subject has a specific ethnicity, anchor it in every caption: “a photo of ohwx person, smiling warmly, close-up portrait.” Without this, the model defaults to whatever features dominate its training data. This is one of the most common reasons LoRA training produces beautiful images of the wrong person.

Use AI to refine captions. After generating base captions from metadata (angle, lighting, expression), send each training image to a vision model like Gemini 2.5 Flash or GPT-4V and ask it to describe what it sees. Merge the AI description with your structured caption. This ensures every caption is genuinely unique and image-specific, not just a random recombination of templates.

Phase 3: Training

Understanding the Architecture (Why Text Encoders Matter)

Before diving into parameters, you need to understand why certain settings make or break face training. Most guides skip this, which is why most people get bad results.

An image generation model like Stable Diffusion XL has three main components:

1. Text Encoders — Convert your text prompt (“a photo of ohwx woman, smiling”) into a mathematical representation (embeddings) that the image generator understands. SDXL has two text encoders: CLIP ViT-L and OpenCLIP ViT-bigG. They work together to interpret your words.

1. UNet — The image generator. Takes the text embeddings plus random noise and iteratively removes the noise to create an image. This is the biggest component and where most of the “drawing ability” lives.

1. VAE — Converts between pixel space and the compressed “latent” space where the UNet operates. You don’t train this.

Here’s the critical insight: When you type your trigger word (“ohwx”), the text encoder needs to know what it means. If you only train the UNet (which is what most tutorials recommend because it’s faster), the text encoder still has no idea what “ohwx” is. It treats it as meaningless noise and essentially ignores it. The UNet learns to draw the person, but there’s no reliable way to trigger that identity through text.

Training the text encoder is the fix. It literally teaches the model that “ohwx” = this specific person’s face. The tradeoff: training is 5x slower because the text encoder outputs can’t be cached (they change every step). But without it, you get a model that produces great poses and lighting — with the wrong face.

Base Model Matching — The Hidden Gotcha

You must train on the same base model you’ll generate images with. This seems obvious but catches everyone.

If you train a LoRA on “SDXL Base 1.0” but generate images with “RealVisXL V5.0” (a fine-tuned SDXL variant optimized for photorealism), the LoRA won’t work correctly. The internal representations diverge during fine-tuning. The result: wrong faces, weird artifacts, style mismatches.

Our v1 training made this exact mistake — trained on vanilla SDXL Base, inferenced on RealVisXL. The poses and scenes looked great (those are general capabilities), but the face identity was lost.

Fix: If you want to generate with RealVisXL, train on RealVisXL. If you want to generate with FLUX.1-dev, train on FLUX.1-dev.

LoRA Training on a Home GPU

Hardware requirements:

• NVIDIA GPU with 8GB+ VRAM (RTX 3060/3070/4060 or better)
• CUDA 11.8+ installed
• 16GB+ system RAM
• ~10GB free disk space

Software setup (using kohya sd-scripts for SDXL):

# Clone kohya's sd-scripts
git clone https://github.com/kohya-ss/sd-scripts
cd sd-scripts

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # Linux/Mac
# venv\Scripts\activate  # Windows

# Install dependencies
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
pip install xformers  # For memory-efficient attention

Key training parameters for faces (SDXL LoRA):

Prodigy optimizer note: Prodigy is an adaptive optimizer that adjusts the learning rate automatically during training. Set --learning_rate 1 and let it figure out the rest. Important: Prodigy cannot use different learning rates for text encoder vs UNet (--text_encoder_lr will crash). It handles all parameter groups uniformly, which actually works better.

# Example Prodigy optimizer args
--optimizer_type Prodigy
--optimizer_args weight_decay=0.01 safeguard_warmup=True decouple=True use_bias_correction=True

Training time reality:

Yes, training the text encoder makes it much slower. The text encoder outputs cannot be cached when the encoders are being updated (because their outputs change every step). SDXL has two large text encoders that must run on every forward pass. This is the price of getting face identity right.

Output: A single .safetensors file (~300-340MB for SDXL LoRA with dim=32) that plugs into any compatible image generator.

Full Fine-Tuning on Cloud GPUs

If you want to go bigger with 500+ images, rent the GPU by the hour instead of buying one. We run this kind of job on RunPod (an H100 is roughly –3/hour there, so a 24-hour fine-tune costs less than a mid-range consumer GPU); Vast.ai and Lambda are comparable alternatives.

# On your rented cloud server (RunPod, Vast.ai, etc.):

# Upload your photo dataset (use rsync or their web UI)
rsync -avz training-set/ user@cloud-server:/workspace/training-set/

# Training takes 12-48 hours depending on dataset size and GPU count
# Output: 12-24GB model checkpoint

# Download the result
rsync -avz user@cloud-server:/workspace/output/ ./model-output/

Typical cloud training session:

1. Rent a 4x A100 server ($6/hr)
2. Upload 100GB of photos (~1 hour)
3. Train for 24 hours (~$144)
4. Download 20GB model (~30 min)
5. Delete server
6. Total: ~$150-200 for a one-time training

The Training Image Cheat Sheet

What Kills Training Quality

Ranked by how badly they wreck your results, based on real failures:

1. Not training the text encoder — The model learns to draw the person but the trigger word doesn’t activate the identity. You get great poses and lighting with a generic stranger’s face. This is the #1 failure mode that nobody talks about.
2. Base model mismatch — Training on SDXL Base 1.0 but generating with RealVisXL or another fine-tuned variant. The LoRA adapts to one model’s internal representations, which don’t transfer cleanly.
3. Identical/generic captions — 40 images with 6 caption patterns means the model can’t distinguish them. Every caption must be genuinely unique. Use AI vision APIs to describe each image specifically.
4. No ethnicity anchoring — If your subject has a specific ethnicity, say so in every caption. Without this, the model defaults to whatever ethnicity dominates its training data.
5. alpha < dim — Setting dim=32 but alpha=16 halves the effective LoRA strength. Use alpha = dim unless you have a specific reason not to.
6. All same angle — Model only generates that one pose
7. Too many images for LoRA — 100+ causes identity averaging. Sweet spot is 30-50.
8. Learning rate too high — With AdamW, anything above 1e-4 risks collapse. With Prodigy, just set 1 and let it adapt.
9. Blurry photos — Garbage in, garbage out
10. Missing noise_offset — Without it, the model struggles to generate true blacks and high-contrast scenes

Post-Processing: Face Swap Pipeline

Even with excellent training, you may want to combine the LoRA’s strengths (poses, scenes, lighting) with a more precise face identity. A face swap pipeline lets you generate an image with the LoRA, then swap in the reference face for pixel-perfect accuracy.

Pipeline:

1. InsightFace — Detects and analyzes faces in both the generated image and the reference photo
2. inswapper_128 — Swaps the face from the reference onto the generated image
3. GFPGAN — Restores face quality after the swap (removes artifacts, enhances detail)

# Simplified face swap pipeline
from insightface.app import FaceAnalysis
import onnxruntime

# Initialize
app = FaceAnalysis(name="buffalo_l")
app.prepare(ctx_id=0, det_size=(640, 640))

# Load the swap model
swapper = insightface.model_zoo.get_model("inswapper_128.onnx")

# Detect faces
source_face = app.get(reference_image)[0]  # Your real photo
target_face = app.get(generated_image)[0]  # AI-generated image

# Swap
result = swapper.get(generated_image, target_face, source_face, paste_back=True)

# Restore with GFPGAN
from gfpgan import GFPGANer
restorer = GFPGANer(model_path="GFPGANv1.4.pth", upscale=1)
_, _, restored = restorer.enhance(result, paste_back=True)

This gives you the best of both worlds: the LoRA’s scene composition with the reference photo’s exact face.

What You Get at the End

A single model file (~340MB for SDXL LoRA, ~200MB for FLUX LoRA, 12-24GB for full fine-tune) that plugs into any compatible image generation workflow. Open a prompt box and type:

• “ohwx woman, professional headshot, studio lighting, white background”
• “ohwx woman, casual outdoor portrait, golden hour, city street”
• “ohwx woman, jogging in a city park, morning light, candid shot”

Each generation takes 10-30 seconds on a consumer GPU. Unlimited images. Runs entirely locally — no cloud, no subscriptions, no per-image fees.

The total cost: $0 if you own a gaming PC (LoRA path), or $150-200 for a one-time cloud GPU rental (full fine-tune path). Either way, it’s a fraction of what a single professional photo shoot costs, and you can generate images forever.

Lessons Learned (From Actual Training Failures)

Here’s what we learned across two complete training runs on consumer hardware:

Key takeaways:

• Text encoder training is non-negotiable for face identity. Accept the 5x speed hit.
• Train on the exact model you’ll generate with. No exceptions.
• Every caption must be unique. Use AI vision to describe each image.
• Anchor ethnicity in every caption if your subject isn’t the default.
• Prodigy optimizer eliminates LR tuning headaches — set LR=1, done.
• noise_offset (0.0357) and min_snr_gamma (5.0) are small settings with big quality impact.

Tools used: Python, OpenCV, InsightFace, GFPGAN, Gemini Vision API, kohya sd-scripts, Google Drive API
Compatible base models: Stable Diffusion XL (RealVisXL, etc.), FLUX.1-dev, SD 1.5
Recommended hardware: Any NVIDIA GPU with 8GB+ VRAM