# Virtual Dress Try-On (Django + MediaPipe + OpenCV)

A customer opens the site, allows the camera, picks a dress and sees themselves
wearing it in near real time. 100% free / open-source.

**How it works:** the browser captures a webcam frame about every 150 ms and POSTs it
(+ `dress_id`) to `/api/tryon/`. Django runs MediaPipe Pose, warps the dress onto the
shoulders/hips with a perspective transform, alpha-blends it and returns a JPEG.

## Photo mode (home page) and live mode

`/` is **photo mode (no live video)**: the page waits with a framing outline. The customer chooses a dress
and presses **Take photo**; the camera switches on out of sight, a 3-2-1 countdown runs, **one photo** is
taken and the camera switches off again. The server edits that photo and shows the result.
**Full screen:** the ⛶ button (or the `F` key) fills the screen; in full screen a dress strip and the
Take photo / Restart / Download buttons stay available. Esc exits.
**Restart** (shown during the countdown, while processing and on the result) cancels and goes back to the
start for the next customer. After the result you can tap another dress to try it on the same photo.
`/live/` is the older live-video mode.

Try-on engines (set `TRYON_ENGINE` in `.env`):

| Engine | What it is | Needs | Look |
|---|---|---|---|
| `warp` (default) | MediaPipe body detection + the garment warped onto your torso | any PC, no GPU, ~0.3 s | flat "paste-on" look; best with front-facing garments |
| `ai` | open-source diffusion try-on (IDM-VTON) through a Hugging Face Space | `pip install gradio_client`, internet | much more realistic; 20-90 s per photo |
| `gemini` | Gemini image editing through Google's Interactions API | Gemini API key, internet | realistic generated edit; customer photos go to Google |
| `grok` | Grok Imagine image editing through xAI's API | xAI API key, internet | realistic generated edit; customer photos go to xAI |
| `claude_grok` | Claude analyzes the customer and garment images; Grok edits the photo | Anthropic and xAI API keys, internet | Claude-guided Grok image edit |
| `openai` | OpenAI image editing | OpenAI API key, internet | realistic generated edit; customer photos go to OpenAI |
| `fashn` | FASHN VTON on a self-hosted GPU server | GPU server | dedicated virtual try-on model |

Notes on the `ai` engine: it was written from the Space's public API and has **not been tested here**
(no access to Hugging Face from the build sandbox). Check the Space page -> "Use via API" if it
fails; any failure falls back to `warp` automatically. Customer photos are uploaded to that Space,
free Spaces are queued/rate-limited, and IDM-VTON's licence is non-commercial (CC BY-NC-SA 4.0), so
check licences before using it in a shop. For a private, commercial setup, host an open model yourself
on a GPU machine (about 8-16 GB VRAM) and point `TRYON_AI_SPACE` at it.

## Realistic AI engine for a real shop: FASHN VTON v1.5 (`TRYON_ENGINE=fashn`)

Most open try-on models (CatVTON, IDM-VTON, OOTDiffusion) are **non-commercial**. FASHN VTON v1.5
is published under **Apache-2.0** (code and weights, per its model card and release notes), which
allows commercial use. Before launch, read its LICENSE and model card yourself; a permissive
model licence does not change the terms of the research datasets it was trained on.

It runs as a **separate GPU server** (`fashn_server/`) so its libraries never clash with the web app,
your customers' photos stay on your own machines, and it can live on a different computer.

**Requirements:** an NVIDIA GPU. The model card says about **8 GB of VRAM** and ~5 s per image on an
H100 (slower on small cards). A 4 GB card is below that and is not expected to work.

1. On the GPU machine (exact steps from the official repo's README):
   ```
   git clone https://github.com/fashn-AI/fashn-vton-1.5.git
   cd fashn-vton-1.5
   python -m venv .venv          # activate it: .venv\Scripts\activate (Windows)
   pip install -e .
   python scripts/download_weights.py --weights-dir ./weights     # ~2 GB
   pip install flask==3.0.3
   ```
   Install a CUDA build of PyTorch first if `pip install -e .` did not (pytorch.org). The repo installs
   `onnxruntime-gpu`; on a CPU-only machine run `pip uninstall onnxruntime-gpu && pip install onnxruntime`.
   Copy this project's `fashn_server/` folder next to it and set `FASHN_WEIGHTS_DIR` to the `weights` folder.
   **To try the model with no installation at all**, FASHN publishes a test page:
   `huggingface.co/spaces/fashn-ai/fashn-vton-1.5` (upload your photo + a garment image; use only your own photos).
2. Start it: copy `fashn_server\\.env.example` to `fashn_server\\.env`, set `FASHN_API_KEY`, then double-click
   `start_ai_server.bat` (website: `start_website.bat`). Or by hand: `python fashn_server/server.py` from the AI environment.
   Check `http://GPU-MACHINE:5001/health`.
3. In the web app's `.env`: `TRYON_ENGINE=fashn`, `FASHN_SERVER_URL=http://GPU-MACHINE:5001`,
   `FASHN_API_KEY=your-secret`. Restart Django.
4. In the admin set each dress's **Garment type** (top/suit jacket, bottoms, or full-body).

If the GPU server is down or fails, customers automatically get the quick `warp` preview instead.
`FASHN_DUMMY=1` starts the server without the model (it returns the photo unchanged), only to test
the connection. **The real model path has not been tested by me**; only the web-app-to-server
connection has.

## Installing on a self-service kiosk (Windows)

**First check the kiosk's hardware.** Open Task Manager -> Performance -> GPU (or run `nvidia-smi`). Most kiosk PCs
have only built-in graphics (no NVIDIA card). The realistic AI engine needs an NVIDIA GPU with ~8 GB VRAM, so on
a typical kiosk use `TRYON_ENGINE=warp`, or point `TRYON_ENGINE=fashn` at a GPU server elsewhere.

**Layout A - everything on the kiosk (simplest).** The kiosk's browser opens `http://localhost:8000`
(camera works on localhost without HTTPS).
1. Install Python 3.10-3.12 ("Add to PATH") on the kiosk.
2. Copy the project folder to the kiosk (USB stick or zip). Do NOT copy `venv`; it is rebuilt there.
   Prepare your dresses on your own PC first and copy `db.sqlite3` and the `media\` folder too
   (dress upload needs ~2.5 GB RAM and downloads AI models; the kiosk does not need to do that).
3. Double-click `kiosk\setup.bat`. Edit `.env` (set `SECRET_KEY`; the template is `kiosk\env.kiosk.example`).
4. Run `python manage.py createsuperuser` in the activated venv (`venv\Scripts\activate`).
5. Double-click `kiosk\start_kiosk.bat`: it starts the server (waitress; gunicorn does not run on Windows)
   and opens Chrome (or Edge) full screen. Put a shortcut to it in `shell:startup` to start on boot.
6. Windows: create an auto-login kiosk user, disable sleep, and consider "Assigned Access" kiosk mode.
7. Mount the camera at chest height; customers stand ~2-3 m away. Photos are processed in memory and are not saved.

**Layout B - kiosk is only a screen + camera; the server is another computer (needed for a GPU server).**
Run the website there and open its address in the kiosk browser. **Browsers block the camera on plain `http://` pages
that are not localhost.** Either serve the site over HTTPS (e.g. a local certificate with Caddy or mkcert), or start
Chrome with `--unsafely-treat-insecure-origin-as-secure=http://SERVER-IP:8000 --user-data-dir=C:\kiosk-profile`.

The kiosk scripts were written for Windows and tested only as far as the server side (production mode with waitress);
the .bat files and Chrome flags themselves were not run on a real Windows kiosk.

## Paid option without any GPU: OpenAI image model (`TRYON_ENGINE=openai`)

Set `TRYON_ENGINE=openai`, `OPENAI_API_KEY=sk-...` (billing must be enabled on your OpenAI account) and optionally
`OPENAI_IMAGE_MODEL` / `OPENAI_IMAGE_QUALITY` (`low`/`medium`/`high`). The customer photo and the cleaned garment image are
sent to OpenAI with a prompt that asks it to change only the clothing. You pay per photo and customer photos go to a
third party (get consent). It is a general image editor, not a dedicated try-on model, so check face/pose/garment
accuracy on your own photos. The key stays in `.env` and is never logged. If it fails, the page shows the reason and
the quick preview is used.

## Gemini image editing (`TRYON_ENGINE=gemini`)

Create a Gemini API key in Google AI Studio, then set `TRYON_ENGINE=gemini`,
`GEMINI_API_KEY=...`, and optionally `GEMINI_IMAGE_MODEL` (defaults to `gemini-nano-banana-2.1`) or
`GEMINI_TIMEOUT` (seconds) in `.env`. The customer photo and cleaned garment image are sent to Google for editing;
get consent and check Google's terms and pricing for your account. The key stays in `.env` and is never logged.
If the service fails, the page shows the reason and uses the quick preview instead.

## Grok Imagine image editing (`TRYON_ENGINE=grok`)

Create an xAI API key, then set `TRYON_ENGINE=grok`, `XAI_API_KEY=...`, and optionally
`XAI_IMAGE_MODEL` (defaults to `grok-imagine-image-2.0`) or `XAI_TIMEOUT` (seconds) in `.env`.
The customer photo and cleaned garment image are sent to xAI for editing; get consent and check xAI's terms and
pricing for your account. The key stays in `.env` and is never logged. If the service fails, the page shows the
reason and uses the quick preview instead.

## Claude-assisted Grok image editing (`TRYON_ENGINE=claude_grok`)

Claude analyzes the customer and garment images and provides a short garment description; Grok Imagine still
generates the edited image. Create API keys with Anthropic and xAI, then set `TRYON_ENGINE=claude_grok`,
`ANTHROPIC_API_KEY=...`, and `XAI_API_KEY=...` in `.env`. Optionally set `CLAUDE_MODEL` (defaults to
`claude-sonnet-4-6`), `CLAUDE_TIMEOUT` (defaults to 120 seconds), `XAI_IMAGE_MODEL`, or `XAI_TIMEOUT`.
Both services receive the images, so get consent and check each provider's terms and pricing. Claude itself
does not generate the edited image. If either API fails, the page shows the reason and uses the quick preview.

## Cloud GPU (Spheron or any Ubuntu GPU rental)

For a real GPU without buying one. Rent an NVIDIA GPU (an RTX 4090 24 GB is plenty; the model needs ~8 GB), then:
1. On your PC: `scp -r fashn_server cloud\\fashn_cloud_setup.sh USER@IP:~/`
2. On the server: `export FASHN_API_KEY="your-secret"` then `bash fashn_cloud_setup.sh` (installs the model, downloads weights, starts the server).
3. On your PC: edit `cloud\\tunnel_to_gpu.bat` (IP, user) and double-click it; keep it open. The website then reaches the cloud GPU at
   `http://127.0.0.1:5001` through a secure SSH tunnel, so the website `.env` is the same as for a local AI server
   (`TRYON_ENGINE=fashn`, `FASHN_SERVER_URL=http://127.0.0.1:5001`, matching `FASHN_API_KEY`).
4. **Terminate the instance when you are done**; it bills until you do. Customer photos are processed on the rented machine:
   get consent and publish a privacy notice. The scripts were dry-run tested only (no real GPU rental).

## 1. Setup (development, SQLite)

```bash
cd virtual-tryon
python3 -m venv venv
source venv/bin/activate            # Windows: venv\Scripts\activate
pip install --upgrade pip
pip install -r requirements.txt
cp .env.example .env                # Windows: copy .env.example .env
```

Generate a secret key and paste it into `.env` as `SECRET_KEY`:

```bash
python -c "from django.core.management.utils import get_random_secret_key as g; print(g())"
```

> Python 3.10–3.12 recommended. If `mysqlclient` fails to install and you only want
> SQLite for now, comment that line out of `requirements.txt` first.

```bash
python manage.py migrate
python manage.py createsuperuser
python manage.py runserver
```

Open http://127.0.0.1:8000/ (camera works on `localhost` without HTTPS).

## 2. Add your first dress and test

1. Go to http://127.0.0.1:8000/admin/ and log in.
2. **Catalog → Dresses → Add**. Fill name, category, price and upload a **front-facing
   dress photo** (plain background works best; a mannequin or flat-lay is fine).
3. Leave **Remove model's face & skin** ticked if the photo shows a person wearing the dress (untick it for flat-lay / mannequin photos). Click **Save**. The **first save takes a while**: rembg downloads its model
   (~170 MB, once) and removes the background. Afterwards the admin shows the
   transparent result and a thumbnail.
4. Open the home page, allow the camera, stand ~1.5–2 m back so head, shoulders and
   hips are visible, and click the dress.

**Hiding the customer's own clothes:** the try-on repaints the customer's torso and sleeves in the dress's
colour (keeping the natural folds/shadows) before drawing the dress on top, so it looks like the dress
*replaces* their outfit. Choose how much to cover per dress in the admin (**Sleeves**: torso only / upper arms /
whole arms). After updating, run `python manage.py migrate` (adds the `anchors` and `sleeves` fields).

**Full screen:** use the ⛶ button on the camera (or press `F`). In full screen you can switch dresses
from the strip at the bottom and take a snapshot with the 📸 button. Esc exits.

**Full-length outfits:** set the dress's **Garment type** in the admin. For `one-pieces` (gowns, full suits with
trousers) the camera guide switches to a head-to-feet outline and the result warns if the legs were not in the
photo; stand ~3 m back with the camera at waist height. The photo can only show what the camera sees, so a
webcam that cuts you off at the thighs can never show trousers or a long hem.

**Dress photo tips:** front-facing, whole garment visible, plain background. If a dress still shows part of
the model, open it in the admin, tick/untick *Remove model's face & skin*, or select dresses in the list and
run the action **Re-run background / model removal**.

**Sarees and other draped outfits:** upload a photo of a person (or mannequin) wearing it, set **Garment type** to
`one-pieces`, and set **Sleeves** to `elbow` (short blouse sleeves). A clothing-segmentation model
(`USE_CLOTH_SEG=True`, on by default, ~170 MB extra download on the first upload) cuts out only the fabric, so
faces, arms, necks and mannequin stands are removed. Photos that are already transparent PNGs are used as supplied.
Best results: front-facing, whole saree visible, at least ~1000 px tall (small photos look blurry when enlarged).
A photo of just a mannequin torso has no body to line up with, so the saree is stretched onto the customer
and looks weaker than a photo of a person wearing it. Uploading a dress needs about 2.5 GB of free RAM for a
few seconds.

## 3. MySQL (production)

```sql
CREATE DATABASE tryon CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'tryon_user'@'localhost' IDENTIFIED BY 'a-strong-password';
GRANT ALL PRIVILEGES ON tryon.* TO 'tryon_user'@'localhost';
FLUSH PRIVILEGES;
```

In `.env`:

```
DB_ENGINE=mysql
DB_NAME=tryon
DB_USER=tryon_user
DB_PASSWORD=a-strong-password
DB_HOST=127.0.0.1
DB_PORT=3306
```

Then `python manage.py migrate`. `mysqlclient` needs system libs
(Ubuntu: `sudo apt install default-libmysqlclient-dev build-essential pkg-config`).
**PyMySQL fallback:** remove `mysqlclient` from requirements, run `pip install PyMySQL==1.1.1`;
`settings.py` detects this automatically.

## 4. Deployment notes (gunicorn + nginx + HTTPS)

**HTTPS is mandatory**: browsers only allow `getUserMedia` on HTTPS (or localhost).

`.env` for production: `DEBUG=False`, a real `SECRET_KEY`, `ALLOWED_HOSTS=tryon.example.com`,
`CSRF_TRUSTED_ORIGINS=https://tryon.example.com`, `SECURE_HTTPS=True`.

```bash
python manage.py collectstatic --noinput
gunicorn config.wsgi:application --workers 3 --threads 2 --timeout 60 --bind 127.0.0.1:8000
```

Each worker/thread builds its own MediaPipe Pose object, so memory use is roughly
(workers x threads) x ~150 MB. Workers ~ CPU cores is a good start.

Example nginx server block (get a free certificate with `certbot --nginx`):

```nginx
server {
    listen 443 ssl;
    server_name tryon.example.com;
    # ssl_certificate lines are added by certbot

    client_max_body_size 5m;

    location /media/  { alias /path/to/virtual-tryon/media/; expires 7d; }
    location /static/ { alias /path/to/virtual-tryon/staticfiles/; expires 7d; }

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```

Static files are also served by WhiteNoise as a fallback. **Serve `/media/` with nginx**
(WhiteNoise does not serve user uploads). Run gunicorn under systemd for restarts.

## 5. Project layout

```
config/    settings (env driven), urls, wsgi
catalog/   Dress model (rembg on save), admin, gallery view + template
tryon/     services/pose_detector.py (MediaPipe + EMA smoothing)
           services/overlay.py       (quad + perspective warp + alpha blend)
           services/dress_cache.py   (in-memory decoded dress images)
           views.py                  (POST /api/tryon/)
static/    css/style.css, js/tryon.js
templates/ base.html
```

Fit is tunable at the top of `tryon/services/overlay.py` (`WIDEN`, `LENGTH_FACTOR`,
`HEM_FLARE`, `NECK_LIFT`).

## 6. Known limitations and next steps

- **Flat 2D warp.** A single perspective transform can't bend with the body, so side
  views, arms crossing the torso and strong turns look wrong. Dress photos should be
  front-facing and symmetric.
- **No occlusion.** The dress is drawn over arms/hands. Fix: use MediaPipe's
  segmentation mask (`enable_segmentation=True`) to paint arms back on top.
- **Latency.** Every frame is a network round trip + CPU inference (~150-400 ms).
  Smoother real-time: run **MediaPipe Pose in the browser** (`@mediapipe/tasks-vision`,
  WASM/WebGL), warp the dress on a `<canvas>`, and send nothing to the server.
  The server would then only serve dress PNGs.
- **Multi-process smoothing state** is per gunicorn process; harmless (only affects jitter).
- **Realistic try-on:** upgrade to diffusion models such as **OOTDiffusion** or
  **IDM-VTON** behind a separate GPU API (e.g. FastAPI). Use the current live preview
  for positioning and call the model for a high-quality "final render" on a button press.
- **Sleeve and length options:** add `sleeve_type` / `length` fields to `Dress` and
  per-type quad rules (e.g. `LENGTH_FACTOR` knee/midi/maxi), or separate sleeve
  layers anchored to elbow landmarks (13, 14).
- **Rembg quality** depends on the photo; try `isnet-general-use` or `u2net_cloth_seg`
  models in `catalog/services/image_utils.py` for clothing.
- No automated tests are included yet.
