CLIPnosis

CLIPnosis recovers text prompts for images generated by SimianLuo/LCM_Dreamshaper_v7 under a known, fixed LCM configuration (model checkpoint, scheduler, inference steps, guidance scale, resolution, and seed). Given a target image, the system searches for prompts that, fed through the same generator at the same seed, re-render images visually close to the target.

The test set is six target images. For each one the pipeline returns three candidate prompts and their renders, ranked by image-side similarity using CLIP image-image similarity, LPIPS perceptual distance (AlexNet), and pixel RMSE.

Approach

The method has two main stages. A seed pool of candidate prompts is produced by a captioner ensemble, and an evolutionary search then refines that pool against the actual evaluation metrics.

Stage 1: Seed pool

Four vision-language models propose prompts for each target: CLIP Interrogator, BLIP-2, GIT (microsoft/git-large-coco), and Qwen2.5-VL. The four come from different training distributions and produce stylistically different outputs, so pooling them covers more of prompt space than any single model would. Near-duplicates are removed by token-level Jaccard similarity above 0.85, leaving roughly 20–30 unique candidates per target.

The seed pool is the initial population for the search.

Stage 2: Evolutionary search

For each target, the seed pool initialises a population of 24 candidates. At each generation the candidates are rendered with the locked LCM seed and scored on the three evaluation metrics. A composite fitness (a weighted sum of the z-normalised metrics) ranks the population, and the top elites carry over to the next generation unchanged. New candidates are produced by three operators:

1. Qwen2.5-VL rewriting an elite, with the target image and the parent's metric scores in context.
2. Clause-level crossover between two elite parents.
3. One wildcard candidate per generation, built from a seed-pool prompt with random style modifiers from a curated bank.

The wildcard prevents the population from collapsing onto a single local optimum. The search runs for up to 25 generations or stops earlier when the best fitness plateaus.

To measure search variance, the procedure is repeated with three different optimiser seeds per target while the LCM render seed remains fixed.

Stage 3: Selection

After the sweep, every candidate scored across the three optimiser seeds is pooled per target. The top three are selected by ranking on a weighted sum of the raw metric values (no normalisation), and their prompts and renders are the final result for that target.

Why this structure

Because the generator and render seed are known, every candidate can be rendered and scored against the actual evaluation criterion. The search therefore optimises directly against the metric used for ranking, rather than against a surrogate such as CLIP-only similarity used by most prompt-inversion methods. Closing the loop with full rendering inside the search is the central design decision.

Setup

The project uses uv for dependency management.

uv sync

This installs PyTorch (with ROCm wheels), diffusers, transformers, LPIPS, and the captioner dependencies. Model weights download to ~/.cache/huggingface on first use.

Target images live in data/. The filename encodes the render seed: 7836.png uses seed 7836, and 1159_25.png uses seed 1159.

Running the pipeline

Each stage has a command-line entry point in clipnosis.scripts.

1. Build the seed pool

uv run python -m clipnosis.scripts.run_seed_pool

Runs the four captioners over every target, dedupes, and writes per-target JSON files to outputs/<timestamp>_seedpool/seed_pool/. Each candidate is rendered with the locked seed and scored, producing a baseline seed_pool_scores.csv that the search is expected to improve on.

The default Qwen-VL captioner is the GPTQ-Int4 quant of the 7B (hfl/Qwen2.5-VL-7B-Instruct-GPTQ-Int4, ~6.5 GB). To use bf16 weights instead, pass --qwen-model-id Qwen/Qwen2.5-VL-7B-Instruct (or -32B-Instruct on a 128 GB unified box, or -3B-Instruct on a 16 GB VRAM box).

2. Run the search on a single target (optional)

uv run python -m clipnosis.scripts.run_search \
    --target 1159_25.png \
    --seed-pool-dir outputs/<seedpool_run> \
    --search-seed 0

Used for validating the loop end-to-end and for tuning hyperparameters before committing to the full sweep. The search uses Qwen-VL mutation by default; --no-llm disables the LLM mutator and falls back to heuristic mutation.

3. Run the full sweep

uv run python -m clipnosis.scripts.run_all \
    --seed-pool-dir outputs/<seedpool_run> \
    --identity sweep-final

Six targets times three optimiser seeds: eighteen search trajectories. The sweep is resumable; if a run is interrupted, rerun with --resume --resume-dir outputs/<sweep_run> to skip pairs whose archive already exists.

4. Build the final tables and selected outputs

uv run python -m clipnosis.scripts.build_report \
    --sweep-dir outputs/<sweep_run>

Pools archives across optimiser seeds, selects the top three per target, and writes the result layout under outputs/<sweep_run>/final/: per-target folders containing the target image, three prompts, three renders, a metrics CSV and LaTeX table; cross-set mean and standard deviation tables; per-search-seed standard deviation; a machine-readable results.json; and a short summary.md.

Hardware

The default configuration uses the GPTQ-Int4 quant of Qwen2.5-VL-7B (hfl/Qwen2.5-VL-7B-Instruct-GPTQ-Int4, ~6.5 GB resident) both as the fourth captioner and as the mutation model. The quant runtime is gptqmodel; on ROCm the captioner and mutator force gptqmodel's pure-PyTorch backend because ExLlamaV2's CUDA-only kernels can't build without CUDA_HOME. On a CUDA box, pass --qwen-exllama / --llm-exllama for the ~2-3× faster ExLlamaV2 path.

For higher quality on a box with headroom (128 GB unified RAM), swap in the 32B bf16 variant:

uv run python -m clipnosis.scripts.run_all \
    --seed-pool-dir outputs/<seedpool_run> \
    --llm-model-id Qwen/Qwen2.5-VL-32B-Instruct \
    --llm-device-map auto \
    --identity sweep-32b

For a no-quant fallback (e.g. when gptqmodel is unavailable), use the 3B bf16 variant:

uv run python -m clipnosis.scripts.run_all \
    --seed-pool-dir outputs/<seedpool_run> \
    --llm-model-id Qwen/Qwen2.5-VL-3B-Instruct \
    --identity sweep-3b

Repository layout

src/clipnosis/
├── config.py          locked LCM hyperparameters and project paths
├── targets.py         target discovery and filename-to-seed parsing
├── rendering.py       LCM pipeline loader and render_prompt
├── artifacts.py       run directories, image saving, CSV writing
├── evaluation/        scorers (CLIP, LPIPS, RMSE), composite fitness, result tables
├── captioners/        four seed-pool captioners and Jaccard dedupe
├── search/            evolutionary loop, mutation, crossover, modifier vocab
├── llm/               Qwen-VL mutation client
└── scripts/           run_seed_pool, run_search, run_all, build_report