Initial commit

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

saved/image_recognition_spike/REPORT.md

@@ -0,0 +1,87 @@
+# Image recognition de-risking — report
+
+Goal: figure out how the app recognizes a Pokemon when the phone is
+pointed at a figure/toy, before committing to an architecture.
+
+## Test set
+
+5 classes (Bulbasaur, Squirtle, Pikachu, Eevee, Charizard), grown over the
+course of the spike to 12 query images spanning very different visual
+styles on purpose: stock photos, Pokemon GO AR screenshots, plush toys
+(including two real photos of figures the user owns), a TCG card, a tiny
+battle sprite, and anime screencaps. Two reference sets were tried:
+scraped stock photos, and Bulbapedia's official artwork.
+
+## Approaches tried, in order
+
+### 1. CLIP (ViT-B-32, OpenAI weights) + nearest-neighbor
+
+Embed a reference image per species, embed the query, cosine-similarity
+match. Initial run used the wrong open_clip model variant (`ViT-B-32`
+instead of `ViT-B-32-quickgelu`), which silently degrades OpenAI-weight
+embeddings — worth remembering if this comes up again.
+
+- Photo references: **4/12** correct top-1.
+- Bulbapedia art references: **8/12** correct top-1.
+
+### 2. DINOv2 (facebook/dinov2-base) + nearest-neighbor
+
+Self-supervised, built for instance/visual similarity rather than
+text-image alignment — expected to beat CLIP at this specific task.
+
+- Photo references: **6/12**.
+- Bulbapedia art references: **8/12**.
+
+**Finding across both:** official art references consistently beat random
+stock-photo references, regardless of model — the illustration-vs-photo
+domain gap we worried about mattered less than material/form-factor
+mismatch (plush vs. rigid figure vs. flat art). `charizard_plush` failed
+in literally every embedding config tried (0/4) — plush toys are the
+genuinely hard case for this whole approach, not photos-vs-domain style.
+
+### 3. Direct vision-LLM recognition (the "cheat")
+
+Skip reference images and embeddings entirely — ask a vision-language
+model "what Pokemon is this" and let its pretrained world knowledge do
+the work.
+
+- Claude (me, just looking at the images): **12/12**.
+- Self-hosted Qwen2.5-VL-3B-Instruct, run locally on the RTX 5070 Ti:
+  **10/12** cold, no fine-tuning, no reference images at all. The 2
+  misses were the two genuinely hardest images in the set (a tiny
+  213x240 keychain thumbnail → correctly returned "unknown" rather than
+  a wrong guess; and a plush the user themselves said "looks like shit,
+  not even sure that's Charizard").
+
+## Decision
+
+Went with **self-hosted VLM recognition** (Qwen2.5-VL-3B-Instruct) over
+the embedding/nearest-neighbor approach. Reasons:
+- Meaningfully higher accuracy (10/12 vs. best embedding score of 8/12).
+- No reference-image sourcing/maintenance needed for 1000+ species —
+  eliminates the "content volume" risk from the original risk assessment
+  entirely.
+- Degrades safely: genuinely ambiguous images tend to get "unknown"
+  rather than a confident wrong answer.
+
+Trade-off accepted: requires a GPU server reachable over the network at
+recognition time (already an accepted dependency — the original plan
+always involved uploading the photo to a home server).
+
+## What shipped from this
+
+`server/` — FastAPI wrapper around the same Qwen2.5-VL-3B pipeline,
+running on this Windows machine (chosen over buying a GPU for Unraid).
+`POST /identify` takes a photo, returns `{recognized, species,
+raw_response}`. Verified working end-to-end over real HTTP.
+
+## Open questions / not yet tested
+
+- Accuracy at real scale (1000+ candidate species) is untested — only
+  ever tried 5 classes. Confusion likely increases with more classes.
+- Never tested against the user's own figures except for two Charizard
+  photos (both plush) — the real target (rigid painted figures) hasn't
+  been tried.
+- Larger models (Qwen2.5-VL-7B+) not tried — likely closes some of the
+  remaining gap to the 12/12 upper bound, at the cost of latency/VRAM.
+- No latency/throughput measurement done — only correctness.

saved/image_recognition_spike/match_spike.py

@@ -0,0 +1,83 @@
+"""
+De-risking spike: can a generic vision embedding model (CLIP) tell Pokemon
+figures/toys apart via nearest-neighbor lookup, with no training?
+
+Pipeline: embed every image under images/reference/ (one file per Pokemon,
+filename = class name) to build a small reference index, then embed every
+image under images/query/ and report the nearest reference neighbor +
+cosine similarity. A "pass" is query/<name>.jpg matching reference/<name>.jpg
+as the top-1 hit.
+
+Usage: .venv\\Scripts\\python.exe match_spike.py
+"""
+
+import sys
+from pathlib import Path
+
+import open_clip
+import torch
+from PIL import Image
+
+HERE = Path(__file__).parent
+REF_DIR = HERE / "images" / (sys.argv[1] if len(sys.argv) > 1 else "reference")
+QUERY_DIR = HERE / "images" / "query"
+
+MODEL_NAME = "ViT-B-32-quickgelu"
+PRETRAINED = "openai"
+
+
+def load_model():
+    model, _, preprocess = open_clip.create_model_and_transforms(
+        MODEL_NAME, pretrained=PRETRAINED
+    )
+    model.eval()
+    return model, preprocess
+
+
+def embed_image(model, preprocess, path: Path) -> torch.Tensor:
+    image = preprocess(Image.open(path).convert("RGB")).unsqueeze(0)
+    with torch.no_grad():
+        features = model.encode_image(image)
+        features = features / features.norm(dim=-1, keepdim=True)
+    return features.squeeze(0)
+
+
+def main():
+    print(f"Loading {MODEL_NAME} ({PRETRAINED})...")
+    model, preprocess = load_model()
+
+    ref_paths = (
+        sorted(REF_DIR.glob("*.jpg"))
+        + sorted(REF_DIR.glob("*.webp"))
+        + sorted(REF_DIR.glob("*.png"))
+    )
+    query_paths = (
+        sorted(QUERY_DIR.glob("*.jpg"))
+        + sorted(QUERY_DIR.glob("*.webp"))
+        + sorted(QUERY_DIR.glob("*.png"))
+    )
+
+    print(f"Embedding {len(ref_paths)} reference images...")
+    ref_names = [p.stem for p in ref_paths]
+    ref_embeds = torch.stack([embed_image(model, preprocess, p) for p in ref_paths])
+
+    print(f"Embedding {len(query_paths)} query images...\n")
+    correct = 0
+    for qpath in query_paths:
+        qembed = embed_image(model, preprocess, qpath)
+        sims = ref_embeds @ qembed  # cosine similarity, both sides unit-norm
+        ranked = sorted(zip(ref_names, sims.tolist()), key=lambda x: -x[1])
+        top_name, top_sim = ranked[0]
+        expected = qpath.stem.split("_")[0]
+        is_correct = top_name == expected
+        correct += is_correct
+        marker = "OK  " if is_correct else "MISS"
+        print(f"[{marker}] query={qpath.stem:<18} -> best={top_name:<10} sim={top_sim:.4f}")
+        runner_up = ", ".join(f"{n}={s:.3f}" for n, s in ranked[1:4])
+        print(f"         runner-up: {runner_up}")
+
+    print(f"\n{correct}/{len(query_paths)} correct top-1 matches")
+
+
+if __name__ == "__main__":
+    main()

saved/image_recognition_spike/match_spike_dinov2.py

@@ -0,0 +1,82 @@
+"""
+Same spike as match_spike.py, but using DINOv2 instead of CLIP.
+
+DINOv2 is trained with a self-supervised image-only objective specifically
+aimed at instance/fine-grained visual similarity, which is a better fit for
+"is this query photo the same object as this reference photo" than CLIP
+(CLIP is trained for text-image alignment and tends to cluster images by
+generic scene/style rather than object identity).
+
+Usage: .venv\\Scripts\\python.exe match_spike_dinov2.py
+"""
+
+import sys
+from pathlib import Path
+
+import torch
+from PIL import Image
+from transformers import AutoImageProcessor, AutoModel
+
+HERE = Path(__file__).parent
+REF_DIR = HERE / "images" / (sys.argv[1] if len(sys.argv) > 1 else "reference")
+QUERY_DIR = HERE / "images" / "query"
+
+MODEL_NAME = "facebook/dinov2-base"
+
+
+def load_model():
+    processor = AutoImageProcessor.from_pretrained(MODEL_NAME)
+    model = AutoModel.from_pretrained(MODEL_NAME)
+    model.eval()
+    return model, processor
+
+
+def embed_image(model, processor, path: Path) -> torch.Tensor:
+    image = Image.open(path).convert("RGB")
+    inputs = processor(images=image, return_tensors="pt")
+    with torch.no_grad():
+        outputs = model(**inputs)
+        features = outputs.last_hidden_state[:, 0, :]  # CLS token
+        features = features / features.norm(dim=-1, keepdim=True)
+    return features.squeeze(0)
+
+
+def main():
+    print(f"Loading {MODEL_NAME}...")
+    model, processor = load_model()
+
+    ref_paths = (
+        sorted(REF_DIR.glob("*.jpg"))
+        + sorted(REF_DIR.glob("*.webp"))
+        + sorted(REF_DIR.glob("*.png"))
+    )
+    query_paths = (
+        sorted(QUERY_DIR.glob("*.jpg"))
+        + sorted(QUERY_DIR.glob("*.webp"))
+        + sorted(QUERY_DIR.glob("*.png"))
+    )
+
+    print(f"Embedding {len(ref_paths)} reference images...")
+    ref_names = [p.stem for p in ref_paths]
+    ref_embeds = torch.stack([embed_image(model, processor, p) for p in ref_paths])
+
+    print(f"Embedding {len(query_paths)} query images...\n")
+    correct = 0
+    for qpath in query_paths:
+        qembed = embed_image(model, processor, qpath)
+        sims = ref_embeds @ qembed
+        ranked = sorted(zip(ref_names, sims.tolist()), key=lambda x: -x[1])
+        top_name, top_sim = ranked[0]
+        expected = qpath.stem.split("_")[0]
+        is_correct = top_name == expected
+        correct += is_correct
+        marker = "OK  " if is_correct else "MISS"
+        print(f"[{marker}] query={qpath.stem:<18} -> best={top_name:<10} sim={top_sim:.4f}")
+        runner_up = ", ".join(f"{n}={s:.3f}" for n, s in ranked[1:4])
+        print(f"         runner-up: {runner_up}")
+
+    print(f"\n{correct}/{len(query_paths)} correct top-1 matches")
+
+
+if __name__ == "__main__":
+    main()

saved/image_recognition_spike/match_spike_vlm.py

@@ -0,0 +1,87 @@
+"""
+De-risking spike, take 3: instead of embedding + nearest-neighbor lookup,
+just ask a self-hosted vision-language model directly what Pokemon is in
+the photo. No reference images, no vector index -- the model's own
+pretrained world knowledge does the recognition.
+
+Usage: .venv_vlm\\Scripts\\python.exe match_spike_vlm.py
+"""
+
+from pathlib import Path
+
+import torch
+from PIL import Image
+from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
+
+HERE = Path(__file__).parent
+QUERY_DIR = HERE / "images" / "query"
+
+MODEL_NAME = "Qwen/Qwen2.5-VL-3B-Instruct"
+
+PROMPT = (
+    "You are the recognition system inside a Pokedex app. Identify the "
+    "Pokemon species shown in this image, even if it's a toy, plush, "
+    "trading card, sprite, fan art, or in-game screenshot of it. "
+    "Reply with ONLY the species name, nothing else. If no Pokemon is "
+    "clearly depicted, reply with exactly: unknown"
+)
+
+
+def load_model():
+    processor = AutoProcessor.from_pretrained(MODEL_NAME)
+    model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+        MODEL_NAME, torch_dtype=torch.bfloat16, device_map="cuda"
+    )
+    model.eval()
+    return model, processor
+
+
+def identify(model, processor, path: Path) -> str:
+    image = Image.open(path).convert("RGB")
+    messages = [
+        {
+            "role": "user",
+            "content": [
+                {"type": "image", "image": image},
+                {"type": "text", "text": PROMPT},
+            ],
+        }
+    ]
+    text = processor.apply_chat_template(
+        messages, tokenize=False, add_generation_prompt=True
+    )
+    inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")
+    with torch.no_grad():
+        generated = model.generate(**inputs, max_new_tokens=16)
+    trimmed = generated[:, inputs["input_ids"].shape[1] :]
+    output = processor.batch_decode(
+        trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=True
+    )[0]
+    return output.strip()
+
+
+def main():
+    print(f"Loading {MODEL_NAME} on GPU...")
+    model, processor = load_model()
+
+    query_paths = (
+        sorted(QUERY_DIR.glob("*.jpg"))
+        + sorted(QUERY_DIR.glob("*.webp"))
+        + sorted(QUERY_DIR.glob("*.png"))
+    )
+
+    print(f"Identifying {len(query_paths)} query images...\n")
+    correct = 0
+    for qpath in query_paths:
+        expected = qpath.stem.split("_")[0]
+        answer = identify(model, processor, qpath)
+        is_correct = answer.strip().lower() == expected.lower()
+        correct += is_correct
+        marker = "OK  " if is_correct else "MISS"
+        print(f"[{marker}] query={qpath.stem:<18} expected={expected:<10} model_said={answer!r}")
+
+    print(f"\n{correct}/{len(query_paths)} correct")
+
+
+if __name__ == "__main__":
+    main()

saved/image_recognition_spike/requirements.txt

@@ -0,0 +1,4 @@
+torch --index-url https://download.pytorch.org/whl/cpu
+open_clip_torch
+pillow
+numpy