How does selfie search in event photos work? Face matching and GDPR

What are the steps behind selfie-based photo search?

In Photonza the feature is called “Find Me”: a guest joins via QR, takes a selfie and sees the uploaded photos they appear in. There are four steps:

• Detection: YuNet returns a box, a score and 5 landmarks (eyes, nose, mouth corners) per face.
• Alignment: the landmarks map the face onto a standard 112×112 crop via a least-squares similarity transform.
• Signature: SFace turns the crop into an L2-normalized 128-dimensional vector.
• Comparison: cosine similarity between selfie and photo signatures, labelled “sure” or “possible” against thresholds.

Both models come from the OpenCV Zoo (YuNet MIT, SFace Apache-2.0). The stored model id is yunet-2023mar+sface-2021dec-int8; the browser downloads about 230 KB for YuNet and 9.9 MB for int8 SFace. No external face recognition API is involved.

Two selfie details: the signature is the mean of the selfie and its horizontal mirror image (with left and right landmarks swapped), and if more than one face larger than 6% of the frame is present, the guest is asked to retake it. A guest can add up to 5 selfies; the best-matching profile wins.

Why did we drop the first model for YuNet + SFace?

The first version used BlazeFace for detection and the Human library’s FaceRes model for 1024-dimensional signatures. On 23 September 2026 we compared both on 7 real group photos and a selfie of someone who appears in 4 of them:

• Old model: the person scored 0.50–0.65, the closest stranger 0.53. One photo of four was found, only as “possible”.
• YuNet + SFace (int8, 128 dimensions): the person scored 0.37–0.55, the closest stranger 0.255 (a face 0.9% of the image width). Four of four found, zero false matches.

The old problem was not the threshold. The two ranges overlapped, so every threshold either missed the guest or showed someone else. The new numbers are lower, but there is a clear gap between them, and tuning only works when such a gap exists.

The switch had a cost. The 20260924000000_face_v2.sql migration changed the vector columns to VECTOR(128), rebuilt the HNSW indexes and deleted all existing face data, including selfie profiles, so guests had to take a new selfie and events within retention were queued for rescanning. The documented rollout order is web, then migration, then scanning server; reversed, a 128-dimensional signature cannot be written into the 1024-wide column.

Where does the computation run: browser or server?

Both. The selfie is processed in the browser with onnxruntime-web on WASM, with models served from the site’s own /models/face-v2/ path. The server receives only the 128 numbers and the detection score. Loading the models takes 0.3 s, processing the selfie another 0.3 s.

Event photos are scanned in a Cloudflare Container running onnxruntime-node, with sharp for decoding. sharp applies EXIF orientation; the earlier tfjs decoder did not, so faces in rotated photos were missed. An upload trigger adds a job row, a cron wakes the container every minute, and jobs are claimed with FOR UPDATE SKIP LOCKED. At most 5 containers run, and idle ones shut down once the queue is empty.

Both sides must produce the same numbers. The core (src/lib/face-v2/) is DOM-free TypeScript, built into lib/face-v2.mjs for the server. An oracle test checks it against OpenCV’s FaceDetectorYN/FaceRecognizerSF: boxes and landmarks within 1.5 px, signature cosine at least 0.995. Browser and server signatures of the same selfie measure 0.9987.

The detector runs at a 1920 px long edge; photos above 2000 px are also scanned as 2×2 tiles with 15% overlap, plus a coarse 640 px pass for faces filling the frame. Up to 50 faces per photo; 0.71 s per photo measured locally (1.82 s before).

How was the match threshold set, and why did a real wedding change it?

The first thresholds came from OpenCV’s recommended 0.363 cosine for SFace and the 7-photo test set. On 28 September 2026 the first real wedding (220 photos, 1,603 faces, 37 guests with a selfie) was reviewed by eye, and those thresholds proved too loose: the 0.33–0.40 band was mostly someone else, while correct matches fell between 0.44 and 0.77. The current rules in src/lib/face-utils.ts:

• 0.55 and above: “sure” regardless of face size.
• 0.48 and above: “sure” if the face is at least 2% of the image width and the detector score at least 0.5.
• 0.42 and above: “possible”. Lower is never shown.

A small or blurry face can only be “possible”; the point is never to show a guest someone else’s photo in a closed event.

Two more rules changed that day. “Person clustering”, which expanded results through photo-to-photo similarity, was removed: one wrong seed pulled the bride, who appears in 150 photos, into female guests’ results. Instead a face now goes to the guest it resembles most; if another guest’s selfie is at least 0.03 closer, it drops out. Accounts of the same person on different browsers (selfies 0.55+ similar) are not rivals. The notes say plainly that the set is still small and thresholds will be recalibrated.

Where is face data stored, and under what condition?

Many people in event photos never use “Find Me”: other guests, staff, children. Since 25 September 2026 new events default to face_store_policy = 'consented'. The scanner computes every signature, compares it in memory with the event’s consenting selfie profiles, and writes it to face_embeddings only if it is at least 0.33 similar to one of them, recording which in matched_profile_ids. Everything else is discarded unwritten. The storage threshold sits deliberately below the 0.42 display threshold so display can be retuned on real data.

For guests who take a selfie later, each face keeps a detection record without any signature (face_detections): box, 5 landmarks, score and the aligned 112×112 crop. A new profile triggers a rescan request; the server re-signs the crops against that profile only, without decoding full photos. Crops sit in the database as bytea in lossless WebP (about 12 KB). JPEG at q92 shifted signatures by up to 0.08 across 242 faces (worst round-trip cosine 0.88); lossless WebP reproduces them exactly.

A 1,000-photo load test produced 34,577 detections and about 414 MB of crops. A single-call rescan overran its 900-second lease and restarted from scratch, so it now runs in 45-second slices with a cursor. Only the server-side service role can access these tables.

How is the KVKK explicit consent flow enforced on screen and on the server?

This is not legal advice; it describes what the product does. Under KVKK, Turkey’s data protection law, a face signature is special-category data, and Photonza relies on the guest’s explicit consent.

• Notice and consent are separate: a three-point summary links to the full biometric notice, which is not “accepted”; the checkbox text is the consent statement itself. The documents base this on principle decision 2026/347.
• Two boxes, both unchecked by default: biometric consent and an 18+ declaration.
• Declining restricts nothing; the gallery stays open. The necessity and proportionality assessment relies on this free, non-biometric alternative.
• The server enforces it. /api/register-face looks for a non-withdrawn kind='biometric' record for the current text version; without one, and without a matching consent payload, it returns consent_required and writes nothing. A ticked box creates the record in the same request, before the signature.
• Consent is versioned. A meaningful text change bumps LEGAL_VERSIONS.biometric (currently 2026-09-25e) and the consent screen reappears on the next selfie.

user_consents stores user, event, kind, version, locale and timestamps. The endpoints also check active participation, event status, the event’s face recognition switch and the photographer’s permission. The stated purpose excludes identification, cross-event matching, model training and advertising.

When and how is face data deleted?

Face data is deleted with the event at most 30 days after it ends; the retention policy justifies this as the window in which guests still download photos. Expired events are marked, and an hourly cleanup job removes files and rows. The endpoints also close selfie registration and search 30 days after the end time.

Earlier paths:

• “Delete my face data” calls delete_my_face_data: profiles and matches are deleted and withdrawn_at is set on the consent. Leaving the event (leave_event) calls it too.
• A trigger (trg_release_profile_face_embeddings) then removes the profile from stored signatures and deletes those tied to no other consenting profile; pending rescans cascade.
• Deleting a photo removes its signatures and detection records via foreign-key CASCADE.

Deletion needed its own scaling work. Deleting the load-test event in one statement (35,000 rows, about 420 MB of crops) hit the 8-second statement_timeout and failed with HTTP 500. purge_event_face_data and purge_user_face_data now delete in bounded batches until they return 0, before the event or account itself is deleted.

Consent records are kept as proof while the relationship lasts, then purged three years after withdrawal, replacement or account deletion. The first version also purged active consents after three years; a KVKK review caught it and a migration fixed it.

What to check if you are building something similar

Our checklist from building this in Photonza and correcting it after a real wedding:

• Measure the model before tuning thresholds. If same-person and stranger ranges overlap, no threshold helps; switching models took us from 1/4 to 4/4.
• Make browser and server produce the same signature, with a shared pure core and an oracle test (ours requires cosine ≥ 0.995).
• Calibrate on a real event and review results by eye. Someone in hundreds of frames, like a bride, breaks any chain-expanding algorithm.
• Never count small or low-score faces as sure matches.
• Do not store signatures of people who have not consented; for later rescans keep lossless crops, not templates.
• Enforce versioned consent on the server, not just on screen.
• Build deletion into the data model with triggers and CASCADE, and batch large deletions.
• Keep a free, equally accessible non-biometric alternative.

At AMG Software Studio we build web products like this, with face matching, consent and retention designed in from the start.