Blog / Engineering

A face detector once restored a toddler's pants

· updated 24 August with the detection-floor measurements · data: implementation, unit tests, lab bench

During early testing, the face detector reported a confident detection on a family snapshot and the pipeline dutifully “restored” it — except the detection was a toddler's corduroy pants. The fabric folds read as a face to the detector, and the face model then repainted them as one. The failure taught us that on old photographs, detector confidence alone is not a usable signal, and it produced the geometric gate every candidate face must now pass. This revision adds the other half of the story, measured: how small a face the detector can find at all. The floor sits between 14 and 20 letterboxed pixels — and the famous 1927 Solvay Conference photograph, whose 29 faces stand at 21–27 px, clears it with every single attendee found.

29 / 29Solvay 1927 faces foundat 21–27 px letterboxed height
14–20 pxthe detection floormisses below, finds above
27 / 29with naive 2×2 tilingboundary faces vanish — overlap matters
3geometry rules in the gateconfidence, eye band, mouth below eyes

Every number in this note was measured on the production pipeline code — same models, same math. Raw per-image records: e5_detect.json.

1 · Why old photos break face detectors

Detectors are trained mostly on modern, well-exposed images. Old scans present textures the training set rarely contains: fabric weave, lace, foliage, water damage, and grain — all capable of producing high-confidence false detections. They also present the opposite problem: real faces that are small, faded, and low-contrast. A confidence threshold alone cannot separate the two, because the false positives frequently score higher than the real small faces. Our detector is SCRFD-2.5G[1], decoded 1:1 from the FaceFusion reference implementation[2]: the photo is letterboxed into a 640×640 input, and each candidate comes back with a box, five landmarks, and a confidence score.

2 · The gate

Instead of trusting the score, we test whether the detection's landmarks are arranged like a face — cheap arithmetic on the eye and mouth points:

pass ⇔ score ≥ 0.55  ∧  0.25 < deyes/w < 0.75  ∧  ymouth > ȳeyes + 0.2·deyes(1)
detection box, width weye distance dmouth> 0.2 × d below eyesgate: 0.25 < d / w < 0.75 · confidence ≥ 0.55
Figure 1. The plausibility gate. Landmarks must be arranged like a face — eyes a face-like distance apart relative to the box, mouth meaningfully below the eye line — before any restoration runs.
RuleThresholdWhat it rejects
Detector confidence≥ 0.55Marginal detections not worth testing further.
Eye spacing vs. box width0.25 – 0.75Landmark clusters too tight or too wide to be two eyes in a face-sized box — the typical geometry of fabric-fold false positives.
Mouth below the eyesby ≥ 0.2 × eye distanceUpside-down and scrambled landmark arrangements.

The thresholds are deliberately loose — they describe any plausible human face, including tilted and three-quarter views, while excluding arrangements no face can have. Since the gate shipped, the pants class of failure has not recurred on our test photos.

3 · New: the detection floor, measured

00.250.50.7515204080120200face height in the 640 px letterbox (px)detection confidencegate ≥ 0.55confidence — face height in the 640 px letterbox (px) 5: 0confidence — face height in the 640 px letterbox (px) 7: 0confidence — face height in the 640 px letterbox (px) 10: 0confidence — face height in the 640 px letterbox (px) 14: 0confidence — face height in the 640 px letterbox (px) 20: 0.59confidence — face height in the 640 px letterbox (px) 28: 0.63confidence — face height in the 640 px letterbox (px) 40: 0.76confidence — face height in the 640 px letterbox (px) 56: 0.73confidence — face height in the 640 px letterbox (px) 80: 0.87confidence — face height in the 640 px letterbox (px) 120: 0.78confidence — face height in the 640 px letterbox (px) 200: 0.71confidence
Figure 2. Detection confidence vs face height in the 640×640 letterbox. Comfortable from ~200 px down to ~28, degrading below, dead between 20 and 14 — every face at or below 14 px was missed outright. The dashed line is the gate's 0.55 confidence floor.
The 1927 Solvay Conference group photograph with detection boxes drawn around all twenty-nine faces
Figure 3. All 29 attendees of the 1927 Solvay Conference detected at whole-frame scale (boxes drawn by the lab script, post-NMS, after the geometric gate). The letterboxed faces stand 20.9–27.1 px tall — hovering just above the cliff in Figure 2.

The Solvay result lands exactly where the ladder predicts, on the good side: all 29 of 29 attendees detected at whole-frame scale, confidence 0.734–0.864, every one surviving the geometric gate. And a control worth publishing because it is counter-intuitive: naively tiling the same photo 2×2 — more pixels per face — finds only 27 of 29. Two faces straddle tile boundaries and are seen whole by neither tile; a 3×3 grid happens to cut elsewhere and recovers all 29. That asymmetry is why the browser engine's small-face pass detects on the full frame and on overlapping quadrants, merging candidates by intersection-over-union — tiled detection needs overlap, not just resolution.

The practical reading for group photos: a face smaller than roughly 3% of the photo's long edge is invisible to the detector. Crop tighter before you restore — the same pixels come back through the letterbox bigger, on the safe side of the cliff — and see the face-rendering note for what happens to small faces after detection.

4 · Limitations

A geometric gate can only reject impossible arrangements. A false positive that happens to be face-shaped — a doll, a statue, a portrait on a wall — passes, and arguably should, since restoring it is harmless. The gate's job is narrow: never again repaint someone's clothing as a face. The floor measurements are one portrait and one group photo — enough to locate the cliff between 14 and 20 px, not to map its exact shape across poses and lighting; and confidence between 20 and 28 px is noticeably noisier than above, so faces in that band should be considered at-risk rather than safe.

References

  1. Guo, J., Deng, J., Lattas, A., & Zafeiriou, S. (2021). Sample and Computation Redistribution for Efficient Face Detection. arXiv:2105.04714. arxiv.org
  2. FaceFusion (open-source face processing platform) — the reference implementation our SCRFD decode and model preprocessing are ported from, 1:1. github.com
  3. Couprie, B. (1927). Photograph of the Fifth Solvay International Conference — 29 attendees, the densest concentration of physics Nobel laureates ever photographed. Public domain, via Wikimedia Commons. commons.wikimedia.org
  4. Lange, D. (1936). “Migrant Mother” (Destitute pea pickers in California), Farm Security Administration. Public domain, via Wikimedia Commons. commons.wikimedia.org
  5. ONNX Runtime — the inference engine both our server (CPU) and in-browser (WebGPU/WASM) engines run on. onnxruntime.ai