Blog / Engineering

Why AI-restored faces look like video game characters

· updated 24 August with the quantitative ablation · corpus: 11 Library of Congress scans, 35 faces (visual bench) + 4 public-domain portraits, 120 measured runs (metric bench)

Face-restoration models can make a face from a damaged photo look like a render from an old video game: plastic skin, painted eyes, a face that is sharp but not quite the person. We benchmarked eight rendering variants across 11 public-domain photographs to isolate the causes — there are three, they are separable, and none is fixed by a “better model.” This revision adds what the original bench lacked: numbers. Re-measuring the chosen rendering with a face-recognition embedding[4] confirms the by-eye selection quantitatively — each dial of the “natural” rendering buys measurable identity, cutting the restoration's identity tax in half — and a full sweep of the fidelity knob shows why it ships at 0.7 even though every curve is monotone toward 1.0.

3separable causes of the plastic looktotal replacement, colour drift, small faces
0.950identity of the degraded inputthe bar every restoration must be judged against
0.868 → 0.909identity, classic → natural renderthe ablation, measured at w = 0.7
−0.041identity tax of the shipped renderhalf the classic renderer's −0.082

Every number in this note was measured on the production pipeline code — same models, same math. Raw per-image records: e2v2_face.json · e2_face.json.

Soft, grainy 1862 black-and-white scan of three men standing at a tentBeforeThe same scan restored: sharpened faces and recovered detail with natural colourAfter
Figure 1. A restoration by this site's tool on one of the corpus photographs (Lincoln at Antietam, 1862 Library of Congress scan): faces rebuilt and the whole image enhanced, with the photograph's own grain and colour character preserved. August 2026.

1 · The three causes

1. Total replacement. The standard pipeline pastes the model's output over the original face at 100% opacity, so every restored face is entirely model-drawn. The photograph's grain, paper texture, and lens character end at an invisible oval boundary around each face.

2. Colour drift. The face model outputs its own colour, not the photo's. Measured on our bench, it invented blue eyes on an 1850s ambrotype and lightened the skin tone in a 19th-century studio portrait of a Black subject — a known failure mode of face models trained on modern, predominantly light-skinned photo corpora[2]. Any pipeline that keeps the model's chroma inherits its biases.

3. Small faces are hallucinations. A face that occupies under roughly 100 px reaches a 512 px face model as a 5–13× bilinear enlargement — a smear. The model then invents a plausible face rather than recovering the real one. This is a physics limit, not a model limit: a 40 px face can be made soft-but-human, never sharp-and-faithful.

2 · Anatomy of a face repair

To see where the dials live, follow one face through the pipeline. Detection (SCRFD, with the plausibility gate) yields five landmarks. Alignment solves the least-squares similarity transform — scale s, rotation θ, translation t — mapping the detected eye and mouth points pi onto the fixed FFHQ template[5] qi:

argmins,θ,t Σi ‖ sR(θ)pi + t − qi ‖²(1)

so every face enters the network at 512×512 with eyes and mouth in canonical position. The model — CodeFormer by default[1], which restores by looking faces up in a learned codebook rather than regressing pixels, and exposes a fidelity weight w ∈ [0, 1] steering between its prior (0) and your input (1) — produces a restored crop. Then comes the part this note is about: the paste-back. The classic recipe composites that crop at full opacity in the model's own colours. Ours warps it home under a radial feather m and composites each pixel as

out = blend · m · faceL⊕photoab + (1 − blend · m) · photo(2)

with blend = 0.75, and faceL⊕photoab meaning the model contributes only luminance (Lab L) while the photograph keeps its own chroma — the exact mirror of our colouriser, which contributes only chroma and keeps the photo's luminance. Small faces get a dedicated ×4 super-resolution pass on their source footprint before restoration, so the model sees recovered detail instead of a bilinear smear.

3 · The visual bench that picked variant H

Eight variants, each a full pipeline run over the corpus, varying the fidelity weight, the paste-back opacity, the colour source, and small-face SR. Outputs were compared side by side against the originals at 100% zoom:

VariantWeightPaste-backColour sourceSmall-face SR
A (then-current)0.70100%model—
B0.90100%model—
C0.7070%model—
D0.70100%photograph—
E0.7075%photograph—
F0.8575%photograph—
G0.7050%photograph—
H (selected)0.7075%photographyes

H shipped as the default rendering. The luminance-only paste-back adds under half a second per photo; the small-face pass adds one to two seconds per qualifying face only. The original limitation section admitted the obvious weakness: selection was by side-by-side human judgment on 35 faces, not a metric. So we built the metric bench.

4 · New: the ablation, quantified

One number frames everything: the degraded input itself scores 0.950 identity against the original. ArcFace was trained under heavy augmentation, so it sees through blur and noise; restoration therefore gets no credit for being sharp — it is only debited when it changes who is in the photo. Every renderer pays some of that identity tax. The question is how much:

Rendering (all at w = 0.7)PSNR (dB)ArcFace ididentity taxTenengrad ratio
Degraded input (baseline)27.320.950—0.217
Classic v2 (100% paste, model colour)26.680.868−0.0820.373
+ luma-only (photo keeps its chroma)26.720.882−0.0680.376
+ 75% blend (grain shows through)27.060.901−0.0490.304
Natural (shipped: both + small-face SR)27.090.909−0.0410.307

The metric bench agrees with the eyes, dial by dial. Keeping the photograph's chroma buys +0.014 identity (colour drift was real, and measurable). The 75% blend buys another +0.019 and most of the PSNR — letting a quarter of the true face show through is worth almost as much as the entire face model's fidelity setting. Together they halve the identity tax, from −0.082 to −0.041, while still roughly 1.4× the input's sharpness. The one honest cost: the classic render is “sharper” by Tenengrad (0.373 vs 0.307) — sharpness it partly spends on texture that isn't the photograph's. That is the perception–distortion tradeoff[6] relocated into rendering, and we spend it on identity.

Four panels of Florence Owens Thompson's face: sharp original, degraded copy, the classic render with plastic skin and shifted colour, and the natural render preserving grain and tone
Figure 2. The ablation in person — Migrant Mother (Lange 1936). Classic v2 (third): plastic skin and a colour cast the photograph never had. Natural (fourth): the model's detail under the photograph's own grain and tone. Identity: 0.868 vs 0.909 against the original.
Four panels of Lincoln's portrait: original, degraded, the classic render with a cold colour shift, and the natural render keeping the warm plate tone
Figure 3. Same ablation, Lincoln O-77 (1863). The classic render (third) drains the plate's warm tone and cools the whole face — cause 2, colour drift, in one look. The natural render keeps the photograph's colour by construction: the model is only allowed to contribute luminance.

5 · The fidelity knob, swept end to end

CodeFormer's weight w steers between the codebook's prior (0) and deference to your input (1)[1]. We swept it in eleven steps under the natural renderer:

0.60.70.80.9100.250.50.71CodeFormer fidelity weight wArcFace cosine vs originaldegraded inputproductionrestored — CodeFormer fidelity weight w 0: 0.71restored — CodeFormer fidelity weight w 0.1: 0.76restored — CodeFormer fidelity weight w 0.2: 0.8restored — CodeFormer fidelity weight w 0.3: 0.83restored — CodeFormer fidelity weight w 0.4: 0.86restored — CodeFormer fidelity weight w 0.5: 0.88restored — CodeFormer fidelity weight w 0.6: 0.9restored — CodeFormer fidelity weight w 0.7: 0.91restored — CodeFormer fidelity weight w 0.8: 0.92restored — CodeFormer fidelity weight w 0.9: 0.93restored — CodeFormer fidelity weight w 1: 0.93restored
Figure 4. ArcFace identity vs fidelity weight under the shipped renderer, mean of four portraits, ×4 degradation. Monotone from 0.710 (a plausible stranger) to 0.932 — and never reaching the dashed input baseline at 0.950: the knob sets the identity tax rate, it cannot make the tax negative. PSNR (25.5 → 27.3 dB) and SSIM are monotone the same way.

We went in expecting a tradeoff curve with an interior optimum that would justify 0.7 as “the knee.” The data refused: on the numbers, w = 1.0 dominates — at every severity:

0.50.60.70.80.9100.30.50.71CodeFormer fidelity weight wArcFace cosine vs originalproduction×2 (mild) — CodeFormer fidelity weight w 0: 0.72×2 (mild) — CodeFormer fidelity weight w 0.3: 0.86×2 (mild) — CodeFormer fidelity weight w 0.5: 0.91×2 (mild) — CodeFormer fidelity weight w 0.7: 0.94×2 (mild) — CodeFormer fidelity weight w 1: 0.97×2 (mild)×4 (standard) — CodeFormer fidelity weight w 0: 0.71×4 (standard) — CodeFormer fidelity weight w 0.3: 0.83×4 (standard) — CodeFormer fidelity weight w 0.5: 0.88×4 (standard) — CodeFormer fidelity weight w 0.7: 0.91×4 (standard) — CodeFormer fidelity weight w 1: 0.93×4 (standard)×8 (severe) — CodeFormer fidelity weight w 0: 0.59×8 (severe) — CodeFormer fidelity weight w 0.3: 0.68×8 (severe) — CodeFormer fidelity weight w 0.5: 0.73×8 (severe) — CodeFormer fidelity weight w 0.7: 0.76×8 (severe) — CodeFormer fidelity weight w 1: 0.79×8 (severe)
Figure 5. The severity interaction: identity vs weight at ×2 / ×4 / ×8 degradation, natural renderer. Higher w wins everywhere, even at ×8 where the input baseline itself has fallen to 0.826. At the production setting the identity tax runs −0.042 (mild), −0.041 (standard), −0.068 (severe); at w = 0 it reaches −0.27 — when little information survives, the prior fills the vacuum with its own opinions.

So why ship 0.7 rather than 1.0? Because the metrics are necessary, not sufficient. ArcFace measures who; it is nearly blind to age and texture — the same training that makes it robust to blur makes it forgiving of smoothing. The bench's original observation stands on its own evidence: at the model's customary 0.5, crow's feet visibly soften on Migrant Mother — Florence Owens Thompson was 32 and carrying the Depression in her face, and the prior would rather she didn't. At w = 1.0 the codebook defers so far toward a ruined input that compression garbage starts surviving. 0.7 keeps nearly all of the metric value (0.909 vs 0.932) while the pictures keep their age. And the s8 panel below is the argument for never letting w near 0 on damaged heritage photos:

Three panels: severely degraded Lincoln, an almost unrecognizable smooth younger face at weight zero, and a closer likeness at weight one
Figure 6. ×8 degradation, Lincoln. From this input (left), w = 0 (middle) confidently reconstructs a smoother, younger, different man — identity 0.585, approaching coin-flip. w = 1 (right) stays recognisably Lincoln at 0.788. When the evidence is thin, the prior's opinions dominate — exactly when you want the knob high.

6 · The other models, same renderer

Face modelPSNR (dB)ArcFace idTenengrad ratioms / face*
CodeFormer (w = 0.7, default)27.090.9090.3077200
GFPGAN 1.427.130.9100.3286046
GPEN-BFR-25626.680.9270.3222633

*Median full-pass wall-clock (detection included) on the 2-vCPU bench box. Under the natural renderer the three models[1][2][3] converge to within 0.02 identity of each other — the renderer, not the model, is now the dominant term, which is itself a finding: the “video game” look was never really about which network you run. CodeFormer stays the default for its robustness on severe damage and for the knob this note just spent two sections on; GPEN remains the speed choice at 2.6 s per face.

7 · Limitations

  • The visual bench is 35 faces; the metric bench is 4 portraits × 120 runs. The monotone shapes held on every portrait individually — that is the claim we rely on; absolute values will drift with subject and scan.
  • ArcFace is one referee — the standard one, with the blur-tolerance caveat we exploit as a baseline. A human perceptual study over thousands of faces is the missing instrument, and the age-line observation rests on eyes, not embeddings.
  • Synthetic degradation. Real shoebox damage adds scratches and silvering the corruption stack doesn't model — the damage stage handles those separately.
  • The small-face limit stands: below roughly 40 px of face, no setting produces a faithful sharp face, and the honest output is a soft one.

References

  1. Zhou, S., Chan, K. C. K., Li, C., & Loy, C. C. (2022). Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 35. arXiv:2206.11253. arxiv.org
  2. Wang, X., Li, Y., Zhang, H., & Shan, Y. (2021). Towards Real-World Blind Face Restoration with Generative Facial Prior. CVPR. arXiv:2101.04061. arxiv.org
  3. Yang, T., Ren, P., Xie, X., & Zhang, L. (2021). GAN Prior Embedded Network for Blind Face Restoration in the Wild. CVPR. arXiv:2105.06070. arxiv.org
  4. Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019). ArcFace: Additive Angular Margin Loss for Deep Face Recognition. CVPR. arXiv:1801.07698. arxiv.org
  5. Karras, T., Laine, S., & Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks (the FFHQ alignment template). CVPR. arXiv:1812.04948. arxiv.org
  6. Blau, Y., & Michaeli, T. (2018). The Perception-Distortion Tradeoff. CVPR. arXiv:1711.06077. arxiv.org
  7. FaceFusion (open-source face processing platform) — the reference implementation our SCRFD decode and model preprocessing are ported from, 1:1. github.com
  8. Lange, D. (1936). “Migrant Mother” (Destitute pea pickers in California), Farm Security Administration. Public domain, via Wikimedia Commons. commons.wikimedia.org
  9. Gardner, A. (1863). Abraham Lincoln, portrait O-77. Public domain, via Wikimedia Commons. commons.wikimedia.org
  10. ONNX Runtime — the inference engine both our server (CPU) and in-browser (WebGPU/WASM) engines run on. onnxruntime.ai