The problem with a perfect score

A detector for AI-generated images that scores 99% on its test set is either a very good model or a model that has learned something much easier than the problem.

The thing it has probably learned is the fingerprint of the generators in its training set. Every text-to-image model leaves traces: a particular noise pattern, a particular colour distribution, a particular way of resolving edges. A classifier trained on a handful of generators can learn those traces rather than learning "this is generated".

And it will not know it has done so. The test set comes from the same pool as the training set, the score is high, and the model looks finished.

How the failure shows up in production

The symptom is not a bad score. It is a score that falls off a cliff the moment the world changes.

A model trained on a set of generators does very well on those generators and very badly on:

• A generator it has never seen. New text-to-image models are released constantly. A fingerprint learned from last year's tools is stale today.
• A real image from a different source. Cameras differ in their noise, in their colour profile, in how they handle low light. A model that learned "real" as a narrow cluster of camera artefacts will call a sharp phone photo fake.
• A manipulated real image. The interesting case, and the one that matters most for the actual use case. A photo with a manipulated region is, in a meaningful sense, not a photograph. But the pixel-level fingerprints the model learned may not fire at all.

The last one is the uncomfortable truth about this class of model: the tool is usually sold as "detects AI images", and the hard case is "detects manipulation", which is a different problem with different methods.

Building an evaluation that can catch it

The fix is not a better architecture. It is an evaluation that cannot be passed by a fingerprint.

Split by generator, not by image. The classic mistake is a random train/test split. If the training set contains 500 images from generator A and the test set contains 50 more from generator A, the test set measures nothing. The split has to be by source - every image from a generator is in one side or the other, never both.

Hold out whole generators. Take generators the model has never trained on and call them unseen. The score on unseen generators is the number that matters, and it is always lower than the headline number. If it is not dramatically lower, the model probably has not learned much.

Include a real-image set from a different distribution. Photographs taken on a different device, in a different country, at night, in rain. If the model is confidently wrong on ordinary photographs, the fingerprint is doing all the work.

Test the manipulated-real case separately. Real photo, real area edited. This measures something the training data may not cover at all, and it is usually the weakest number in the report.

Report the confusion, not just the accuracy. A model that predicts "fake" for 3% of real images is not a model that works; it is a model that will eventually flag someone's grandmother's photograph. A real deployment has to live with that rate, and the report should show it.

The part that is hard to do honestly

Negative results are hard to publish. "Our model does not generalise to generators released after the training cutoff" is an honest and useful finding, and it is also a bad headline. The pressure is to report the in-distribution number instead, because that is the one that looks good.

The evaluation set becomes the training set. Every time a model is tuned against a test set, that set is partly a training set. Keeping an evaluation honest means holding it out - not just from training, but from the people deciding what to try next.

Real images are hard to collect. The obvious source - the internet - is full of generated images now, so a naively-scraped "real" set is contaminated. Images taken on a known device, with known provenance, are worth much more than a large and dirty scrape.

What this changes about the product

If the model only reliably detects the generators it was trained on, the honest product is not "detects AI images". It is something narrower and more useful:

"Tells you whether this image looks like it came from one of the tools I trained on, and gives you a low-confidence result when it does not recognise the source."

That framing changes the interface. Instead of a confident red badge, the output is a probability with the closest known reference - this resembles output from tool X - and the interface refuses to make a claim it cannot support. A tool that says "I am not sure, and here is why" is more trustworthy than one that is confidently wrong on a third of real photographs, and the difference matters most to the people who would be harmed by a false accusation.

QuishGuard is built on that basis: the evaluation includes unseen generators and a real-image set from a different distribution, and the evaluation set is the thing that decides whether a model is good enough to ship.

The rules I would keep

• Split your data by source, never by row. Random splits measure memorisation.
• Your headline number is your unseen number. If they differ, the honest one is the lower one.
• A "no" that is wrong has consequences. Design the product so that uncertainty is expressible.
• Keep a held-out set that the tuning loop never touches. Everything else is a training set with extra steps.