A 10-year-old CNN beats the frontier at skin cancer

Dr. Magnus Lynch, a consultant dermatologist at Guy's Hospital in London, ran a test that two frontier labs will not like. He showed the same 200 biopsy-proven skin lesions to GPT-6 Astra, Claude Fable 5.1, and a fine-tuned Inception v3, a convolutional network Google published in 2015. The 10-year-old CNN named the right diagnosis for 125 of 200 lesions (62 percent). The two frontier models managed 84 and 85 (42 percent each). The thread passed 550 points on Hacker News, and the full test kit, every prompt and every model reply, is downloadable from his writeup.

The setup deserves respect before the numbers. The 200 lesions come from MILK10k, a public dataset from the International Skin Imaging Collaboration where every lesion is photographed twice, once as a clinical close-up and once through a dermatoscope. Every lesion was biopsied, including the benign ones, so these are the hard cases: a mole somebody chose to cut out worried a clinician first. The CNN was fine-tuned on 68,011 images from the ISIC Archive with all 200 test lesions removed before training. Its architecture is deliberately not state of the art. The LLMs got both images per lesion, one run each, the same prompt, no tools and no retrieval.

The results, from the writeup:

• Correct diagnosis in one shot: 125 (CNN) versus 84 (GPT-6 Astra) and 85 (Claude Fable 5.1)
• Correct within two guesses: 173 versus 134 and 138
• Melanomas caught as first choice: 13 of 30, versus 8 and 11
• Moles recognized: 37 of 50, versus 16 and 33
• Seborrhoeic keratoses: 26 of 50, versus 13 and 6

The failure modes split cleanly, and that split is the real finding. Each LLM had a favorite answer. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, including 14 of the 30 melanomas. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. The CNN's melanoma errors went the other way: it called 8 of its 17 wrong melanomas a benign mole and 6 a seborrhoeic keratosis. One system over-produces a diagnosis it saw often in text. The other under-fires on a rare class, the way a trained classifier usually fails.

He also asked the models as a patient would: here is a photo, do I need a doctor? Both advised seeing one in nearly every case, 99 and 98 of 100 melanomas, harmless lesions included. That is the safe-but-useless failure, high sensitivity with the specificity of a coin flip.

The caveats in the writeup are honest and worth keeping next to the headline. Two hundred lesions cannot rank models that sit close together, and 30 melanomas make the most important number uncertain. Each LLM ran once, and answers vary between runs. On ordinary archive images, not the biopsied subset, the CNN separates "keep" from "discharge" far better, an AUC of 0.96 versus 0.86 here. Contamination cuts the other way: MILK10k has been public since 2025, so the LLMs may have seen it in training, and it still did not help them.

The discharge numbers need the most careful reading. At its preset threshold, the CNN discharged 76 of 100 safe lesions while keeping 77 of 100 unsafe ones; it also discharged 18 of 30 melanomas, which is disqualifying in a clinic. When the threshold was matched to the LLMs' conservative operating point, melanoma discharge fell to 2 or 1. Thresholds are a clinical decision, not a benchmark number.

Why did a 2015 architecture win? Because the comparison is rigged in a way nobody advertises. Inception v3 was fine-tuned on 68,011 labeled dermoscopy images for about twenty minutes on one H100. The LLMs, for all their scale, learned dermatology the way the rest of us read about it: incidental exposure inside a corpus built for language. A specialist model with the right input format beats a generalist that has to translate the world into tokens first. That is the whole result.

This lands the same week Moonshot published a 9.5-point capability gap it trained into its own K3.1 Pro on purpose, and IBM's CEO said on its Q3 earnings call that open-weight models are now good and cheap enough to run on-prem, as The Register reported. Both stories point at the same conclusion. The frontier's average keeps climbing, and specific jobs keep going to models that cost fractions of a percent to train and run. The CNN runs in a phone browser on WebGPU; Lynch's writeup links it, nothing gets uploaded, and the random sample button draws from the same 200 lesions. A clinical specialist that fits in your pocket was trained in twenty minutes.

For the teams building with these APIs, the takeaway is a checklist question, not a slogan: is your task really language-shaped, or is it perceptual with a label on it? If it is perceptual, a small supervised model trained on your own labeled data will embarrass the flagship, and the flagship's next version will not close that gap by itself. Run the cheap baseline first.