How Does Face Recognition Work? From a Photo to a Name

Hold your phone up and it opens. Hold up a photograph of yourself and it stays shut. The phone tells your face apart from a picture of your face in under a second, without sending the scan to a server. That last part is the surprise. The check happens on the device, in a chip built for the job.

Four steps sit behind that yes or no. A camera finds the face, the system straightens it, a network turns it into a list of numbers, and a threshold turns those numbers into a decision. Judge any face system by that threshold and by the size of the gallery it searches. Those two settings decide almost everything about how often it gets the answer wrong.

Four Steps From a Photo to a Name

Every face system runs the same four steps, in the same order. Find the face. Straighten it. Turn it into numbers. Compare those numbers with the ones already stored. Products differ in the details, but the order stays the same. The same question sits behind every login check. A CAPTCHA answers that question a different way, and how a CAPTCHA tells people from bots is the bot-side version: a challenge, a response, and a server that decides.

1. Detection: a detector scans the frame and returns a box around each face, plus the landmarks inside it, the corners of the eyes, the tip of the nose, the edges of the mouth. Modern detectors such as RetinaFace and MTCNN find faces at several sizes and angles in one pass.
2. Alignment: the system rotates, scales, and crops the face so the eyes sit level and the face fills a standard frame, usually 112 by 112 or 160 by 160 pixels. The network that follows expects the same layout every time.
3. Embedding: a deep neural network reads the aligned face and outputs a list of numbers, 128 or 512 of them, called an embedding. That list is the faceprint the system stores and compares.
4. Matching: the system measures the distance between the new embedding and each stored template, then checks the closest one against a threshold. Above the line, the faces match. Below it, they do not.

You can watch detection work without any special tools. Open your phone camera and point it at a group of people. The small boxes it draws around each face are the detector running in public, on every frame, before any recognition starts.

Does the system keep a photo of your face? No. It keeps the list of numbers from step three. A list of 128 numbers takes 512 bytes. A phone photo takes thousands of times more space, and the numbers are what the system compares, so the photo has no job left once the numbers exist.

How a Face Becomes a String of Numbers

An embedding is a point on a map. The map has 128 or 512 dimensions, so no one can picture it, but the rule is simple: photos of the same person land near each other, and photos of different people land far apart. The network learns that map by studying millions of faces and adjusting itself until the rule holds. That learning from examples is the core of what artificial intelligence is.

The idea took its modern shape in three papers. Facebook’s DeepFace, published in 2014, trained a nine-layer network on four million face images. It reached 97.35 percent on the Labeled Faces in the Wild test set. The human score on the same set is 97.53 percent. DeepFace is a neural network, and the training follows how a neural network works: layers of simple units adjust their weights until the output matches the labels. Google’s FaceNet, published in 2015, trained on triplets. Each triplet holds an anchor photo, a second photo of the same person, and a photo of someone else. Training pushes the anchor closer to its match than to the stranger. It reached 99.63 percent with a 128-number embedding. ArcFace, published in 2019, added an angular margin. The margin pushes different people further apart on the sphere where the embeddings live. It reached 99.83 percent with 512 numbers.

No one programs the features by hand. Early systems measured distances between landmarks, the gap between the eyes, the width of the jaw. Modern networks learn their own features from the training set. The training sets are large. The MS-Celeb-1M set holds about 10 million images of 100,000 people. VGGFace2 holds 3.3 million images of 9,131 people. A network trained on that much variety stops caring about one bad photo. It starts caring about the shape that stays the same across all of them.

Why does the same face pass on one day and fail on the next? Light, angle, and expression move the point. A face in shadow or turned away lands further from its enrolled point than the same face in good light. Near the threshold, a small move flips the answer. The threshold decides.

The Threshold Is the Whole Decision

Matching is arithmetic. Two lists of numbers go in, one score comes out, and a line decides the answer. The score most systems use is cosine similarity, the cosine of the angle between two embeddings. A score near 1 means the vectors point almost the same way. A score near 0 means they point in unrelated directions.

Four numbers are enough to show the arithmetic. Take an enrolled face with the chosen vector 0.90, 0.30, 0.20, 0.25. A fresh photo of the same person gives 0.85, 0.35, 0.25, 0.30. Multiply the matching entries and add them. 0.90 times 0.85, plus 0.30 times 0.35, plus 0.20 times 0.25, plus 0.25 times 0.30, comes to 0.995. Divide by the two vector lengths, both near 1, and the cosine similarity is 0.9950. The angle between the two vectors is 5.73 degrees.

A different person gives 0.10, 0.90, minus 0.30, 0.20. The same arithmetic gives 0.3586, an angle of 68.98 degrees. A chosen near-twin vector, 0.88, 0.32, 0.22, 0.24, gives 0.9994. Set the threshold at 0.5 and the same person passes, the stranger fails, and the twin passes too. The threshold cannot separate two vectors that sit almost on top of each other. Real systems use 128 or 512 numbers, and the principle is the same at that size.

Every threshold is a trade. Raise the line and fewer strangers pass, but more rightful owners get turned away. Lower the line and the owners get in easily, but so do more strangers. A vendor that advertises 99 percent accuracy without naming the threshold has told you nothing. Ask for the false match rate beside the accuracy, and ask at what gallery size both were measured.

Verification compares one face with one stored template. Identification searches a whole gallery, and every extra template is another chance for a false match. At a false match rate of 1 in 100,000, the expected number of false matches grows with the gallery. The FBI photo database holds over 640 million photos, as Avast reports citing Forbes.

Gallery size	Expected false matches per search
1,000,000 faces	10
1,600,000 faces (Idemia FRVT test set)	16
640,000,000 photos (FBI database)	6,400

That is why a 1:N search returns a ranked list of candidates for a person to review. The list narrows the search, and a person still makes the call.

What Breaks It: Tilt, Light, and a Flat Photo

Detection looks solved until the photo gets hard. I ran two real detectors on two generated portraits on 15 November 2026, tilting, dimming, and shrinking each portrait to see where detection fails. The portraits are synthetic faces made for this test, so no real person is involved. Each detector ran once on each variant.

OpenCV’s Haar cascade, a 2001 design, looks for light-dark patterns in the image. The cascade found the face in 17 of 22 variants. It missed both 30-degree tilts of the first portrait, both 45-degree tilts, and the 45-degree tilt of the second portrait in one direction. Two variants gained a second box elsewhere in the frame. The cascade gave no warning about the false alarm, because it returns no score at all.

YuNet, a neural network from 2021, found the face in 22 of 22 variants at a score threshold of 0.5. Its best score on the full-size portraits was 0.59 to 0.71, below its usual 0.9 cutoff, and it rose to 0.92 to 0.96 after shrinking the image. On most full-size variants it returned one or two extra boxes, with scores from 0.52 to 0.68, pointing at empty parts of the frame. A low threshold buys recall and pays for it in false alarms, which is the same trade the matching threshold makes.

Detection is all the test covers. It says nothing about recognition accuracy, because no embeddings were computed, and nothing about real photos, crowds, or video. Two portraits and one run each is a small check. Treat the numbers as one run’s numbers. They are not a benchmark. The lesson it does support is why alignment exists. A detector’s job gets harder as the face tilts, so the pipeline straightens the face before the network ever sees it. A network that always receives an upright face never has to learn every tilt.

Recognition breaks in the same places detection does, and in a few more. Profile views, heavy occlusion from masks or sunglasses, strong backlight, and motion blur all move the embedding away from its enrolled point. Three checks on your own phone tell you which kind of system you carry. Tilt your head 30 degrees and try to open it. Cover your eyes with sunglasses and try again. Hold a printed photo of yourself in front of the camera. A depth-based system fails the photograph every time, and the tilt and glasses tests show how much angle and cover it tolerates.

Can a photograph open your phone? It depends on what the phone measures. A system that reads only a flat image can be fooled by a good print. A system that measures depth cannot, because a photograph has no depth to measure.

Why a Photo Does Not Fool a Good System

A good system checks for life as well as for a face. Depth sensing asks whether the face has shape. Infrared imaging asks whether it gives off the warmth and texture of skin. Texture analysis looks for the difference between skin and paper or a screen. Challenge-response asks the person to blink or turn their head, and watches for the movement. One check can be beaten. Four checks together need a replica, and a print is not one.

Apple’s Face ID runs all four checks in one glance. The TrueDepth camera lights the face with infrared, projects more than 30,000 invisible dots onto it, and reads where each dot lands. The dot pattern gives the depth map, and the infrared image gives the texture. A portion of the Neural Engine, protected inside the Secure Enclave, turns both into a mathematical representation and compares it with the enrolled data. The data never leaves the phone, and it is never backed up to iCloud. Older iPhones carry a second biometric instead, and how a fingerprint sensor reads a finger shows the contrast: a print is a flat pattern pressed against glass, while a face is a shape measured in depth.

Face ID also checks for attention: the eyes must be open and looking at the phone. Five failed matches ask for the passcode instead. Apple puts the chance of a random stranger opening a phone at about 1 in 1,000,000. The older Touch ID sensor was rated at 1 in 50,000. The figure does not hold for identical twins, for siblings who look alike, or for children under 13, whose features have not settled yet.

Phones without depth hardware read a flat image, and a flat image is all a photograph is. Those systems trade security for cost. Their setup screens say so in the fine print: a photo or a lookalike may open the phone. If your phone opens for a photograph, you carry a 2D system, and a passcode is the safer lock for anything that matters.

How Accurate It Is, and What to Check Before Trusting It

The best algorithms are measured by NIST, the National Institute of Standards and Technology, in its Face Recognition Vendor Test. On clean visa-style photos, the best algorithms miss under 0.1 percent of true pairs at a false match rate of 1 in 100,000. On mugshots the miss rate is 0.3 to 0.4 percent at the same rate. On wild photos, taken at odd angles in bad light, it rises to about 3 percent. The numbers describe the algorithm on good photos. They say nothing about a camera in a dark street.

Those tests show the errors are not spread evenly. NIST’s demographic report, published in December 2019, tested 189 algorithms on 18.27 million images of 8.49 million people. False match rates varied by factors of 10 to more than 100 across demographic groups. They ran higher for Asian and African American faces, for women, and for the very young and the very old. The most accurate algorithms showed the smallest gaps. A 2018 study by Buolamwini and Gebru found the same pattern in commercial systems. Error rates reached 34.7 percent on dark-skinned women, against 0.8 percent on light-skinned men.

Where the template lives decides who can take it. A phone keeps it in a protected chip, and it never leaves the device. Server systems store it encrypted instead. A breach of that server hands the attacker a template that cannot be changed the way a password can. Passwords can be replaced after a breach, and a password manager holds the replacements. Illinois law requires consent before a company collects a faceprint. The European Union treats biometric data as a special category, and its AI Act puts real-time identification in public spaces under strict rules.

Before you trust a face match, check three things. The threshold it was set at, and the false match rate beside it. The size of the gallery, and who reviews the matches that come back. Where the template lives, and who can read it. If no one can answer, treat the match as a starting point. A verdict needs more than a score. A face check is one factor. Where the stakes are high, add a second one, and remember what a second factor proves at login.

Frequently asked questions