Face Recognition Behind a Mask

Face masks have recently become a symbol of the global effort to slow the spread of COVID-19. The Centers for Disease Control and Prevention (CDC) made a recommendation on April 3 that all adults should wear face coverings in public spaces, and many states are beginning to enforce this recommendation.

Although putting on a mask over your nose and mouth is one of the easiest ways to protect yourself and others from Coronavirus, face masks may be subtly (and not so subtly) interfering with how we perceive and recognize one another. Three aspects of face processing that rely on perceiving the lower half of the face—visually aided speech perception, interpretation of facial expressions of emotions, and individual gaze patterns toward faces—may be especially affected.

Visually aided speech perception

Individuals with and without hearing rely on visual cues to aid speech perception. One of the most well-known psychological illusions—the McGurk Effect, discovered by Harry McGurk and John McDonald in 1976—shows this in a compelling way. In the demonstration, an actor is silently mouthing the syllable /GA/, synchronized with a recording of the actor speaking the syllable /BA/. Seeing and hearing the actor at the same time leads to a perplexing impression that he is saying neither /BA/ nor /GA/, but a completely different syllable: /DA/. The McGurk Effect is so powerful that it continues to work even after observing it hundreds of times (which I have done over the course of several years of teaching courses on perception).

But the McGurk Effect is more than just an interesting illusion; it demonstrates that we unconsciously rely on the movements of the mouth and lips when interpreting speech. When a person speaks through a face mask, these mouth movements are completely inaccessible to the listener. This makes speech perception more difficult for hearing listeners, and almost impossible for deaf lip readers or others with hearing loss (see here). Luckily, there are new efforts to promote the production and use of face masks that reveal the mouth region using transparent material to aid the deaf community (for example, see here and here).

Interpretation of facial expressions of emotion

In the early 1970s, Paul Ekman conducted a series of experiments in different cultures around the world, arguing for the existence of six universally recognized facial expressions of emotion: surprise, fear, disgust, anger, happiness, and sadness. These findings, while still hotly debated in the field, have sparked countless follow-up studies examining which facial cues people rely on to discern these emotions. One common finding is that we often rely on the eyes to read a face’s emotional expression. But the mouth can also be highly informative, helping us distinguish, for instance, between fear and surprise, or between sadness and disgust (see research by Mark Schurgin and colleagues). When wearing a face mask, these emotions become difficult to distinguish.

Facial expressions behind a mask.

Source: Nicolas Davidenko

Individual differences in gaze behavior

Finally, research shows that we don’t all look at faces the same way. Some people tend to look more at the eyes while others tend to focus more on the mouth when recognizing faces. Individual differences in how we look at faces are stable, with people showing the same gaze patterns inside and outside the lab (Peterson et al., 2006).

Face masks that cover the mouth and nose are necessarily changing these gaze behaviors, making the lower half of the face practically uninformative for the purposes of recognition. Indeed, face masks may make it more compulsory to look at the eyes, something that many people express discomfort with. In particular, research in Autism Spectrum Disorder finds that individuals with ASD often avoid looking at the eyes, preferring to focus on the mouth or to look away from the face altogether, exasperating their difficulties with expression recognition. Masks' occlusion of the mouth region may therefore be systematically altering the way people look at faces.

Rewiring the brain?

As face masks become more ubiquitous in our daily lives, people may be learning to rely less on the mouth and more on the eyes across many aspects of face perception and recognition. Brain regions such as the fusiform face area (FFA) and superior temporal sulcus (STS) that are tuned to typical facial configurations through years of experience, may need to develop new strategies to process partially occluded faces. Will face masks have lasting effects on the brain’s face-recognition system?