Second Impressions: Face Recognition When the Mask Comes Off

As the country cautiously reopens, we are seeing more and more unmasked faces. In some cases, the face we see when the mask comes off is not what we expected. In fact, the phenomenon of seeing someone's unmasked face for the first time (after having learned their face with a mask on) presents a novel opportunity for face perception researchers.

Recently, watching the local news, a reporter appeared on-air for the first time without her face mask. Upon seeing her full face, something seemed off. Her nose seemed too long for her face, her lips were a little more crooked than I expected, and the entire shape of the face didn't look quite right. It was as if the reporter's actual face did not match my assumptions of what her face would look like. But what was the nature of the face I had conjured? What were my assumptions about the face beneath the mask?

Since I wrote about Face Recognition Behind a Mask in April 2020, dozens of research articles have been published about how masks affect our ability to recognize faces. Some articles focus on the difficulty of recognizing identity when the lower half of the face is hidden. Other articles have tackled the problem of emotion recognition. For example, a recent paper by Kennon Sheldon and colleagues examined how people perceive two different types of smiles, with and without masks. Participants observed faces posing different expressions, including Duchenne smiles (which involve both the eyes and the mouth) and non-Duchenne smiles (or "social smiles", defined mostly by the mouth). As expected, face masks substantially interfered with participants' ability to identify non-Duchenne smiles defined mostly by the mouth region. In fact, participants most often assumed these faces were not smiling at all, but instead expressed a neutral or even a slightly negative expression.

Although this study did not directly compare how the same face would be perceived with a mask and after removing the mask, it does suggest that we make assumptions about the hidden part of the face. In this case, participants assumed the mouth had a neutral (or negative) expression based on the appearance of the eyes. What else do we assume about the hidden part of the face?

What do we assume is behind the mask?

Source: Oladimeji Ajegbile / Pexels

One possibility is that when we encounter a masked face, our brain automatically constructs a model of the full face. Decades of research in face perception suggest we process faces holistically; that is, we automatically integrate all the parts of a face, rather than perceiving individual features on their own. In that case, when we see a face with the lower half covered, our face perception system may automatically fill in the missing part of the face based on what it already knows. Given the appearance of the upper part of the face, the brain may determine the "most likely" scenario of what the remaining features look like. With repeated exposures to the masked face, this mental model may become sharper and sharper, eventually providing a fully detailed face. The problem is that this imagined face may not match reality at all.

Source: Oladimeji Ajegbile / Pexels

Open questions for future research

The second impressions we experience when we see a mask-less face for the first time may provide an opportunity for researchers to study our mental models of faces. This leaves many open questions to address: How do expectations about faces vary across different cultures? Do children and adults make the same assumptions about masked faces? Do we have a bias to assume the hidden parts of a face are more symmetric, more prototypical, or more attractive than they actually are? And when the lower face is finally revealed, do we modify our internal model, or replace it entirely? Future research may address the question of whether our internal model of a masked face continues to play a role even after the full face is revealed.