Researchers generally agree that human emotions correspond to the execution of a number of computations by the central nervous system (CNS). Previous researchers (most famously Duchenne and Darwin) have assessed the hypothesis that some of these computations yield visible facial muscle actions. These facial muscle articulations are known as action units or AUs. Each AU defines a facial muscle action that can be visually interpreted by an observer. For example, contracting the middle section of the muscle in the forehead is observed as the raising of the inner corners of the eyebrows. This is called AU 1. The assumption is that such facial articulations are not random and are used to visually communicate emotions like happiness and surprise to others. That is, we can let our friends and foes know how we feel. Of course, one may be able to mask these facial muscle movements or fake them, allowing us to lie to others.

We have recently hypothesized, however, that the computations executed by the CNS include blood flow and/or blood composition changes that are visible as facial color variations, allowing people to communicate emotion to others even in the absence of any facial muscle movements. This communicative system is most likely involuntary and may allow others to know how we feel even when we do not want them to.

For example, when we are angry, testosterone secretion tends to increase, whereas cortisol decreases. Heart rate and blood pressure also increase. Because the surface of the face is innervated with a large number of blood vessels, these blood flow variations are visible to others. But how consistent are these changes across people, gender, ethnicity, and skin color? Note that for this visual signal to efficiently transmit emotion, these color changes have to be identical when we experience the same emotion and differential when we experience distinct emotions.

Using a machine learning approach, we identified the color patterns that define each emotion category. This analysis showed that, indeed, the experience of a particular emotion yields consistent facial color changes across people. The analysis also demonstrated that these visible facial color changes are different when we experience distinct emotions. This result allowed us to design a computer vision algorithm that can detect emotion from faces even in the absence of any facial muscle movement.

But, can people (not a computer vision algorithm) detect emotion from these color patterns too? To test whether these color patterns are indeed perceived as expressing emotion, we performed an experiment. In it, participants seat comfortably in front of a computer monitor. Participants first see a mask (i.e., a noisy image), followed by a fixation cross at the center of the screen, followed by an image pair (left and right of the cross). Participants have to choose which of these two images expresses a specified emotion. The response is given by keypress: left or right image. In a second experiment, we only provided an image and asked participants to assign an emotion label to it.

Can participants select the correct image in each pair and the correct emotive level in the single image? Yes. In fact, participants were very accurate in their assessments. These results suggest that the color features used by our computer vision system to detect emotion from color alone also facilitates the recognition of emotion by human observers.

Finally, we set out to determine if the color features that transmit emotion are at least partially independent from the emotive information transmitted by action units. That is, are the computations executed by the CNS that yields specific emotive AUs and facial colors the same or different computations.

To test this, we added the identified color features to images of faces expressing emotion with AUs. Here, we note that we can add congruent or incongruent colors to an image of a facial expression. Congruent means that the AUs and facial color transmit the same emotion. Incongruent means that AUs and facial color communicate distinct emotions. We found that people readily identify the image with congruent AUs and facial colors as expressing the emotion more clearly.

In conjunction, these results provide the first evidence for the existence of a mechanism for the visual transmission of an emotion signal through the modulation of facial color. Importantly, this signal is at least partially independent of that used to transmit emotion with action units. Hence, these results suggest it is possible to visually transmit and detect emotions even in the absence of any facial muscle movements.

Next read: Hips don’t lie: attraction revealed by dancing body language by Nick Neave

More from Psychology