Facebook Wants To Generate Emojis By Staring At Your Face

Patent filings show the company is eyeing technology to record emotions by constantly monitoring keystrokes and facial expressions

Illustration: R. A. Di Ieso
Jun 13, 2017 at 10:54 AM ET

If you still aren’t covering your laptop’s webcam with a piece of duct tape, here’s a great incentive: Facebook wants to use it to track your emotions in real-time.

According to several recently published patent applications, the social media giant is looking to monitor users’ emotions in several new and invasive ways, including monitoring the rhythm of their keystrokes and peering through webcams to analyze facial expressions in real-time. According to the filings, the goal is to allow the company to incorporate their users’ emotional state into typed messages, automatically translate facial expressions to reaction emojis, and of course, use intimate data to target ads.

Some of the patents were originally filed as far back as 2014, but are being published now, as face and emotion recognition technology is become increasingly prevalent. One filing, published May 25, 2017 and titled “Augmenting Text Messages With Emotion Information,” describes a method for predicting a user’s emotions by analyzing user input interactions in Facebook Messenger — including the rhythm of keystrokes and interaction patterns from using a mouse or touchscreen  — and changing the visual formatting of the text to reflect their emotional state.

“By integrating emotion information within a text message, the message system can retain context associated with the message that might otherwise be lost in a conventional text message,” the patent’s description states. “Automatically formatting the text in a message to convey emotion information allows the sender to more accurately convey a specific meaning of the message without requiring that the sender include other content or explicitly state an emotion in the text.”

Another patent filing, published in May, depicts a system for using facial expressions observed through a webcam to automatically generate Facebook’s emoji reactions. The filing describes a method for analyzing portions of the user’s face, then matching the real-time image data with an appropriate emojis, like the current selection of “wow,” “angry,” “sad,” and “haha.” Of course, having a seamless, hands-free emoji experience like the kind the document describes would mean giving Facebook constant, unhindered access to your laptop or smartphone’s forward-facing camera — allowing the company to track and profile your emotion responses.

The patents are not surprising, given the widespread usage of machine learning for emotion recognition and Facebook’s well-known ambitions for exploiting users’ emotions and behavioral data. The company has previously shown its willingness to use emotional data to secretly manipulate users’ behavior through its infamous “Emotional Contagion” experiment, which influenced the emotional content of their posts by altering the appearance of positive or negative articles appearing in their news feeds.

As Vocativ previously reported, independent researchers have created relatively simple systems that identify users and predict their emotional state by analyzing keystroke patterns — and data-hungry tech conglomerates like Google and Facebook are presumably much farther along in implementing such systems for their billions of users. The emoji-generating patent also follows Facebook’s November purchase of FacioMetrics, a startup company that prides itself on gesture controls and systems that can “recognize facial expressions and perform related actions.”