May 21, 2020 9:00 AM

Facebook Messenger Adds Safety Alerts—Even in Encrypted Chats

By using metadata instead of content to spot suspicious behavior, the social network can keep privacy intact.

A security officer enters a chat bubble.

For the last year, governments around the world have pressured Facebook to abandon its plans for end-to-end encryption across its apps, arguing that the feature provides cover for criminals and, above all, child predators. Today Facebook is rolling out new abuse-detection and alert tools in Messenger that may help address that criticism—without weakening its protections.

Facebook today announced new features for Messenger that will alert you when messages appear to come from financial scammers or potential child abusers, displaying warnings in the Messenger app that provide tips and suggest you block the offenders. The feature, which Facebook started rolling out on Android in March and is now bringing to iOS, uses machine learning analysis of communications across Facebook Messenger's billion-plus users to identify shady behaviors. But crucially, Facebook says that the detection will occur only based on metadata—not analysis of the content of messages—so that it doesn't undermine the end-to-end encryption that Messenger offers in its Secret Conversations feature. Facebook has said it will eventually roll out that end-to-end encryption to all Messenger chats by default.

"We’re introducing safety notices in Messenger that will pop up in a chat and provide tips to help people spot suspicious activity and take action to block or ignore someone when something doesn’t seem right," reads a blog post from Facebook's director of product management for Messenger privacy and safety Jay Sullivan. "As we move to end-to-end encryption, we are investing in privacy-preserving tools like this to keep people safe without accessing message content."

Facebook hasn't revealed many details about how its machine-learning abuse detection tricks will work. But a Facebook spokesperson tells WIRED the detection mechanisms are based on metadata alone: who is talking to whom, when they send messages, with what frequency, and other attributes of the relevant accounts—essentially everything other than the content of communications, which Facebook's servers can't access when those messages are encrypted. "We can get pretty good signals that we can develop through machine learning models, which will obviously improve over time," a Facebook spokesperson told WIRED in a phone call. They declined to share more details in part because the company says it doesn't want to inadvertently help bad actors circumvent its safeguards.

The company's blog post offers the example of an adult sending messages or friend requests to a large number of minors as one case where its behavioral detection mechanisms can spot a likely abuser. In other cases, Facebook says, it will weigh a lack of connections between two people's social graphs—a sign that they don't know each other—or consider previous instances where users reported or blocked a someone as a clue that they're up to something shady.

One screenshot from Facebook, for instance, shows an alert that asks if a message recipient knows a potential scammer. If they say no, the alert suggests blocking the sender, and offers tips about never sending money to a stranger. In another example, the app detects that someone is using a name and profile photo to impersonate the recipient's friend. An alert then shows the impersonator's and real friend's profiles side-by-side, suggesting that the user block the fraudster.

Photograph: Facebook

Facebook does have one exception to its claim that it doesn't look at message contents: Messenger's abuse-reporting mechanism has long included a feature that sends messages to Facebook when a user flags them. That gives Facebook another potential clue as to what other bad behavior that sender might be up to, but isn't generally considered a breach of end-to-end encryption since the recipient of an encrypted message can always choose to share the decrypted version with a third party.¹

Photograph: Facebook

For now, the safety notices feature will only explicitly suggest blocking or ignoring a potential abuse. (Users can still report abusive behavior with the usual method of tapping on the sender's name and then "Something's Wrong," then specifying what happened.) "We wanted to give people an immediate action, and blocking is the most immediate action someone can take to avoid harm," a Facebook spokesperson says. "Reporting is something we’re looking at bringing to the feature as well."

Facebook Messenger's addition of abuse alerts based on metadata alone is a "good start," says Alex Stamos, the former chief security officer of Facebook. But he argues that the company could—and should—do more. Stamos, who now leads the Stanford Internet Observatory, has argued that Facebook, Google, Microsoft, Snap, and others should all monitor for signs of bad behavior on user devices.

"I think they should have client-side looking-at-content. And once they do that, they should prompt people to be able to report it," Stamos says. That on-device content analysis and reporting allow Facebook to find bad actors faster than mere metadata scanning, Stamos says, while still maintaining end-to-end encryption.

Stamos also notes that if Facebook emphasized reporting rather than mere blocking in its alerts to, those reports could create evidence that law enforcement could use against serious criminals. "You're not going to arrest somebody because your data shows that they tried to 'friend' a bunch of teenage girls," Stamos adds. "Whereas if somebody actually sends a request for nudes to a kid, that is probably illegal in most cases. That could be used to get a search warrant and to possibly prosecute the person. You really want the content of the communications, and the best way to do that in end-to-end encryption is just to encourage the recipient to report the conversation."

A Facebook spokesperson, asked about that possibility of client-side content analysis, says the measure "wasn’t considered and isn’t necessary for this safety feature."

Facebook, along with plenty of other tech companies, has come under growing pressure from the Trump administration and Congress to build mechanisms into encryption that allow law enforcement access to communications and stored data. In March, a bill called the EARN IT Act was introduced that could in practice ban strong end-to-end encryption if passed in its current form. And just this week, Attorney General William Barr criticized Apple for not helping to decrypt a pair of older iPhones that belonged to Al Qaeda–linked Mohammed Saeed Alshamrani, who killed three people in a mass shooting incident last December.

Facebook argues that its new safety notices aren't a response to that political pressure—that, in fact, they've been in the works since even before Mark Zuckerberg announced the company's slow transition to a unified, end-to-end encrypted messaging system last year.

But Stamos argues that exactly this sort of tool is necessary to appease encryption's critics—and that more may still be both necessary and possible without compromising encryption's fundamental privacy guarantees. "I think it's a good sign that they're experimenting here," says Stamos. "It's an acknowledgement that they believe they have a responsibility to deal with some of the downsides [of encryption], which I think is not just morally right. It is a predicate of end-to-end encryption surviving."

¹Updated 5/21/2020 at 2pm EST to clarify how Facebook’s reporting process for encrypted conversations works.