Scams and creepers abound on Facebook Messenger. To combat them, Facebook is now turning to artificial intelligence. A new AI-powered safety feature for Messenger will now help people, particularly kids, identify bad actors and notify them about suspect accounts—all without reading the actual messages themselves.
The feature, which was rolled out on Android in March and will hit iOS next week, works by monitoring a potential scammer’s behavior. In an announcement blog, one example Facebook provided is a scenario in which an adult sends a lot of messages and friend requests to users under 18 years old. It might also notify a user if an account is impersonating a friend by using their photo when initiating a message. On top of behavioral signals, Facebook says it might also use reports and reported content to help train the AI. And, since the tool is basing its safety notifications on behavior, rather than message content, it would be compatible with end-to-end encryption, whenever it finally makes its way to Messenger. (Facebook has been talking about it forever, but earlier this year said it could take years before Messenger is encrypted by default.)
The safety notifications don’t automatically block fake or scam accounts—Facebook is leaving that up to the user. They’re more like inline popups that appear when someone might wander into a harmful interaction. The notifications also offer tips on how to recognize scams and a reminder to refuse requests to send money to strangers. The new tool will also remind anyone under 18 to be wary of chatting adults they don’t actually know.
The new AI-feature is likely meant as a sort of stop-gap for the Messenger Kids app. That app was meant to keep children safe by limiting who they could chat with to authorized users. But, surprising no one, a bug let minors talk to strangers via group chats—or the exact thing the app was designed to prevent.
Moderation and privacy have never been Facebook’s strong suits. That said, the company is facing increased scrutiny as scammers try to use its platform to capitalize on the global pandemic. The Better Business Bureau recently issued a warning that scammers are impersonating friends and acquaintances on Messenger to hawk bogus covid-19 cures and charities. It’s a problem Facebook is aware of and it has since launched a campaign against covid-19 misinformation across its platforms. While Facebook isn’t directly linking this new safety tool to the rise in covid-19 scams, the timing sure is helpful from a PR perspective.