In the hopes of deterring violence, schools are turning to big data analytics to examine social media posts for the earliest signs of violence—depression, resentment, and isolation. Shawsheen Valley Technical High School in Massachusetts has turned to Social Sentinel, a data analytics company that says it can use the type of threat detection police agencies use to identify students at risk. But experts worry student social media mining, even with the best intentions, is a slippery slope to treating students the way we treat suspects.
Speaking to NPR affiliate WBUR, Social Sentinel’s founder and CEO, Gary Margolis, says the company worked with linguists and psychologists to train its AI on missives of students who went on to harm themselves or others. Margolis says they’ve created a library of 450,000 indicators that may trigger alarms.
“We went back, for example, and looked at the language that school shooters, as one example, have used in the past in various manifestos — what’s been published or that they’ve shared on social media,” Margolis said. “And we went to understand similarities and patterns. And we can teach computers, to an extent, how to identify some of that nuance.”
The students in the WBUR story reportedly had no idea the school was using Social Sentinel’s service, but often social media posts are open to at least some type of metadata analysis without people realizing. The complexity of this analysis means the service, ideally, flags accounts before any explicit threats are made and lets the school intervene with social services.
“We are not a surveillance tool; we are not a monitoring tool; we’re not an investigative tool,” Margulis said.
Shawsheen Valley is not the only school to try this approach. The Miami-Dade school system requested $30 million in state funds to overhaul its security system, including hiring staff to search social media. Schools in Wilson County, Tennessee also use a social media monitoring software that sends alerts when problematic keywords are detected, while another school in Fresno, California is revisiting the issue after initially deciding it was too invasive.
While we’re all looking for preventive, proactive approaches to the school shooting crisis, many are wary of increasing surveillance to deter gun violence. In New York and Florida, schools are turning to 24/7 surveillance, face recognition and biometric database matching, and license plate readers, all technology developed for police and counterterrorism agencies. Social Sentinel worked directly with police departments before opening up its services to schools.
Margulis admits there are false positives, where someone is flagged when they don’t pose a risk, but critically, there can also be false negatives—students deemed unremarkable by the AI who go on to do violence. Experts are worried that unleashing this technology in schools will only replicate the imbalances we see when these tools are used in public policing.
“This is an expansion of the schools’ ability to police what students are doing inside of school or on campus to their outside-of-school conduct,” says Kade Crockford, who directs the Technology for Liberty Program at the ACLU of Massachusetts. “In many cases across the country, schools have been using social media surveillance tools in ways that have harmed, specifically, students of color. So we certainly have concerns about technologies like this being used to expand what we call the school-to-prison pipeline.”
Social Sentinel declined to offer specific numbers on its success rate, but Margulis says educators have thanked them for their service in finding and reaching out to the students who need it most.