Social Media Companies Scan For Potential Terrorists

Following the politically motivated shooting in Pittsburgh and the mailing of pipe bombs to political officials and journalists across the country, public outcry has risen against social-media companies. 

Suspected pipe-bomber Cesar Sayoc and Pittsburgh shooting suspect Robert Bowers used various platforms to post content indicating their potential for ideological violence. 

Some have asked why social-media companies didn’t do more, sooner, to stop the threat.

It’s a question that Facebook, Twitter, YouTube, and others have faced before, going back to 2014 when the problem was content from extremists of a different sort: violent jihadist groups such as ISIS.

Since then, many social-media giants have developed technological and policy-based ways to help prevent extremist content from proliferating across their sites and even to help law enforcement better track potential violent actors. But those efforts were aimed at foreign Islamic extremists, not domestic threats.

Monika Bickert, Facebook’s head of product policy and counterterrorism, has helped her company move farther along in this regard than some others. 

What role did Facebook play in the events that unfolded last week? A limited one: Robert Bowers, the charged Pittsburgh shooter, kept his threats and violent posts to a relatively obscure right-wing platform called Gab. Soon after he joined Gab in January, he began to post and spread images and content threatening to Jews.

Cesar Sayoc had a Facebook profile that he used to advance conspiracy theories. He also threatened people on Twitter, such as political analyst Rochelle Ritchie. Ritchie reported the threats to the platform. Last weekend, the company apologized for failing to act sooner.

Although Sayoc had a small presence on Facebook, the company might still have had a lot of information about him. Facebook monitors extremist rhetoric and content, of the Islamist variety, on sites that aren’t Facebook. It employs contractors to watch extremist chat rooms and other places so they can be ready to identify and tag threatening language, images, and content on Facebook.

Facebook’s Erin Marie Saltman, a policy manager at Facebook who oversees counterterrorism efforts in Europe, Africa, and the Middle East, disclosed this at the GLOBSEC security summit in May.

“There are a lot of people in other parts of the world that are not Facebook and not government,” Saltman said. “They are intel providers that sit and squat on a lot of these other sites and they tell us, in as close to real time as possible, when bad content is being released and so we know about it as soon as possible. 

So when the Abu Bakr al-Baghdadi speech was released a little while ago, and it wasn’t in video form, just audio, we were able to hash[tag] it before it started hitting our site.”

Predicting potentially violent behavior requires as much digitally collected data as possible, precisely the sort of data that intel vendors watching sites like Gab might notice. But when Defense One asked Facebook representatives whether they monitor sites like Gab for such content, or potential indicators of violence, they declined to say.

“As Erin mentioned, we work with intel and research firms who monitor many platforms, but we prefer not to disclose further details as bad actors actively work to circumvent our detection techniques,” a Facebook spokesperson said. 

“Since the bombing attempts, and the shooting in Pittsburgh, teams across our company have been monitoring developments in real time to understand both situations and how they relate to content on our site,” they added.

In 2011, J. Reid Meloy, a forensic psychologist and consultant to FBI’s Behavioral Analysis Units at Quantico, identified eight behaviors that can predict lone-wolf attacks based on ideological extremism. Sayoc and Bowers exhibited several of them across multiple social media sites. 

If social-media companies could search for these subtle indicators of a potentially dangerous person, behaviors, such as fixation or obsession, in the context of overtly troubling posts and comments such as direct threats, patterns could emerge to predict an individual’s behavior.

Cross-platform analysis of individuals’ data residue is what contemporary micro-targeting for advertisements is based on. It works to predict whether a person might be open to a specific product pitch but it also works to predict potentially harmful behavior. 

Facebook is already using AI to spot suicidal tendencies signaled by text patterns. The same algorithms could be applied to spot violent extremism, as could network analysis and even semantic text analysis. 

That information, coupled with the identification of violent messages or threats spread on other sites, could go a long way toward predicting and preventing violent behavior and the posting of extremist content. And it is, but it’s mostly violent Islamic behavior and content.

Consider the case of Demetrius Nathaniel Pitts, a Cleveland man recently charged with plotting a jihadist-inspired terrorist attack. Authorities monitored Pitts’s Facebook posts carefully after he commented on a photo of an al-Qaida training camp. 

His posts extorted Muslims to learn how to operate firearms, posts that law enforcement officials described as “disturbing” to USA Today. But pages urging non-Muslims (or people who are not explicitly Muslim) to own and practice with firearms are common on Facebook.

“We continually enforce our Community Standards through a combination of technology, reports from our community, and human review. This includes our hate speech policy that prohibits content that attacks people based on their race, ethnicity, national origin, religious affiliations, sexual orientation, caste, sex, gender, gender identity, and serious disease or disability,” said the spokesperson.

Following the public outcry against the proliferation of jihadist extremist messaging, Facebook and other sites tried a technique called hashing: essentially, to mark Islamic extremist content as individuals tried to spread it from one site to another. In 2016, Facebook executives led an effort to share data on hashed images across platforms.

“It creates the equivalent of a digital fingerprint so you can know when these things are coming up. We encourage that type of sharing, the hash sharing. Anybody using types of video, photo matching, would be able to use the hashes we are trying to share,” said Saltman.

Could hashed images and data from accounts like Bowers’s and Sayoc’s be relevant to law enforcement? Potentially, but the practice of hash sharing doesn’t involve the government, said Saltman. Instead, she said, the goal was to make a “safe tech space” for technology platforms to use whatever tools they saw fit.

“This is a by-industry, for-industry effort; it doesn’t include government or NGOs. It’s really so we can create a safe space so that some of these smaller platforms that are really scared about talking outside of industry, and admitting you have a problem is step one, can come together in a safe tech space and start operationalising around some of this.”

In a conversation with New York Times reporters on Sunday, Gab founder Andrew Torba denied that he or any Gab employer should monitor content on the site. 

“Twitter and other platforms police ‘hate speech’ as long as it isn’t against President Trump, white people, Christians, or minorities who have walked away from the Democratic Party,” he wrote. “This double standard does not exist on Gab.”

Defense One:

You Might Also Read:

A Genocide Incited On Facebook:

The Weaponization Of Social Media

« British Policing Faces The Future
Blockchain Can Improve Manufacturing »

Infosecurity Europe
CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

Infosecurity Europe, 3-5 June 2025, ExCel London

Infosecurity Europe, 3-5 June 2025, ExCel London

This year, Infosecurity Europe marks 30 years of bringing the global cybersecurity community together to further our joint mission of Building a Safer Cyber World.

CSI Consulting Services

CSI Consulting Services

Get Advice From The Experts: * Training * Penetration Testing * Data Governance * GDPR Compliance. Connecting you to the best in the business.

Resecurity

Resecurity

Resecurity is a cybersecurity company that delivers a unified platform for endpoint protection, risk management, and cyber threat intelligence.

The PC Support Group

The PC Support Group

A partnership with The PC Support Group delivers improved productivity, reduced costs and protects your business through exceptional IT, telecoms and cybersecurity services.

Authentic8

Authentic8

Authentic8 transforms how organizations secure and control the use of the web with Silo, its patented cloud browser.

mile2

mile2

Mile2 develop and deliver proprietary vendor neutral professional certifications for the cyber security industry.

Baker McKenzie

Baker McKenzie

Baker & McKenzie is an international law firm. Practice areas include Data & Technology.

Pole SCS (Secure Communicating Solutions)

Pole SCS (Secure Communicating Solutions)

SCS is a world-class competitiveness cluster dedicated to digital technologies in the fields of Microelectronics, Internet Of Things, Digital Security, Artificial Intelligence And Big Data.

Featurespace

Featurespace

Featurespace is a world-leader in Adaptive Behavioural Analytics and creator of the ARIC platform for fraud and risk management.

Corelight

Corelight

Corelight is the most powerful network visibility solution for information security professionals.

Absio

Absio

Absio provides the technology you need to build data security directly into your software by default, and the design and development services you need to make it happen.

International Cybersecurity Institute (ICSI)

International Cybersecurity Institute (ICSI)

ICSI is a UK company offering specialized and accredited professional qualifications in cybersecurity for young IT graduates as well as mature professionals.

689cloud

689cloud

689Cloud is a cloud content collaboration platform that allows users to protect, track, and control files AFTER they have been shared.

Cyber Skyline

Cyber Skyline

Cyber Skyline is a revolutionary cloud platform to practice, develop, and measure your team's technical cybersecurity skills.

Almond

Almond

Almond is positioned as a key independent French player in audit and consulting in the fields of Cybersecurity, Cloud and Infrastructure.

Cytek

Cytek

Cytek is a leading provider of cybersecurity and HIPAA compliance for dental practices and other industries.

Nokod Security

Nokod Security

Nokod Security delivers an application security platform for low-code / no-code custom applications and Robotic Process Automation (RPA).

Blattner Technologies

Blattner Technologies

Blattner Technologies mission is to be the leading provider of predictive transformation services and tools in the Data Analytics, Artificial Intelligence and Machine Learning industry.

Backslash Security

Backslash Security

With Backslash, AppSec teams gain visibility into critical risks in their apps based on reachability and exploitability.

RST Cloud

RST Cloud

RST Cloud is a cutting-edge technology company that specialises in threat intelligence solutions for businesses of all sizes.

Efex

Efex

Efex is one of Australia’s leading Managed Technology Solutions providers. We service local companies across Australia, providing accessible, fast and straightforward IT.