Feb 8, 2019 7:00 AM

Twitter Still Can't Keep Up With Its Flood of Junk Accounts, Study Finds

Iowa researchers built an AI engine they say can spot abusive apps on Twitter months before the service itself identifies them.

Casey Chin

Since the world learned of state-sponsored campaigns to spread disinformation on social media and sway the 2016 election, Twitter has scrambled to rein in the bots and trolls polluting its platform. But when it comes to the larger problem of automated accounts on Twitter designed to spread spam and scams, inflate follower counts, and game trending topics, a new study finds that the company still isn’t keeping up with the deluge of garbage and abuse.

In fact, the paper's two researchers write that with a machine-learning approach they developed themselves, they can identify abusive accounts in far greater volumes and faster than Twitter does—often flagging the accounts months before Twitter spotted and banned them.

Flooding the Zone

In a 16-month study of 1.5 billion tweets, Zubair Shafiq, a computer science professor at the University of Iowa, and his graduate student Shehroze Farooqi identified more than 167,000 apps using Twitter's API to automate bot accounts that spread tens of millions of tweets pushing spam, links to malware, and astroturfing campaigns. They write that more than 60 percent of the time, Twitter waited for those apps to send more than 100 tweets before identifying them as abusive; the researchers' own detection method had flagged the vast majority of the malicious apps after just a handful of tweets. For about 40 percent of the apps the pair checked, Twitter seemed to take more than a month longer than the study's method to spot an app's abusive tweeting. That lag time, they estimate, allows abusive apps to cumulatively churn out tens of millions of tweets per month before they're banned.

"We show that many of these abusive apps used for all sorts of nefarious activity remain undetected by Twitter's fraud-detection algorithms, sometimes for months, and they do a lot of damage before Twitter eventually figures them out and removes them," Shafiq says. The study will be presented at the Web Conference in San Francisco this May. "They’ve said they’re now taking this problem seriously and implementing a lot of countermeasures. The takeaway is that these countermeasures didn’t have a substantial impact on these applications that are responsible for millions and millions of abusive tweets."

The researchers say they've been sharing their results with Twitter for more than a year but that the company hasn't asked for further details of their method or data. When WIRED reached out to Twitter, the company expressed appreciation for the study's goals but objected to its findings, arguing that the Iowa researchers lacked the full picture of how it's fighting abusive accounts. "Research based solely on publicly available information about accounts and tweets on Twitter often cannot paint an accurate or complete picture of the steps we take to enforce our developer policies," a spokesperson wrote.

Twitter has, to its credit, at least taken an aggressive approach to stopping some of the most organized disinformation trolls exploiting its megaphone. In a report released last week, the social media firm said it had banned more than 4,000 politically motivated disinformation accounts originating in Russia, another 3,300 from Iran, and more than 750 from Venezuela. In a statement to WIRED, Twitter noted that it's also working to curb abusive apps, implementing new restrictions on how they're given access to Twitter's API. The company says it banned 162,000 abusive applications in the last six months of 2018 alone.

But the Iowa researchers say their findings show that abusive Twitter applications still run rampant. The data set used in the study runs only through the end of 2017, but at WIRED's request Shafiq and Farooqi ran their machine-learning model on tweets from the last two weeks of January 2019 and immediately found 325 apps they deemed abusive that Twitter had yet to ban, some with explicitly spammy names like EarnCash_ and La App de Escorts.

In their study, the researchers focused exclusively on finding toxic tweets produced by third-party apps, given the outsize effects of the automated tools. Sometimes the malicious apps controlled accounts that spammers or scammers themselves created. In other cases, they hijacked accounts of users who had been tricked into installing the applications or had done so in exchange for incentives like a boost in fake followers.

Tweet Dreck

Amid the 1.5 billion tweets the researchers started with—Twitter makes only 1 percent of all tweets available through a research-focused API—457,000 third-party applications were represented. The pair then used that data to train their own machine-learning model for tracking abusive apps. They noted which accounts each application posted to, along with factors including the age of the accounts, the timing of tweets, the number of usernames, hashtags, links the tweets included, and the ratio of retweets to original tweets. Most importantly, they observed which accounts were eventually banned by Twitter during the 16-month period they watched, essentially using those bans to denote abusive accounts.

With the resulting machine-learning-trained model, they found they could identify 93 percent of the applications that Twitter would ultimately ban without looking at more than their first seven tweets. "We're in some sense relying on seeing what Twitter eventually labels as malicious apps. But we found a way to detect them even better than Twitter," Shafiq says.

Twitter countered in its statement that the Iowa researchers' machine-learning model was faulty, because they couldn't actually say with certainty which applications Twitter had banned for abusive behavior. Since Twitter doesn't make that data public, the researchers could only guess by looking at which applications had tweets removed. That could have been from a ban, but it could also have resulted from users or applications deleting their own tweets.

"We think the methods used for this research do not accurately measure or reflect the health of our developer platform—principally because the factors used to train the model in this research are not strongly correlated with whether or not an application in fact violates our policies," a spokesperson wrote to WIRED.

But the Iowa researchers note in their paper that they only marked an application as having been banned by Twitter if 90 percent or more of its tweets had been removed. They observed that for popular, benign apps like Twitter for iPhone or Android, less than 30 percent of tweets are removed. If users of some legitimate app do delete their tweets more often, "these would be a small minority, these apps would not be used by a lot of people, and I don’t expect their results would be affected by that," says Gianluca Stringhini, a researcher at Boston University who has worked on previous studies of abusive social media apps. "So I would expect that their ground truth is reasonably strong."

Beyond those educated guesses at which apps had been banned, the researchers also honed their definition of abusive apps by crawling sites that advertised fake followers and downloading 14,000 applications they offered. Of those, about 6,300 had produced tweets in their 1.5 billion-tweet sample, so they also served as examples of abusive apps for the machine-learning model's training data.

One drawback to the Iowa researchers' method was its rate of false positives: They admit that about 6 percent of the apps their detection method flags as malicious are in fact benign. But they argue that the false-positive rate is low enough that Twitter could assign human staffers to review their algorithm's results and catch mistakes. "I don't think it would take more than one person to do this kind of review," says Shafiq. "If you don't aggressively target these applications, they’re going to compromise many more accounts and tweets, and cost many more man-hours."

The researchers agree with Twitter that the company is moving in the right direction, tightening the screws on junk accounts and more importantly, in his view, abusive applications. They noticed that around June 2017, the company did seem to be more aggressively banning bad apps. But they say their findings show that Twitter is still not exploiting machine learning's potential to catch app abuse as quickly as it could. "They’re probably doing some of this right now," Shafiq says. "But clearly not enough."