An algorithm trained on emoji knows when you're being sarcastic on twitter
Source: Will Knight
Scroll through Twitter and you’ll find plenty of sarcastic comments—not to mention lots of cases where sarcasm apparently went straight over someone’s head.
Luckily, an algorithm MIT researchers developed to analyze tweets can now detect sarcasm, and emotional subtext in general, better than most people.
Detecting the sentiment of social-media posts is already useful for tracking attitudes toward brands and products, and for identifying signals that might indicate trends in the financial markets.
But more accurately discerning the meaning of tweets and comments could help computers automatically spot and quash abuse and hate speech online. A deeper understanding of Twitter should also help academics understand how information and influence flows through the network. What’s more, as machines become smarter, the ability to sense emotion could become an important feature of human-to-machine communication.
The researchers originally aimed to develop a system capable of detecting racist posts on Twitter. But they soon realized that the meaning of many messages couldn’t be properly understood without some understanding of sarcasm.
The algorithm uses deep learning, a popular machine-learning technique that relies on training a very large simulated neural network to recognize subtle patterns using a large amount of data. The secret to training this algorithm was that many tweets already use something like a labeling system for emotional content: emoji. Once they took advantage of this to help the system read tweets for emotion in general, the researchers had a head start in teaching it to recognize sarcasm.
“Because we can’t use intonation in our voice or body language to contextualize what we are saying, emoji are the way we do it online,” says Iyad Rahwan, an associate professor the MIT Media lab who developed the algorithm with one of his students, Bjarke Felbo. “The neural network learned the connection between a certain kind of language and an emoji.”
To train the algorithm, dubbed DeepMoji, the researchers collected 55 billion tweets, and then selected 1.2 billion containing some combination of 64 popular emoji. First they trained the system to predict which emoji would be used with a particular message, depending on whether it was happy, sad, humorous, and so on. After that, the system was taught to identify sarcasm using an existing data set of labeled examples. The algorithm that had been pre-trained using emoji was far better at detecting sarcasm than one that hadn’t. They will release the algorithm for anyone to use.
To see how good DeepMoji is, the researchers tested it against several benchmarks for sensing sentiment and emotion in text. They found that it performed far better than the best existing algorithms in each case.
They also tested it against humans, using volunteers recruited through the crowdsourcing site Mechanical Turk. They found it was better than the humans at spotting sarcasm and other emotions on Twitter. It was 82 percent accurate at identifying sarcasm correctly, compared with an average score of 76 percent for the human volunteers.
Screen Shot 2017 08 04 at 3.34.31 PM The DeepMoji website can automatically generate emoji for a tweet. MIT Tech Review Screenshot
“It might be that it’s learning all the different slang,” Felbo says. “People have very interesting uses of language [on Twitter]—let’s put it that way.”
The researchers have built a DeepMoji website to demonstrate the emoji part of the system. It will automatically append suitable emoji to a piece of text. It seems to work pretty well, although when I tried inputting Donald Trump’s now-infamous “covfefe” tweet it was as confused as everyone else.
The site also lets users contribute to the research by annotating their own tweets with emotions. This is an important element of the work, Rahwan says. Usually researchers have volunteers tag other people’s tweets or posts with perceived emotion, which provides a less direct measure. “These benchmarks don’t capture what psychologists would consider true sentiment,” he says.
Gary King, director of the Institute for Quantitative Social Science at Harvard University and an expert on mining social networks for meaning, says using emoji as a means of training is a clever idea. But he questions how valuable it is to identify sarcasm if it’s lost on most people. “If the sarcasm is so nuanced that a human reader would likely miss it, then it really doesn’t matter,” he says.
Nonetheless, the work reflects the fact that computers are gradually getting better at sensing human emotion. Sentiment analysis of text is already a widely used technique. For example, it can help companies determine from the contents of an e-mail or chat message if a customer is particularly irate.
It may become ever more common for computers to try to decipher our emotions. Imagine, perhaps, a robot coworker that understands when its human colleagues are getting frustrated—or when they deliver a sarcastic compliment.
“If machines are going to cooperate with us, then they’re going to have to understand us, and emotion is really hard,” Rahwan says.
| }
|