Opinion mining - The key to mitigating political risk

Yesterday’s politicians will envy the online data that future candidates and office bearers will routinely use to curb their losses and guide their victories.

When Bill Clinton delivered his first State of the Union address to a joint session of Congress on 17 February 1993, Jack Dorsey was 16 and his peer, Mark Zuckerberg was only eight years old. Just days after the address, Stanley Greenberg, Bill Clinton’s polling advisor entered the White House to deliver the public’s response to the newly-elected President’s State of the Union address. In his book, Dispatches from the War Room, Greenberg recounts how he delivered the news to the President that two-thirds of the American public was in favour of the economic programme he had outlined in his address. Sitting in the Oval Office, Greenberg handed Clinton a pile of postcards from residents of Dayton, Ohio who were asked to write to the President immediately after watching his speech. It was an opportunity for ordinary citizens to express what they wanted from their President.

“I enjoyed the speech tonight. It was great to hear things most of us have felt for a long time. A great change is needed and you are the man to do it.”

“I hope that what was said in the President’s address is not a lot of hot air and tickling the American people’s ears. Since you talk the talk, I hope you walk the walk.”

“…Stop ‘the money’ from running the government. Get the crooks out of government.”

Greenberg was cognisant of the role he was playing in empowering ordinary citizens by delivering their messages directly to the Oval Office. Having the President read the Dayton residents’ postcards was a rare moment, where, outside of the confines of a structured poll or questionnaire, citizens were afforded the chance to voice their own opinions in response to a particular political event. Nobody was guiding them on what to say or how to say it.

The contents of the Dayton messages were candid and brief, due in part to a postcard’s dimensions. The Dayton postcards to President Clinton are likely 1993’s closest comparable to a tweet.

It is twenty-four years since Greenberg hand delivered those messages to the President. Today social media is a ubiquitous tool that has democratised political opinion. Twitter has an average of 328 million monthly active users, who are able to publicly voice and share their responses to an inauguration speech or healthcare policy, much like the 1993 postcards, nobody tells people what to tweet or how to write them.

Greenberg, a polling veteran, who worked with Tony Blair, Al Gore, John Kerry and Nelson Mandela, understands how valuable public sentiment is for guiding decision-making and ultimately mitigating political risk. Back in the Clinton White House, Greenberg could only have fantasised of the insights now available to political operatives through mining social media data for sentiment in order to gauge actual public opinion. This mining process commonly referred to as opinion mining or sentiment analysis, utilises machine learning to decipher public opinion from large volumes of online conversation.

Bo Pang, a research scientist at Google and Lillian Lee, a computer science professor at Cornell University co-authored the 2008 paper, Sentiment Analysis and Opinion Mining. According to the paper, interest in sentiment analysis and opinion mining took off around 2001 with the growth of social media, the rise of machine learning in information retrieval and the commercial opportunities that these tools presented, particularly to marketers.

Machine learning though regularly misses the nuances of human conversation. Sarcasm, humour and irony are still best observed and understood by people, not algorithms. To achieve a high level of accuracy, a sample of the data can be reviewed by human verifiers. This is achieved by referring a statistically significant sample of data being analysed to a group of human contributors, who evaluate the data for relevancy, sentiment and the underlying issues driving opinion.

Consider how valuable mining of social media conversation could be for a political candidate standing for office. The content of a candidate’s speech at a college campus could be guided by student sentiment toward them or their policies. By monitoring public sentiment, a candidate could acquire a real-time understanding of which counties they are losing ground in and schedule public appearances accordingly. Additionally, mining for sentiment data provides insights into political rivals.

Hillary Clinton’s presidential campaign team would have found sentiment data valuable. Imagine, for example, if they had seen real-time data showing how public sentiment had shifted, following FBI Director, James Comey’s announcement that he was re-opening an investigation 11 days before the election. Instead, Clinton’s campaign relied far too heavily on data models whose assumptions were not able to keep up with campaign developments. As Greenberg points out in his (September 21) American Prospect article, when you can’t keep up, “you get surprised by the voters”. More startling is Greenberg’s revelation that Hillary Clinton’s campaign did not conduct a single state poll in the final three weeks of the 2016 election campaign.

Greenberg’s post-mortem of the campaign points to a fatal failure to use real-time data. The story told by social media data during the 2016 US election, went unnoticed not just by Clinton’s campaign but by most major media outlets. On 4 November 2016, just four days before Americans went to the polls, DataEQ observed that Donald Trump was “crushing it online”. Sentiment analysis of social media data showed that the former Secretary of State was in trouble and warned of a Trump lead in key swing states.

After most polls failed to predict Trump’s victory, politicians will increasingly look to new technology that mitigates risk by providing accurate and real-time insights. As this technology improves, accurate opinion mining systems will become progressively powerful. Much like Greenberg’s postcards, future presidential hopefuls and incumbents, intent on keeping their finger on the pulse of public sentiment, will surely find systems analysing big data of social media indispensable.

This article was originally published on Business Day.