Opinion Mining – The future of social intelligence?

The digital age is characterised increasingly by the collective. The information generated by tapping into the minds of many is driving decisions in both the public and private sector; research is becoming social.

On the back of this, a new science has emerged – known as opinion mining – which uses the latest advances in artificial intelligence (AI) to mine public opinion for sentiment. This structured data is known as opinion data. By analysing online public opinion, governments and global organisations can now access a set of insights that can shape strategy and better measure the public’s experience of their policies, service and brands.

Crucially, the field of opinion mining looks not only at sentiment, but the topics driving that sentiment. This has also opened up new research capabilities in the world of market and political research. The sheer quantity of online conversations (Facebook alone had 1.15 billion mobile daily active users on average for December 2016), coupled with the instantaneous reactivity of this digital chatter, has meant that sentiment-driven opinion data has become a mineable, monetisable resource.

Approaches to opinion mining

There are two main approaches to opinion mining. The first exclusively uses AI to structure the data. The second adds an extra layer of analysis by processing the data through a Crowd – a team of people that verify the data for sentiment and the topics driving the sentiment. The crowd’s role is to help correctly classify the unstructured data, as pure AI based approaches struggle with the nuances of human conversation. This problem is especially common in social media where conversations are typically filled with humour, slang, innuendo, sarcasm, colloquialisms and emoticons.

For example, a tweet from someone saying “I just spent 5 hours in the queue at bank X… best customer service ever!” is clearly sarcastic, but to a machine will likely be seen as positive.

By combining AI and human understanding you get the best of both worlds – the ability to gather and process huge data sets and still gain an accurate understanding of the publics’ feelings.

The changing of the guard

Historically, such data was used primarily to guide commercial decision-making – largely by providing companies with deeper insight into consumer experience.

A much wider ambit is possible, however. While part remains firmly within the commercial realm, an increasing proportion lies in the governmental/state/city arena. In the latter, social analytics are proving useful in both deepening the understanding of electorates and driving strategic political decisions.

New markets and prospects

Agility, reactivity and leanness – these are among the buzzwords for businesses operating within the current landscape of changing customer relationships, increasing competition and the threat of disruptive innovation. Moreover, with growth having stalled in much of the developed world, many multinationals are looking for data that can help them move into new untested markets.

With the global adoption of mobile phones and the concurrent explosion in the use of social media, companies now have access to millions of relevant data points in both mature and developing markets. The data, if mined correctly, can provide insights that not only assist in decision making, but can shape product innovation and help reduce risk when moving to new markets.

Research and consulting houses: Differentiating the value proposition

The tough economic climate combined with a whole new set of disruptive technologies has created both risk and opportunity for consultancies and research houses. Clients are coming to them for help in understanding today’s consumer behaviour to drive innovation, but many of their traditional offerings and research methodologies are not fit for purpose. They are too slow, rely on small sample sizes, are expensive and come with biases that can lead to big errors. The recent failure of pollsters to predict Brexit and Trump’s victory are pertinent examples of this.

Given this backdrop, attracting and retaining clients by offering insights based on opinion data coupled with their specialist expertise may be a crucial differentiator for the sector going forward.

For example, a major application of opinion mining is its use in understanding the drivers underlying the purchasing behaviour of the market. Opinion data can allow researchers to predict what people are going to buy and understand the factors driving that behaviour.

A tweet in which a consumer expresses dissatisfaction with a new model of car or phone, for example, may not shed that much light in isolation. However, if many people do the same – and the drivers behind their dissatisfaction can be analysed, pinpointed and quantified – the resulting data can be used to improve subsequent product lines, or understand why competitors’ offerings may be outperforming.

Media: Personalised and on-demand

The future of the media landscape will be one characterised by an increasing focus on personalised content and on-demand viewing.

From an opinion mining perspective, helpfully, many viewers are both vocal and honest about their tastes, often turning to social media to express their feelings: be it about a pilot programme, a new channel or a major happening in a much-loved series.

By delving into this information, and the drivers behind it, media companies can gain access to the type of granular data needed to tailor their programming strategies – both by channel and by market – in order to deliver viewers the content they want in the form they want to consume it.

Mainstream news media is also struggling to understand their audiences and their feelings towards today’s stories. Fake news and “news feed bubbles” are creating significant pockets of public opinion that are not based on truth, undermining the legitimacy (and demand for) their journalism. Opinion mining offers a way for media to accurately measure and report on public opinion, regardless of its veracity.

Insight into the electorate

Governments and other state bodies are another group of organisations likely to turn to social media analysis in the search for greater clarity. 2017 will see a series of major political events: March witnessed the triggering of Article 50, April the announcement of a snap UK election, while May ushered in new era for French politics with Emmanuel Macron chosen as the French President. Later in the year, Germany will vote. The Chinese Communist Party’s five-yearly congress is scheduled for later this year and further interest rate hikes by the Fed are also due.

As states begin to push for a greater degree of foresight around the potential outcome of such key events, the insights hidden within the voice of the collective will likely be sought out increasingly through opinion mining.

Future of opinion mining

A wealth of unstructured opinion data – across social media and elsewhere – was, up until this point, a largely inaccessible resource. However, by tapping into the text analysis powers of artificial intelligence, and by overlaying it with the refinement of crowd-based analysis, the world of opinion mining is set to change the way governments and business conduct research, make decisions and measure the impact of these decisions.

This article was originally published here