A wealth of unstructured opinion data exists online. Every day, millions of consumers add to this data when they share their opinion on a range of things, including feedback about their experiences with products and services. This feedback is volunteered, it contains the raw, unsolicited views and opinions about a brand, individual or event. The challenge of analysing and making sense of this data at scale has led to a type of analysis known as opinion mining.
In the broadest terms, opinion mining is the science of using text analysis to understand the drivers of public sentiment.
All text is inherently minable. As such, while social media may be an obvious source of current opinion, reviews, call centre transcripts, online forums and survey responses can all prove equally useful. Social media, however, provides a volunteered source of consumer opinion.
Whereas sentiment analysis – a predecessor to the field of opinion mining – examines how people feel about a given topic (be it positive or negative), opinion mining goes a level deeper, to understand the conversation drivers behind the sentiment, i.e. why people feel the way do
Individual opinions are often reflective of a broader view. A single customer who takes issue with a bank or telco’s new product design on social media likely speaks for many others.
Gather enough opinions – and analyse them correctly – and you’ve got an accurate gauge of the feelings of many consumers. This relates not only to how people feel, but the drivers underlying why they feel the way they do.
Automotive topic wheel looking at main drivers of sentiment towards automotive brands
By understanding what is driving the sentiment and how one is performing based on Net Sentiment, opinion data can be used to expose critical areas of strength and weakness. This data allows decision-makers in business, from customer experience and marketing to risk and compliance teams, to make the targeted, strategic overhauls needed to reinvigorate profitability or reclaim slipping market share.
By analysing conversations for both sentiment and the topics driving that sentiment, a retail bank might, for example, discover, that branch queue length and call-centre waiting times are the most prevalent topics in negative consumer feedback.
As this data is public, organisations can also assess their competitor’s performance. For example, a fast-food chain might be interested to know that relative to their closest competitor, many consider their fries portion size too small but consider their burger pricing most favourably.
Yet it’s possible to delve deeper still. The topic of customer service could be further broken down into the sub-categories of turnaround time, order correctness and delivery time. The business may have a great in-store turnaround, but fail when it comes to deliveries. Knowing which issue to target – and why – is key. This can be further mapped to customer journey stages and channels, allowing organisations to understand where they are succeeding with customer experience.
The quantity of minable text available is vast. Given the volume, a degree of computational brute force is necessary. With this in mind, artificial intelligence can be used to good effect to carry out a large part of the deciphering work.
Definitions
- Artificial intelligence (AI)
- The simulation of human intelligence by machines, including processes related to learning, reasoning and self-correction, with applications in speech recognition, machine vision and text analysis, among others.
- Machine learning
- A sub-field of artificial intelligence that allows computers to learn without being explicitly programmed.
- Algorithm
- A set of rules or procedures to follow when solving a given problem; a sequence of actions to be performed
The quantity of minable text available is vast. Given the volume, a degree of computational brute force is necessary. With this in mind, artificial intelligence can be used to good effect to carry out a large part of the deciphering work.
The area of machine learning dedicated to making sense of the written word is known as natural language processing.
There are various approaches, including:
Let’s start with a simple example: an online author tweets about a car brand, “Love the new model – great ride!”. The first step in making sense of this text is to establish the comment’s overall polarity. In other words, how the author feels about the topic at the broadest level. One of the simplest approaches to determining polarity is by using the presence of certain keywords to assign a comment to one of three buckets: positive, negative or neutral.
On detecting a word such as love – or equally, like, hate, dislike, enjoy, etc. – a textual analysis algorithm would classify the comment that possesses it accordingly. Unfortunately, understanding language is not always this simple. This approach also offers little nuance or indication of why the author felt that way.
On detecting a word such as love – or equally, like, hate, dislike, enjoy, etc. – a textual analysis algorithm would classify the comment that possesses it accordingly. Unfortunately, understanding language is not always this simple. This approach also offers little nuance or indication of why the author felt that way.
AI-driven approaches, though increasingly sophisticated, remain imperfect. People communicate in complex ways: slang, vernacular, emojis, misspellings, figurative language and long, meandering sentences can all limit machine interpretation. The offhand, unstructured dialogue that fills the web, in particular, is culturally and socially complex.
The same things that make language lively and human – humour, slang, innuendo, sarcasm, colloquialisms, figures of speech – are the same things that confound machines. Adding a layer of human insight can significantly improve the accuracy with which these nuances can be interpreted. An emerging approach is to use crowd-sourcing technology to add human insight to data at scale. This involves sending short-form texts, such as tweets to real people and asking them to analyse it and label it. Their assessment is checked against other crowd members to achieve maximum accuracy. If this data is fed back to an algorithm, it can also be used to teach machines to better interpret data based on human understanding.
In addition to humans’ cultural and social edge over machines, people are also better able to understand words and sentences with possible double meanings and identify unclear objects of reference.Humans are also better able to extract meaning in the face of unintentional ambiguity, such as misspellings and punctuation issues, and may also be superior at gauging the strength of the sentiment.
Given that the most crucial work – isolating the specific drivers of sentiment, with fine discernment – is also the trickiest, human integration can be a valuable approach and provide the missing link when it comes to generating accurate opinion data.
Consider: “The new phone is awful – hate the buttons!”
Here, the opinion expressed is multi-layered:
Humans get humans and can provide context and understanding to complex conversations that pure AI struggles with currently. This approach can be applied to diverse topics including brand perceptions, market research and political issues.
At selected outlets, the chain launched two new milkshake products that initially saw promising sales figures as curious consumers were willing to try the new flavours. However, social media feedback about one of the flavours was overwhelmingly negative and the chain was able to act quickly and remove the item from their menus. Thanks to this data they avoided costly rollouts and further damage to their customers’ experience. Typically, this feedback about the product would only have been surfaced months later in traditional survey data. The real-time monitoring and mining of social media provided them with crucial cost-saving insights.
Online privacy is a growing concern for many. So reading that online conversations are “mined” might understandably cause some discomfort. Critically, opinion mining is not surveillance – nor is it profiling. Rather, it is a process of data aggregation that generates consumer insights that improve decision making and yields better outcomes for consumers. Responsible opinion mining keeps personal privacy paramount.
Organisations at the forefront of customer experience and social customer service are already making use of opinion mining techniques. By tapping into the universe of unstructured opinion data, these firms are enabled to make customer-centric operational and strategic improvements that lead to better customer outcomes.
As more customers seek customer service on social media, opinion mining techniques are being used by leading firms to identify customer conversation - from within all of the irrelevant noise - that requires their attention and action. By prioritising the most valuable data in real-time, like a customer threatening to cancel their contract, organisations like banks and insurers, are using opinion mining techniques to improve retention & acquisition rates and deliver superior customer experience.