A wealth of unstructured opinion data exists online. Every day, millions of consumers add to this data when they share their opinion on a range of things, including feedback about their experiences with products and services. This feedback is volunteered, it contains the raw, unsolicited views and opinions about a brand, individual or event. The challenge of analysing and making sense of this data at scale has led to a type of analysis known as opinion mining.
Opinion mining: The basics
In the broadest terms, opinion mining is the science of using text analysis to understand the drivers of public sentiment.
All text is inherently minable. As such, while social media may be an obvious source of current opinion, reviews, call centre transcripts, online forums and survey responses can all prove equally useful. Social media, however, provides a volunteered source of consumer opinion.
Sentiment analysis versus opinion mining
Whereas sentiment analysis – a predecessor to the field of opinion mining – examines how people feel about a given topic (be it positive or negative), opinion mining goes a level deeper, to understand the conversation drivers behind the sentiment, i.e. why people feel the way do
Why mine?
Individual opinions are often reflective of a broader view. A single customer who takes issue with a bank or telco’s new product design on social media likely speaks for many others.
Gather enough opinions – and analyse them correctly – and you’ve got an accurate gauge of the feelings of many consumers. This relates not only to how people feel, but the drivers underlying why they feel the way they do.
Automotive topic wheel looking at main drivers of sentiment towards automotive brands
What is driving the sentiment
By understanding what is driving the sentiment and how one is performing based on Net Sentiment, opinion data can be used to expose critical areas of strength and weakness. This data allows decision-makers in business, from customer experience and marketing to risk and compliance teams, to make the targeted, strategic overhauls needed to reinvigorate profitability or reclaim slipping market share.
By analysing conversations for both sentiment and the topics driving that sentiment, a retail bank might, for example, discover, that branch queue length and call-centre waiting times are the most prevalent topics in negative consumer feedback.
As this data is public, organisations can also assess their competitor’s performance. For example, a fast-food chain might be interested to know that relative to their closest competitor, many consider their fries portion size too small but consider their burger pricing most favourably.
Yet it’s possible to delve deeper still. The topic of customer service could be further broken down into the sub-categories of turnaround time, order correctness and delivery time. The business may have a great in-store turnaround, but fail when it comes to deliveries. Knowing which issue to target – and why – is key. This can be further mapped to customer journey stages and channels, allowing organisations to understand where they are succeeding with customer experience.
AI vs the Crowd
The quantity of minable text available is vast. Given the volume, a degree of computational brute force is necessary. With this in mind, artificial intelligence can be used to good effect to carry out a large part of the deciphering work.
Definitions
- Artificial intelligence (AI)
- The simulation of human intelligence by machines, including processes related to learning, reasoning and self-correction, with applications in speech recognition, machine vision and text analysis, among others.
- Machine learning
- A sub-field of artificial intelligence that allows computers to learn without being explicitly programmed.
- Algorithm
- A set of rules or procedures to follow when solving a given problem; a sequence of actions to be performed
The quantity of minable text available is vast. Given the volume, a degree of computational brute force is necessary. With this in mind, artificial intelligence can be used to good effect to carry out a large part of the deciphering work.
Machine learning approaches
The area of machine learning dedicated to making sense of the written word is known as natural language processing.
There are various approaches, including:
- Supervised machine learning, which make use of structured data and/or human annotation (i.e. a set of ‘answers’) from which machines learn in order to make future inferences;
- Semi and unsupervised machine learning, under which no guidance is supplied, and machines must learn to interpret unstructured data with minimal or no guidance;
- Deep learning. These are complex, multi-level systems. Recent advances in the sphere have shown promising results.
Buckets and basics
Let’s start with a simple example: an online author tweets about a car brand, “Love the new model – great ride!”. The first step in making sense of this text is to establish the comment’s overall polarity. In other words, how the author feels about the topic at the broadest level. One of the simplest approaches to determining polarity is by using the presence of certain keywords to assign a comment to one of three buckets: positive, negative or neutral.
On detecting a word such as love – or equally, like, hate, dislike, enjoy, etc. – a textual analysis algorithm would classify the comment that possesses it accordingly. Unfortunately, understanding language is not always this simple. This approach also offers little nuance or indication of why the author felt that way.
On detecting a word such as love – or equally, like, hate, dislike, enjoy, etc. – a textual analysis algorithm would classify the comment that possesses it accordingly. Unfortunately, understanding language is not always this simple. This approach also offers little nuance or indication of why the author felt that way.
Improving accuracy with humans
AI-driven approaches, though increasingly sophisticated, remain imperfect. People communicate in complex ways: slang, vernacular, emojis, misspellings, figurative language and long, meandering sentences can all limit machine interpretation. The offhand, unstructured dialogue that fills the web, in particular, is culturally and socially complex.
The same things that make language lively and human – humour, slang, innuendo, sarcasm, colloquialisms, figures of speech – are the same things that confound machines. Adding a layer of human insight can significantly improve the accuracy with which these nuances can be interpreted. An emerging approach is to use crowd-sourcing technology to add human insight to data at scale. This involves sending short-form texts, such as tweets to real people and asking them to analyse it and label it. Their assessment is checked against other crowd members to achieve maximum accuracy. If this data is fed back to an algorithm, it can also be used to teach machines to better interpret data based on human understanding.
There are three main approaches to crowd analysis:
- Manual coding – Using internal resources to look at each data point and manually label it for sentiment and topics. This method can be very accurate but is time-consuming and lacks scalability.
- External crowd platform – Exporting collected data and sending it to an external crowd-sourcing platform to process and label. This can improve scalability, but the visibility of how crowd members are handling the data may be limited, and expertise in setting up questions for the crowd may be required.
- An integrated approach – Data collection and opinion mining happen seamlessly in the same platform. The crowd-sourcing process is managed and therefore requires less set up to initiate a project. Results may be accessed faster and can be analysed alongside the original data.
(Human) mind over matter
In addition to humans’ cultural and social edge over machines, people are also better able to understand words and sentences with possible double meanings and identify unclear objects of reference.Humans are also better able to extract meaning in the face of unintentional ambiguity, such as misspellings and punctuation issues, and may also be superior at gauging the strength of the sentiment.
Drilling down to the drivers and understanding when certain behaviours occur
Given that the most crucial work – isolating the specific drivers of sentiment, with fine discernment – is also the trickiest, human integration can be a valuable approach and provide the missing link when it comes to generating accurate opinion data.
Consider: “The new phone is awful – hate the buttons!”
Here, the opinion expressed is multi-layered:
- Phone brand – negative
- Driver of sentiment – new products
- Specifics – keypad
Humans get humans and can provide context and understanding to complex conversations that pure AI struggles with currently. This approach can be applied to diverse topics including brand perceptions, market research and political issues.
Opinion mining at work
Fast-food chain used structured social media data to streamline their menu and product testing.
At selected outlets, the chain launched two new milkshake products that initially saw promising sales figures as curious consumers were willing to try the new flavours. However, social media feedback about one of the flavours was overwhelmingly negative and the chain was able to act quickly and remove the item from their menus. Thanks to this data they avoided costly rollouts and further damage to their customers’ experience. Typically, this feedback about the product would only have been surfaced months later in traditional survey data. The real-time monitoring and mining of social media provided them with crucial cost-saving insights.
Using data responsibly: What opinion mining is and is not
Online privacy is a growing concern for many. So reading that online conversations are “mined” might understandably cause some discomfort. Critically, opinion mining is not surveillance – nor is it profiling. Rather, it is a process of data aggregation that generates consumer insights that improve decision making and yields better outcomes for consumers. Responsible opinion mining keeps personal privacy paramount.
Opinion mining today: Find and prioritise your most valuable customer interactions
Organisations at the forefront of customer experience and social customer service are already making use of opinion mining techniques. By tapping into the universe of unstructured opinion data, these firms are enabled to make customer-centric operational and strategic improvements that lead to better customer outcomes.
As more customers seek customer service on social media, opinion mining techniques are being used by leading firms to identify customer conversation - from within all of the irrelevant noise - that requires their attention and action. By prioritising the most valuable data in real-time, like a customer threatening to cancel their contract, organisations like banks and insurers, are using opinion mining techniques to improve retention & acquisition rates and deliver superior customer experience.