By Maria Hakim , Middle East Account Manager DataEQ
This article was originally featured in Datatechvibe.
In an economy where data is the new currency, it is predicted that artificial intelligence (AI) will add as much as $320 billion to GDP across the Middle East by 2030, with the UAE pegged to see the largest impact of near 14 per cent of GDP. There will, however, still be areas in which AI inherently falls short – social media sentiment analysis being one of these.
Sentiment analysis is the process of extracting information from a body of text. The sentiment of the text (positive, negative, or neutral) reflects how people think and feel about a particular topic, issue or individual. By extension, social media sentiment analysis entails mining social platforms for public views that are not available in other forums.
Compared to traditional polling methods, social media sentiment analysis offers a more reliable and unfiltered understanding of what is happening in today’s world. Over-and-above the pertinence provided by the real-time nature of social media, people are typically more honest about their true feelings on social platforms than in conventional questionnaires.
It’s no surprise that social media sentiment analysis correctly predicted both Brexit and Donald Trump’s victory over Hilary Clinton in 2016, despite traditional media and polls confidently calling the opposite results in both instances.
Online media has empowered millions of citizens around the globe, and the democratisation of public opinion means that anyone with internet access can volunteer their opinion. Being able to accurately measure and understand how and why people feel the way they do is therefore of high strategic value and can provide a significant competitive advantage.
So why can AI drive a car and compose music but falls short with sentiment analysis? Essentially, it comes down to the complexity of human communication, especially when involving a category IV language like Arabic. With over 25 dialects, the Arabic language has a complex morphology, involving a high degree of ambiguity and new word formations with rich semantic meanings.
Adding further complication, Arabic online chatter is rife with irony, sarcasm, figurative language, slang, and localised idiosyncrasies – all of which pose interpretive problems for natural language processing (NLP) algorithms.
The outcome – social media sentiment analysis driven purely by NLP tends to deliver inaccurate results. But AI still plays an important role in screening at scale, analysing conversations for relevance and performing keyword matching. It just means that to ensure the accuracy of sentiment-bearing conversation, an element of human insight must be overlayed over the analytical work performed by NLP.
One approach that has proven popular – and was utilised in correctly predicting both the US election and Brexit results – is that of crowdsourcing. In other words, real people – a Crowd – are enlisted to perform micro jobs of correctly assigning sentiment in return for micropayments.
Ultimately, this unique combination of using scaled human intelligence alongside machine learning has proven key to providing best-in-class sentiment analysis and, importantly, structures the raw data for things that AI cannot achieve on its own. This, in turn, unlocks enterprise data use cases in the customer experience, customer service and compliance management space that cannot be supported by “AI-only” social listening competitors.
For example, financial services firms are increasingly using social media sentiment analysis to identify online complaints accurately and then structure them against the Treating Customers Fairly (TCF) market conduct framework – a task that is impossible for AI to achieve but one that is now a legal requirement in many markets. Net Sentiment is also becoming a valued KPI to understand business performance against customer experience indicators in real-time.
Of course, it’s not out of the question that AI technology may one day advance to a point where machines can accurately assess human emotion and sentiment in real-time, but until that day comes, human intervention will continue to play a critical role in the process of accurate social media sentiment analysis.