A pre-trained machine learning model that has been tuned for sentiment classification of opinionated utterances.

Multilingual Sentiment Analysis Applications in Conversational Agents

Currently, conversational agents play an important role in the early diagnosis and symptom evaluation of a patient. At the same time, their effectiveness can further be improved after incorporating sentiment analysis in the respective conversational data. In this work, we explore the sentiment analysis model employed by the conversational agent developed by UNIC, as part of the ALAMEDA AI toolkit. The component performs sentiment analysis at sentence level and is based on the Holistic Cumulative sentiment classification model as proposed in [Agathangelou et al., 2022]. First, we delve into the challenges and innovations that were incorporated in the original architecture as proposed by [Agathangelou et al., 2022], until the sentiment analysis component for the ALAMEDA AI toolkit.

 

Sentiment Analysis Component Challenges and Innovations

From the beginning of this project a number of challenges were outstripped to turn the original architecture into a service application for the ALAMEDA AI toolkit. First, we had to comply with the private data regulations which forbid utilizing end users' post data for training any algorithm. On the other hand training a sentiment analysis model on general data is most likely to yield sup-optimal results. In general sentiment analysis is a data mining process which is deeply related to the domain characteristics of a corpus. The training of a model in one domain hardly does well in another. Such was the case we had to alleviate towards providing a performance-wise solution for the ALAMEDA AI toolkit and the sentiment analysis component. In solving this, generic and target-based features were utilized to deliver an artificial intelligence solution that would perform well in an application interface where end users' posts can range from general to personalized interest comments, reviews or feelings.

 

The Methodology challenge

As earlier discussed, our strategy involved the employment of a sentiment analysis method [Agathangelou et al., 2022], that among several sentiment analysis attributes also considers cognitive aspects of user's online reviews. Initially, we experimented with several domain datasets to train our algorithm, but unfortunately the prediction performance was poor when we validated the trained method in another domain. These first efforts, although unsuccessful, aligned with our knowledge about the domain characteristics of the sentiment analysis task we mentioned above. Finally, we selected one representative (dataset) for
training our algorithm and a number of fine-grained small-scale variable domain datasets for the validation. Table 1, displays the Amazon dataset which we employed for the training and the respective validation process of our sentiment analysis component. The intuition behind the selection of the Amazon
dataset was that it embeds a rich set of user profiles, consisting of opinions on several everyday consumer items or reviews on customer feedback services. Additionally Amazon dataset also contains a rich set of sentiment features (e.g., emotion polarity words, phrases, and more) that are essential to provide a well-established sentiment analysis and prediction method.


Table 1. The dataset for the training and the evaluation process

Training: Amazon reviews from products and services
(http://jmcauley.ucsd.edu/data/amazon/
Validation (Domain): (85%) Health and Personal Care, (15%) other domain reviews datasets


The Multilingual challenge

Another challenge we encountered in this project was the inclusion of several different languages for our model development and implementation. These different user-base languages should also be addressed in a real time setting. Normally, one would expect one model development for each user-base language, but in our implementation we innovated by applying one model in a multi-language solution. Initially, a corpus of 80k English (default language of communication within the ALAMEDA consortium) documents
augmented via a translation process to reach the amount of 320k documents that also included Greek (languages of the pilots), Italian and Romanian translated reviews. From a technical-based perspective our training corpus comprised 315k vocabulary terms with the respective language vocabulary items distributed as shown in Figure 1.

 

Figure 1. The distribution of the different language terms in our training corpus.

The Model development challenge

Next, follows the development of the architecture which provides the sentiment analysis task for the alameda sentiment analysis component. The employed method [Agathangelou et al., 2022], namely HolC (Holistic Cumulative), is a deep learning model, which by default alleviates the sentiment analysis task as a sequence of opinion phrases or reviews (see Figure 2a). HolC comprises three blocks: a Convolutional network for extracting features at sentence level, a Bi-directional Long Short Memory to grasp relations among nearby sentences in a document level, and an Attention network (Attention-Bi-LSTM) to further improve prediction performance. Finally, a single perceptron layer grasps cognitive sentiment assigning aspects of users' posts. This document-level setting, however, could not be applied directly in the alameda project because exploiting a sequence of phrases would also imply the exploitation of some of the end-user's private data.

Eventually, we proposed and implemented the modified architecture as this is presented in Figure 2b.


 

Figure 2. The generic (a) and the modified (b) architecture of the sentiment analysis task for the alameda project.

This modified architecture Figure 2b, complies both, with the alameda project regulations while also it provides a performance-wise solution. Figure 3 presents the performance statistics of the HolC architecture on the evaluation data as discussed above.

Figure 3. The evaluation results of alameda HolC implementation on the validation dataset

Pipeline structure

The pipeline of the sentiment analysis component is depicted in Figure 4. The component is a neural network framework that consists of a pre-trained deep learning model, the vocabulary of the trained corpus, source code, and appropriate libraries. When an end-user post is submitted to the conversational system, the content is processed as follows:
Initially, module (1) clears and filters-out unnecessary language terms. Next, module (2) tokenizes the text content, and converts it into an appropriate representation for the deep learning model to apply the prediction process in module (3). The prediction result, a distribution of probabilities i.e, [0.2, 0.8, 0.1] is received by the output module (4) which converts it into a meaningful representation i.e, [negative: 0.2, neutral :08, positive: 0.1].

Figure 4. The pipeline of the sentiment analysis component in the alameda project

In order to provide an intuition how our model performs sentiment analysis, Table 1 depicts some real examples with the respective predicted sentiment orientation and confidence scores.

Table 1: Some sample examples and their sentiment / confidence scores

Lang post sentiment /
conf. sc.
EN I have a hard time sleeping and when I do sleep, it's not a restful sleep neg / 68%
IT Mangio senza latticini. Il mio medico mi ha consigliato di eliminare le mie normali proteine a base di soia e mi ha suggerito di provare questo prodotto nei miei frullati quotidiani pos / 98%
GR Τρώω χωρίς γαλακτοκομικά. Ο γιατρός μου συνέστησε να αφαιρέσω την κανονική μου πρωτεΐνη με βάση τη σόγια και μου πρότεινε να δοκιμάσω αυτό το προϊόν στα καθημερινά μου smoothies. pos / 97%
RO Mănânc fără lactate. Medicul meu m-au sfătuit să-și elimin proteinele obișnuite pe bază de soia și mi-a sugerat să încerc acest produs în smoothie-urile mele zilnice pos / 98%
EN I can't taste the honey or lemon, just a mild menthol flavor but not a bad taste. I do have a slight cold and while these did help open my sinuses, it did nothing for my throat. neu /73%

 

Future Directions

While the current sentiment analysis model processes and classifies end-user posts individually, future plans aim to incorporate the dialogue context, speaker personality, and emotional intent to enhance the conversational agent’s services. The inclusion of emotional encoding in conversations is also on the roadmap for future updates. An updated version of the sentiment analysis model specifically trained for conversations will be included in D5.4. Moreover, having gained the experience of the first version, our efforts will be focused on the overall performance improvement. Some promising results relate this improvement with the joint utilization of other modals. In this setting text features are combined with audio or image features. We intend to investigate our next model development in the fusion of text and image altogether in a single unit.

Conclusions

Sentiment analysis plays a crucial role in improving conversational agents’ effectiveness, and the ALAMEDA AI toolkit incorporates a powerful sentiment analysis model. Through careful data cleaning and processing, the model achieves accurate results at the sentence-level. The model exhibits scalability by supporting multiple languages, resulting in enhanced performance and robustness. Future updates will focus on exploiting conversation aspects and incorporating emotional encoding, further advancing the capabilities of conversational agents within the ALAMEDA project. [Agathangelou and Katakis, 2022] P. Agathangelou, I. Katakis, Balancing between holistic and cumulative sentiment classification, Online Social Networks and Media 29 (2022)100199.doi:https://doi.org/10.1016/j.osnem. 2022.100199.835 URL https://www.sciencedirect.com/science/article/pii/S2468696422000039