Understanding emotions as a business task is the challenge software development companies often face today. Technology progresses at a fast pace, and the future from good old science fiction is upon us. Emotion recognition is one of the discoveries that seemed to be impossible a decade ago and is a matter of technology to implement now.

Following current hot trends and in cooperation with our partners from Belarus, we have built a portable system of real-time emotion recognition via facial expressions for beauty salons. The article discovers practical technical aspects, and also it touches the privacy issue. A reader may find what we’ve managed to achieve, how our emotion recognition system is applied, and which points require improvements, and finally, what potential of our emotion recognition system remains uncovered.


Emotion recognition on the service of a beauty industry

The birth of an idea

Businesses care about customer engagement to gain trust and loyalty and thus return a client. To be a client-centric business is to serve your customers the way they feel special, in other words, to rouse emotions. No surprise that in the era of automation, emotion recognition becomes a technology. In a competitive chase, an effective business may need a tool to evaluate satisfaction from the buyer’s experience. Emotion recognition systems can help a company to provide better services. Such software can capture and filter the customers’ feelings impartially, gather, process, and analyze the data, structure, and systemize results in accordance with business purposes.

A few years ago, to learn a new trend of face recognition and to fulfill the thirst of knowledge, our experts started experimenting with toolkits and frameworks. The best way to develop a skill is by doing. We’ve tried different application areas of computer vision on practice:

Hosting our client in SYTOSS’s office in Kharkiv, we shared our research, and that conversation set the start of a new project.

Business tasks to solve by recognizing emotions

Orientation on customer engagement and personal experience is the trend and the only possible method of doing business for some industries, especially for services like beauty salons, sports, SPAs, and others. Building connections and boosting satisfaction are the additional tasks of specialists providing the services, like barbers, nailers, coaches, or fitness instructors. The usual practice for business owners is to award those masters who work on increasing engagement. But how to evaluate clients’ satisfaction in an unbiased, quick, and effective manner?

In our case, the emotion recognition software is intended for beauty salons, so the initial requirements were as follows:

  • Easy-to-use, a receptionist and a business owner should be able to handle software painlessly.

  • The real-time operation, a beauty salon client should not wait, and an owner may receive current data.

  • Portable hardware, which can be moved and reorganized anytime and anywhere without complex mounting or assembling.

  • Reasonable pricing, an average beauty salon could afford emotion recognition implementation.

  • And, privacy compliance, the system should not store photos and clients’ sensitive information about clients without their permission.

Taking into account the requirements, our AI experts investigated the field and offered the solution, consisting of affordable hardware available at the market, and custom-developed software using open-source toolkits. The demand for real-time operation narrowed our lookout of tools significantly, and we had to exclude popular cloud solutions by Amazon and Microsoft from the comparison.

A portable kit of emotion recognition hardware:

  • a laptop and

  • one 2 Megapixel IP network camera.

Emotion recognition tools and frameworks:

Emotion recognition software in operation

Buyer's journey. A receptionist’s post is usually the first point of communication when a client enters a beauty salon, and the last one when it leaves. A receptionist greets clients, asks them about their appointments, has small talks, and invites a client to the appointed specialist. After the procedure ends, a client goes back to a receptionist post to pay for a service, or to schedule the next appointment. That is why the post of a receptionist is an excellent choice to install hardware for the emotion recognition system.

Hardware mounting. As a result of different tests and trials, we’ve found that the distance between the IP camera and a client should not exceed two meters. At the bigger distance, the precision of recognition results suffers, as the system zooms shots to detect a face. Also, lightning conditions are essential for the quality of the photos. Direct sun or lights may blind the camera, and on the opposite, the poorly lighted room may cause the decline of photos quality as well. As soon as emotions are recognized in real-time, a laptop from the middle price range is enough to process the data input from up to ten cameras.

Software at work. A 2 megapixel IP camera makes up to 15 shots of a client in a second. The emotion recognition system marks a client with the ID number, and four neural networks start processing the visual data.

Emotion recognition software

  • The first neural network detects faces.

  • The second neural network filters photos according to a face position and leaves front-faced shots only.

  • The third one identifies the client's gender and age.

  • And, the last neural network recognizes the emotion.

The algorithm of the delivered software can recognize five emotions:

  • happy (satisfied, excited)

  • angry (irritated, annoyed)

  • surprised (reacting on something unexpected or unusual)

  • neutral (calm, emotionless)

  • unhappy (sad, disappointed).

The system does not store the client’s photos. AI makes the decision on the expressed emotion based on the sums of emotion probabilities calculated for one session of a client. As soon as the system works in real-time, we use algorithm accelerators to speed up the data processing.

The results about the client's emotions are stored in the database, displayed as a dashboard of diagrams, and are related to a specialist who accepted the clients. The beauty salon owner can evaluate the level of customer engagement in reference to a specified specialist. And, driven by the emotion analysis data, a business owner makes a decision about an employee's performance and this master wage bonus.

Emotion recognition

The future of emotion recognition

The implemented system of emotion recognition for beauty salons is scalable and extendable. Now a business owner can add up to ten beauty salons and correspondingly ten cameras to one piece of installed software.

Among the distinct advantages, the prevailing is the automation of the process, saving time and resources, as well as data-driven, unbiased decision making.

On the other hand, the implemented emotion recognition software often detects neutral results. However, the issue of undefined emotion is common for emotion recognition technology using facial expressions as the only data input. Emotions are a complicated subject to understand. People tend to feel a mixture of emotions. Also, there are a lot of cultural differences in the way people express emotion, primarily when we deal with their facial movements.

The possible improvements of the emotion recognition software may be:

  • Continue training neural networks on updated datasets, or selected photo sets relevant to the clients’ culture;

  • Extend the system to a multi-modal emotion recognition software, which will use facial expressions and a voice to make predictions about emotions and clients’ satisfaction.

Conclusions

What would you feel about the statement that you do not need to be a human to recognize emotions?

We are social beings, which means that we love to interact with each other. Understanding emotions is the secret power to break communication and discover another person. Imagine if we had some cognitive disorder and lost our empathy for a second. You do not know what the meaning of a smile is anymore. Other people would be like aliens from another planet. They become hardly intelligible. It would be a shocking experience, wouldn’t it?

Emotion recognition technology is an impressive breakthrough with many potential areas of application: service industry, healthcare, social care, sports, psychological assistance to list a few. So, if your business is open for innovations and you look for a software development partner with the experience of implementing frontline technological trends into highly effective systems, SYTOSS may be a match.

Helping One Look Good: the Less Known Human Appearance-Related Uses of Image Recognition
  09/16/2019  

Helping One Look Good: the Less Known Human Appearance-Related Uses of Image Recognition

While creating a lot of Quality Assurance-related opportunities in a number of industries, Image Recognition is known to be a sheer game changer in the security and surveillance industry.

However, besides t...
Read More keyboard_arrow_right

Image Recognition: Can an Image Recognition App Become the Quality Boost Your Business Needs?
  02/15/2019  

Image Recognition: Can an Image Recognition App Become the Quality Boost Your Business Needs?

While the Image Recognition technology is, generally, on the rise and finding more interesting applications currently, Image Recognition apps are seldom spoken about as an often unique means of improving the existi...
Read More keyboard_arrow_right

Face Recognition: What Can a Face Recognition App Be Capable Of and How to Make It Happen?
  01/04/2019  

Face Recognition: What Can a Face Recognition App Be Capable Of and How to Make It Happen?

Face Recognition apps are likely to shortly impact profoundly an even greater number of industry sectors and business niches than they already have.

However, developing a robust and well-performing Face Rec...
Read More keyboard_arrow_right