Feels Music Pro was the outcome of our time at re:cognition hack, a two day hackathon organised by Microsoft and Moov2. The focus of the hack was using Microsoft’s Cognitive Service APIs and a mix of other Microsoft software and hardware to create something interesting and cool.

Microsoft’s Emotion API can take a photo that you send it and return a list of emotion probabilities for each detected face.

By capturing a frame from the webcam and periodically sending it to the API (around every 0.5 seconds), it gives us the most likely emotion with a fairly low latency, low enough that we could consider it realtime for out needs.

We decided to turn this live emotion detection into a game, taking inspiration from Guitar Hero and similar rhythm games. Emojis would come down slowly down the screen in time to the music and you’d have to match your facial expression by the time it reached the bottom.

We built the game using HTML/CSS/JS. It was quick to develop a working prototype on the first day, then move onto fine tuning and styling on the second day of the event.

The only issues we really had were with detecting emotion when the computer screen was reflected onto glasses lenses. This obscured the eyes significantly and it seems the API could detect a face, but struggled with figuring out the emotion. Another issue we had was sometimes facial hair interfered with expected output, like moustaches looking like frowns.

Since the hack, we’ve fixed some of the code and added new graphics and effects for use as a kiosk game at events. This newer version deserved an updated name, Feels Music Pro 2017 Deluxe.

Due to API key limitations, we haven’t been able to share it around quite yet, but we are looking into it and will soon we will release the source code on Github.

Links & resources

Empton API: Microsoft’s Emotion API
re:cognition hack gallery: Flickr gallery