Major Studio 2 | Smile

For our first project of Major Studio 2 (forever after referred to as MS2) my wonderful classmates got to pick what I would be working on for the next couple of weeks. The final decision was “Smile” my proposal to build an urban art installation that would be triggered by users smiles.

I had never done anything around video processing, facial recognition, let alone emotion recognition before, so this has been an interesting experience.

The first place I landed when looking for some inspiration for this project was Kyle McDonald’s OpenFrameworks FaceTracker library. This is actually a build of a library created by Jason Saraghi that is now maintained by Kyle for C++ OpenFrameworks builds. It is free for non-commercial use, but you have to license it for commercial use (which I did not realize).

Upon trying to compile some of the examples for this I had alot of issues. I couldn’t get anything to work, not even the examples provided.

Luckily, I found a great example for Processing (which I haven’t touched since Bootcamp :O) that compiled right away, and I was able to easily change out the code they have (which draws a yellow bar graph thing with the likelihood of a smile) to instead use the smile as a trigger. Woo hoo, it works!

My first thought for the urban installation was that the smile would trigger a beautiful video and/or motion-graphic interaction – it has to be enticing, right? With that in mind, I decided that the best direction to move from here was to load in video and have the smile trigger playing it/stopping it/pausing it.

I first tried it out using my favorite prototyping music video, Big Bang’s “Fantastic Baby”, where smiling was required to get the video to play, not smiling would shut it off.

Here is the result. Pretty fantastic… baby.

This was a lot of fun to play with, but the video didn’t really mean anything, it was just a prototyping tool. I then started thinking about what types of videos or interactions could be cool with this. For some reason, that had me moving towards the idea of imposing the video over the face (not fully) but partially in a fun and exciting way.

As I was staring at my own face in the computer screen over and over, it made me think of one of my favorite movies of all time, The Rocky Horror Picture Show, and the begining credits which just feature (I think) Tim Curry’s lips singing the intro song “Science Fiction Double Feature”.

I thought this would be a really fun video to try next for a prototype, while blending the background levels in order to have it somewhat impose over the face. Here is the result.

With all of these lovely experiments working, I decided it was time to go back to the drawing board to think about what to do next.

The interaction of having a smile trigger something, a video, music, animation etc. is not difficult and it should be easy to implement into an urban art piece, it would just require considerations around having a mobile web camera, or being able to setup the interaction using a Raspberry Pi – I say easy, but I mean the steps seem clear I’m sure it is challenging!

However, while doing this project I kept thinking about how much my face hurt having to sit there smiling stupidly at the computer in order to watch a video. I felt I was having a super pavlovian type of reaction, having to make an emotion/do something to get what I wanted. Of course thoughts like this immediately down my giant echo-chamber of science fiction background and what comes back up is whispers of surrendering agency, being manipulated into actions to get what we want, even if they are silly. I actually thought it would be kind of funny (based on this) to have a phone app that would only work if you were making different faces at it, just to see how far people would go in public! Anyway, I realize that this is not the only path forward for such interactions, and that interactive and emotionally aware technology has wonderful beautiful applications. However I can’t help but think that there are other consequences as well. Something to think about as I do this project!

In order to get inspired for how to move forward, I started looking into some precedents of other smile and emotion controlled technology.

The first one I found is quite famous, and basically my project in a much nicer wrapper!

Smile TV, By David Hedberg

Just a decade ago it was much clearer who dictates information which we absorb. Now, with content widely accessible the question is no longer if we can receive but if we are receptive. This installation uses facial recognition technology to re-consider viewers engagement &  how content is accessed.

The next precedent I looked at was around communication using emotion recognition and how the two might be used together. It is actually from an MFADT alum!

Emotichat, Decho Pitukcharoen


This MFADT 2014 Thesis project explored how online communication is often misinterpreted and can use facial recognition to supplement the interactions.

Both of these projects use the emotion recognition for very different reasons yet both of them also fundamentally centered around accessing/modifying content based on emotional triggers.

This was very interesting to me, as I have always considered my emotions rather private, even though I am someone who has things written all over my face and who blushes furiously when put on the spot. I kept coming back to the idea of what makes a meaningful interaction between someone and the technology. I am especialy interested in this given that we seem to tradeoff much of our time interacting with each other for interacting with our technology…

After all of this, I decided to reframe my research question a bit and think of some sub-questions to answer in this project.

What can we unlock with the power of a smile? What interactions can we make that are meaningful?

  • How can we use emotion recognition to power interactions between people and technology that is more meaningful and empathetic?
  • What are the dystopian possible futures in which this technology could be abused?
  • Is one person’s happiness another’s misery?
  • What does it mean to “just keep smiling”?

From here I also redid my design statement for the project (knowing that it might change again!)

I will design an interactive art piece that is triggered by users smile…

  • Show positive content in response in order to evoke a playful happy feeling of more personable interactive tech.
  • Show negative content and make users smile to avert it, or juxtapose negative content and positive content in interesting ways.
  • Blend content together in an interesting and beautiful way.

I keep coming back to this idea of our emotions controlling our content in different ways, whether the content is positive or negative. I think there could be good applications for this (imagine a program that would make Netflix recommendations based on your emotions, yes please!!!), but I also think there could be some sick research applications of this (imagine having to keep smiling or else you’d see horrible content, or have something horrible happen to one of your friends?)

Since I often get inspired by songs, one song that is super inspiring to me right now in this regard is Mama’s Broken Heart, by Miranda Lambert

It is all about how one has to maintain dignity and “face” even when you are upset. It is a bit gendered in its application, the idea of “fixing your makeup” in order to show you aren’t falling apart, but this can be universally translated to “quieting one’s thoughts”, “gathering one’s emotions”, “schooling one’s face” etc. These are all things I’m interested in moving forward. What will it mean when we have to “master our emotions” all of the time lest they trigger unpredictable behaviors from our surroundings and technology!?!

I have also been doing some reading lately courtesy of the Islip Public Library in the town I am from. They actually have books available and really great hours (and apparently a 3D printing demo all winter, what?) so I stocked up. One great book I picked up is Brave New Worlds a collection of dystopic short stories from famous scifi authors, edited by John Joseph Adams. Some of the stories in this actually explore the idea of emotion recognition as well, and of our reliance on technology in general.

“The Pedestrian” by Ray Bradbury had some really interesting points about our gradual reliance on screens for all forms of entertainment. It features a walker who gets stopped and questioned because he is out walking – everyone else is at home watching TV. As he goes around he muses on what everyone is watching and how no one is ever out anymore, they just go straight from their homes to cars to offices and right back. I think this is interesting given it was written before the rise of mobile phones and continual connectivity. I wonder what he would think about us taking our screens with us as we  walk instead..

Another two of the stories directly had to do with some of the potential implications of emotion recognition.  In “Dead Space for the Unexpected” by Geoff Ryman, a middle manager in a corporation has his performance judged not just by the results he gets, but his emotional and biometric readouts as he gets them. Every face, heart rate increase, sweaty palm, and eye tick is recorded during his interactions with subordinates and he is rated for his meetings which contribute to his all important average rating. Every employee has one and is regulated the same way. Creepy stuff. As cool as emotion recognition is, if forced on people or normalized in certain social interactions, it can quickly become  a method of control. I also finally read the famous “The Minority Report” by Philip K. Dick which talks about the idea of precognition of crimes and prosecution as such. In this book, they don’t quite have emotion recognition, but they certainly have facial recognition as he is chased around trying to evade the police. It also calls into question the reliability of results that are computed purely from probability. In this case there is a fundamental error with the pre-cog system of 3 mutants humans with physic abilities where they all have diverging takes of the future event.

Next Steps

MS2 - Project 1 - First Prototypes

Next, I will come up with some working prototypes of my interactive art piece. I think I want to use a web camera or the Kinect I have had lying around my apartment so that I don’t have to rely on my laptop’s built-in camera anymore. It’s really not fun to sit and stare at the computer this way!!


Here is my full presentation for this first round of MS2 Project 1.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.