Machines that can read your face and interpret your emotions are no longer only a domain of science fiction movies. Sean Mulholland is one reason for it. He initially studied to become a video and graphic artist for film and music videos, but could not deny his lifelong passion for technology. Today, he works for one of the most respected design firms in the world – IDEO. How does one go from the arts to embracing the entrepreneurial spirit of San Francisco? Read the following interview with our former speaker and find out!
So Sean, could you tell us a little bit more about where you come from and how you got into the tech field?
I’m originally from Chicago but I’ve been living in San Francisco for about 18 years now. I was always a kid that took well to technology but I didn’t want to go for an engineering type of degree. I ended up going to an art school but being in San Francisco, the center of technology, I naturally started to work in tech during my second year of art school. Eventually, I was able to bring art and tech together and started to do strategy and design for digital products that brought me to where I am today. IDEO is kind of the best of both worlds, it brings both of them together.
Did you always think that you were going to end up in a tech sphere or was it a coincidence?
I think so, I definitely thought I’d end up in tech. My initial thought when I went to art school was that I would to special effects for Hollywood or music videos. I thought I would be using computers more as an artistic tool but I ended up starting to design for the tech itself.
How did having an art degree help you while working in tech?
It definitely helped in the sense of trying to find the creative application of technology. That’s something where having an artistic background is very useful. Engineering sometimes focuses heavily on function, but for the end-user, usability matters, and solving for actual user needs really matters. I think we take for granted now that many of our applications are well-designed but 10, 20 years ago, computers were difficult to use, the interfaces were terrible.
You worked for both startups and large companies have been successful in your field. Is there someone who you consider to be your hero or who inspired you on this journey?
Oh wow, that’s a good question! I guess I never really thought about that a lot. I guess I’ve always respected entrepreneurs, there’s not really a specific one, but any story of people who come together and take risks and live in a small apartment with friends. This has been embodied by everyone from Elon Musk to Bill Gates. I just love that entrepreneurial spirit.
Actually, now that I think about it, Bruce Lee is another who always inspired me. He is kind of a cultural icon. He came to the US and developed Jeet Kune Do, his personal martial art and philosophy, and broke away from traditional crafts such as Kung Fu. He had that spirit of going somewhere else, not having friends and family there, and taking a chance. There is a quote from him that sums my approach:
“Research your own experience. Absorb what is useful, reject what is useless, add what is essentially your own.” – Bruce Lee
Sean has designed a similar wheel with quotes from Donald Trump for The New Yorker. 
How should interfaces like Siri and Alexa look like? More on this in Sean’s IDEO article. 
Sean has also worked on improving the UX of foodservice in Peru.
That’s great Sean. You are coming to Košice to speak at the Art & Tech Conference but you’re also bringing you art installation called ‘ERNEST’. How did you come up with it?
I’ve always been a fan of science fiction so this idea that you can have a machine that can see you and know your emotional state has always been fascinating to me. That was sort of the impetus for it. There’s a whole new wave of interfaces that are going to come online in the near future and I love exploring that and artworks are great for that.
Why does it look the way he does?
When I first started getting into technology, the tech looked like ERNEST, it was terminal screens, old keyboards so I think I always had a soft spot for that kind of computer aesthetic.
In addition to that, because ERNEST is exploring AI, facial, and emotional recognition, these topics are often scary for people. Science fiction movies such as Kubrick’s 2001: A Space Odyssey have made technology seem like a threat to humanity. So when we were testing what ERNEST should look like, every time we had something that looked advanced, it was off-putting to people. So by putting it into this retro aesthetic, it actually makes it feel less threatening.
Also, when it comes to systems, from a user experience standpoint, people are more willing to forgive a system like that. ERNEST is very playful, almost like a child, it is not a super-advanced AI even though it has some pretty sophisticated technology built into it. But we think of it as training a child. Because of that, it helps people forgive when it makes mistakes and puts them in a mindset where they want to help.
“ERNEST is very playful, almost like a child, it is not a super-advanced AI even though it has some pretty sophisticated technology built into it. But we think of it as training a child.“
Why should people come and see ERNEST?
For one, it’s fun. If you’re curious to see a machine reading your facial expressions and emotions in real-time, ERNEST will be able to do that. Everyone from young children to old people can use it. It is trying to learn how to make people happy.
What social impact can such tech have in the future?
This is a very important angle. For example, accessibility is one. Voice interfaces or eye-tracking devices could obviously be very useful to people with physical impairments. Solving for those type of challenges doesn’t just help those with impairments, it will make tech easier to use for everyone. This will allow anyone to use this kind of technology.
What ethical concerns arise with technology and designing systems?
AIs are only as good as the training data they have put to them. Once we put the data into them, that also has to be validated. That’s kind of what ERNEST touches upon. It won’t always react correctly, with certain people it might misinterpret them, and that’s indicative of the fact that the training data set maybe wasn’t deep enough to capture all faces and expressions in humanity. For example, I have a colleague who has a big beard and he always wears a hat and glasses and ERNEST has trouble with that because it wasn’t trained on a person looking like that. That’s obviously a problem. So as designers, when working on a learning system that is usable by all, it’s actually the data set we have to think about in addition to the product itself.
How does this link into the topic of this year’s Art & Tech Conference which is Language and Communication?
There is research by Paul Ekman about facial expressions and he found that across all cultures, there are some universal facial expressions like a smile and the sad face. In terms of communication, that is important that there are these universal elements across humanity that makes us more similar than different. There is so much more to communication than just the words, and tech is increasingly aware of these non-verbal factors.
Is that something you will touch upon in your workshop?
Yes, we will definitely be approaching ways in which we can incorporate signals from libraries like poseNet or facial recognition libraries into an art installation or application. Then we’ll do some thought experiments – in the future, when these things become universal, what might it look like?
Why do you want to come to Košice?
I’ve heard about Budapest and Bratislava before but not about Košice. It is fascinating to me that a festival like this is happening here. I was happy to see that Gene Kogan had come before and the fact that it has the title of UNESCO Creative City of Media Arts. Personally, I like this idea of not having art and tech existing only in San Francisco and New York, London and these centres. I was excited to see that so much is already happening in Košice.
On a side note, did you decide to quit your job at some point and move to a farm in Oregon?
There are two parts of the story! My girlfriend, now wife, moved to that area so that influenced my decision. Also, I only lived in big cities and I started to have this sense that not everybody in the world has iPhones and loads of money. San Francisco can feel like a bubble. Part of it was to step out of it and get away from all of that. It helped me balance my perspective.
Excited about coming to Košice?
Yes! I like exploring and not going to big cities only, to explore the region. I’ve never been to Eastern Europe so it’s a fun challenge to explore a new area I’ve never been to. I can’t wait to be there.
Sean gave an overview of how his professional and creative practice has combined technology, design, and storytelling into digital and physical experiences during his talk at the Art & Tech Conference. To meet similar artists and tech experts come to the next Art & Tech Days in Košice.