James Patten: Creating Interactive and Kinetic Experiences

James Patten: Creating Interactive and Kinetic Experiences

Fern Diaz
July 14, 2014

TED Fellow James Patten made a special appearance at Huge Brooklyn to shed light on the groundbreaking work he makes at Patten Studio with clients and partners like Barneys, Bjork, and the Museum of Science and Industry. Patten's creations challenge our conventional notions of interactivity by going beyond the touchscreen, reaching towards a holistic digital experience. Speaking about how he picks his projects, Patten explained, "One of the things I like to do is think:what's really cool that's not gonna get done unless we do it?" Patten also shared some insights into the future of interactivity design with Huge's UX team, and a condensed version of the conversation is below.

Brandon Schmittling, Senior Interaction Designer, Huge: We're really glad to have James Patten here.  I invited him because I feel like his work is such an inspiration and shows where we all need to be going in our own work, and it was really interesting to know that there was somebody doing that right around the corner from us.  Our team is quite diverse here. We come from a lot of different backgrounds and a lot of different educational experiences.  When people ask you what you do, what do you say, and why, and is this where you thought you would end up? 

James Patten: This is where I thought I'd end up.  No…I had no idea where I would end up.  I grew up in a tiny town in the middle of nowhere in Virginia and every morning when I bike over the Manhattan Bridge, I'm kind of like, "What am I doing here?"  But yeah, in terms of how I got here and what I say to people, I say that I use technology to expand the vocabulary of design and I use design to take technology to places that it wouldn't otherwise go.  And I think that that's ultimately the key part of it. A lot of it is about trying to get new ideas out into the world.  

I feel like so much of our world is kind of obsessed with screens — screens you have in your pockets, screens you look out to on the wall, screens on your laptop — and I’m just trying to kind of get away from that because the more time we spend focused on screens, the less time we're actually interacting with each other in a real way. And so to try and get more face-to-face social interaction back into technology, making technology to adapt to us rather than the other way around. 

BS: When you were in Virginia, were you there for college? 

JP: Yeah, I grew up there and I went to the University of Virginia for college. And one day, I stumbled into a virtual reality lab at school and it was kind of like, "Whoa, this is the future.  Like everyone's going to wear these displays."  I started working there and I started to realize that if people wear these things for so long – for too long, they get sick.  Like literally nauseous.  And the reason that happens is because your brain is getting conflicting information about what's going on around you.  Your eyes are telling you that one thing's happening and the rest of your body – your eyes are saying, "I'm flying over this canyon."  And the rest of your body's saying, "You're standing right in the middle of this room."  And so your body's like, "Oh, I must have eaten something bad that's like causing me to hallucinate or something.  So I should probably throw up now so I can get rid of whatever that is." And so it just kind of seems like, Oh, this is probably not the way.  This is not the future. 

And one of the things that we were kind of studying is: if you show somebody a picture and in that picture they're holding something in their hand, then it's not very convincing [that they’re holding something]. But if you give them an object to hold in their hand like a beer bottle and you show them a picture where they're holding a sword, they totally believe it, because all of a sudden, the sense of touch is starting to give you information that matches up with what you're seeing. So, this got me really excited about the sense of touch as a way to interact with technology and that's kind of been like the main focus ever since then. 

JP: I was going to this conference about Cigraph, it's like a big computer graphics industry conference and I saw this thing.  It was called the "inTouch".  There’s two pairs of wooden rollers and they were connected by a telephone line and the idea was that one of these could be in say, LA and the other one could be in New York and they had this connection.  So that however one set of rollers moves, the other set of rollers moves in the exact same way. So, two people touched those rollers and whether they're right next to each other or far apart, it feels like they're touching the same physical object.  And so it's this tactile connection.  I'd never seen anything like this before and I just thought it was such a beautiful response to all this other stuff that was going on at the time. I guess this was maybe like 1998.  So, like computers were big beige boxes and everybody was like, "Oh, there it is, that's how it should look.  It's a big beige box." And that was horrible. And here was this response to [the big beige box], made out of wood and stainless steel and just beautifully crafted. I really responded to the idea that design has a place within consumer electronics in a way that wasn't really recognized at the time, and that we can interact with the sense of touch.  That project, I think to some people, comes dangerously close to feeling like cybersex or something — but I think that the idea is really compelling. The idea of using the sense of touch.  It could be with a romantic partner and it could be just any kind of communication. Like a band of communication that's largely an interaction, that's largely ignored. So that was kind of the 'ding' that made me say, "I have to… like who made this?  I need to like learn from them."  Then it was Hiroshi Ishii at the MIT Media Lab.  So I went to do a PhD with him. 

BS: I want to ask about the Thumbles. For anybody who's not familiar with Thumbles: These are little robots that crawl out onto the LED display and they react by movement and also they exert pressure on you if the video display so chooses. This to me feels a lot like Phase I that we keep hearing about which is like “Smart City” or “Smart Life” where things are analogous to — in small form — to the larger environment.  Do you think we be shooting for that kind of control over our city, control of our environment with small models… or do you think that's even something that's viable? JP: That's an interesting idea. So I think the closest thing we've done so far to this is a dispatcher table where the little robots on a mat represent, say, different police and trucks and it's kind of like, "Oh, there's a fire there."  Okay, so this unit responds and then it goes back to represent where the fire truck is right now, and as the fire truck kind of moves toward the destination, you see its real time location updating on the map. So it's very much like a Sand Table that the military uses where these are all our tanks – they have a person whose job it is to kind of manually update the positions of all of the objects.  But I think that the fact that they're willing still to do this instead of just having a big flat screen that shows where everything is really underscores the value of physical objects. There's so many ways that having physical objects can tap into our mind in a way that stuff on the screen doesn't.  It's worth it to go through the trouble to have physical objects that represent things and control things. So, I mean I'd be interested in talking more about this idea of Smart City because we're developing the next version of Thumbles right now and we're trying to find like: what's the killer app that we're going to develop first? Because we really want to get this idea out there as much as possible.  Because before developing Thumbles, we did a lot of stuff with passive objects on tables.  Where the objects couldn't move and even still there was a lot of value derived from having these physical objects to represent things.  But it really limits the software interactions that you can create when you can't tell an object to move.  Like, say, "Undo," it's like, "Oh, let's put this over here," and there's no way the software can say, "Oh no, you actually can't do that."  So you have to design the software in such a way that anywhere you move an object and anything you do makes sense and not really limits what you can actually do. 

BS: So we have a phone, and we have much smaller pieces of tech emanating from there.  Is there a level at which the fidelity of objects is most useful for controlling interfaces or does it start to break down? 

JP: Well, it's interesting – that's a real interesting question.  I think mobile is like a challenging space for this.  Yeah, I don't know – it's a challenging space for this paradigm of interaction because nobody wants to have a bunch of like little things that they have to always – I've got a bagful of junk that like, you know, I wish I could sort of not carry it around with me. I think that the kind of end game is that these types of – you know, the types of objects and technology, all of this stuff is just in the world around us and so it's not something that we have to think about carrying with us.  But it's something that's there in the sense that WI-FI Hotspots are kind of everywhere now, that new types of interaction and maybe interface devices that will kind of connect to your phone as you walk out to them and give you some sort of interaction with the information that's on your phone.  That's – with an interface, it's more fluid, more powerful than what you have with the phone.  But yeah, for something I think on a phone scale, it becomes – it's tough to apply this paradigm.  I mean I've seen some really beautiful ideas like one – this was like a Speaker project where someone had – just a little thing with a pager motor in it and you could squeeze it and when you squeezed it, there was another one somewhere else in the world that would vibrate when you squeezed yours.  So it was kind of a way to say to someone, "Hey, I'm thinking about you."  But – or you're supposed to have 50 of these for all of the people that you care about?  So, it's kind of – I think that space is pretty challenging. 

BS: I'm going to ask you about installations in just a second, but if you have these physical objects and could position them in the space around, does that imply that screens are going to go away? Do we even need them anymore? 

JP: Oh, well I hope that screens become more interaction surfaces, in terms of having lots of objects on top of them rather than just being like totally flat, clean spaces.  I mean if you think about, like if you have – something like the first version of Microsoft Surface, the kind of coffee table style interaction.  It's kind of hard to imagine you would have like a bunch of beer bottles and snacks and whatever like sort of splayed out over this thing because, "Oh, that's the computer." I think part of this is just kind of changing the way people think about this stuff but also changing the styles of interaction so that our interaction with graphical information is merged more seamlessly with other types of interaction. You have all the different things you might do on a desk ranging from making something, to collaborating with a team, planning something out, you know – like all of these different kinds of tasks and all the tools that they use can fit together, that kind of physical/digital interaction space that is not sort of a screen and everything else separate. 

BS: Are you following a plan for where you're going with all of this? Do you have an ultimate vision or are you just kind of doing experiment after experiment and then connecting back, showing a line of movement towards this future thing? 

JP: I'd say it's a few different like projects that are hopefully...each one is kind of moving closer to a goal. But that goal…I don't think is kind of the end vision or whatever; it's just sort of a really interesting point like on a long progression.  If you think about the graphical user interface…I mean I don't think that Doug Engleberg when he invented the vast majority of that was like, "This is it.  We're done now."  Everyone can kind of stop inventing new things, but I think that it was a really powerful paradigm say for like 30 years.  And so I think this idea that our physical environment is going to change in under software control as a way of interacting with technology.  I think that's an idea that has a few solid decades of like really interesting stuff coming out of it and I think that at the end of that time, people are going to be like, "Oh, okay.  Who knows what it will be at that point," like some sort of nano tech or bio – like some crazy thing that we can't even imagine. 

BS: I wanted to talk with you a little bit about digital installations because people like,  they get them, they create a sense of space and also a sense of unexpectedness and that might be why they love them.  I wanted to ask you, do you think there's anything else in here, a digital installation be that storytelling, value, its monetary value, its ecological value? 

JP: I think one of the most important things just to make it playful and just sort of be like very – very careful. I'd kind of stay away from like a technology arms race.  I mean if you think about things like projectors, LED screens, whatever, it's kind of like, Oh, every year there's one that's like brighter and richer color and higher density and all of that.  And I think that anything that picks a point on one of those timelines is dating itself and in two years, people are going to be like, "Oh, well, that's old."  I think that kind of thinking about technology – appropriating technology and using it in a way that it wasn't really designed to be used — is a way that sort of just skip that whole problem. There's one project that I'll show in my talk where we have these – the screen with pixels that are like – each pixel is like that big.  And so, it's not trying to be this like super high-resolution thing.  A lot of times, it's not really about your interaction with the piece.  It's about the interaction between people who are interacting with the piece.  So, I guess my favorite piece of interactive art anywhere is this – I think it's called "The Kendall Band," it's by Henry Matisse, Matisse's grandson. It's on a subway stop near MIT in Cambridge and it's these – I think they're like 16 different steel cylinders and there's this handle where you can move a hammer back and forth to hit them.  But you know, you have to kind of move it in just the right way in order to get this thing to do anything.  And so, it's something that encourages people to kind of work together to figure out, "Oh, how do you actually work this thing?"  Because it's immediately obvious that, oh you have to move the handle.  But it's not so obvious how you actually have to move the handle and so this is kind of the immediate reward that you get.  "Okay, you're using this thing correctly," or, "At least it's responding to you."  But then it rewards this sustained interaction with playing this beautiful music.  So, I think from my perspective, I love to make things that are playful in that sense that they encourage people to meet each other and interact in unexpected ways. 

BS: As we keep making decisions about where we're putting money into our infrastructure, either building or rebuilding, do you think that digital installation, or digital sculpture will find its way into like the normal course of things and if so, do you see maybe digital installation or digital sculpture being permanent? 

JP: That's a great question.  I don't know.  I mean I think it's really hard to design interactive stuff in general so that you know it's going to like still be there say in 30 years or something.  But that piece that I just talked about, it breaks all the time because even though it's really well designed, with people kind of whaling away at this thing all day, there's just only so much you can do and that doesn't even have any digital technology in it. But if you think about how susceptible most digital technology is to the elements, I think it becomes a real challenge to design stuff that like takes that into account but it's not some sort of armored thing. I do think that there are a lot of ideas in this space that are interesting and it will just take a while for them to kind of percolate up. A friend of mine did a piece which is a – it's a chip. I called it a DSP Chip but it synthesizes music.  And he wrote a program for it so that it will synthesize 30 years of music and never repeat itself and I think it's buried in concrete somewhere in Cambridge and you know, in theory, you can go and plug your headphones into it.  Who knows if that part still works?  But it's just the idea that this thing is going synthesize music for 30 years is – and never repeat itself is a good deal. 

BS: We’re all super concerned about where the web and wearables and consumer tech are converging and I wanted to ask your opinion on that seeing as how you're not – you're more interested in kind of the durability and kind of the functionality of objects as opposed to specific instances like wearables. Do you have an opinion or do you have a view on where this is going? 

JP: You know, I haven't really worked a lot in the web space to be honest.  I mean I think that the notion of web converging of wearables…I guess the first thing that comes to mind is social media, sort of being more in touch with your networks and the like different social stuff. But I guess that's really the first thing that comes to mind. I think it's really challenging because people want to express themselves through fashion and people want to express themselves through their consumer choices – like their devices that they buy and use.  I would expect it would be hard to sell products that have those two kind of modes of expression, kind of seamlessly melted together.  So, I think that that kind of places a limit on how much – like how closely wearables can be connected with clothing and like a lot of friends of mine are working on wearable technology which usually takes the form of like new articles of clothing, not so much about you interacting with information the way you would on like a phone or something like that.  I don't really know where that area's going to go.  

BS: It may be too fresh because it's – we're just now cutting edge of wearable stuff. 

JP: But it's interesting for things like the Up-Band and the – what's it called – the Fit bit and this kind of stuff. I think like 90% of people who buy these devices are not using them within like a very short – I forget what the statistic is – like a month or something.  I don't know if that means that the devices just are frustrating to use or not rewarding enough or if it's just a matter of someone hasn't gotten it right or is it something that's more fundamental?  I don't know. 

BS: Well, I've got some easy questions for you that I think the group really, really wants to know.  Can you tell us – for somebody who wants to move closer to what you do which is arguably a more engineering rigor, or if you want to supplement our knowledge already cause we come from kind of a web centric world, what are some tools, what's some knowledge maybe even what's some reading that we absolutely must know and do? 

JP: Well, I mean I think that like there's really so much to learn in terms of making things it’s really beyond the grasp of any one person or group really.  But I think there's this moment of kind of like frustration and release where you kind of realize, "I'm never going to be able to know all this stuff."  You just have to let it go and sort of decide this is your frontier, this is my frontier.  Like anything past that, I'm just going to ask for help. But I mean there's the electronic side. I think a great first book to learn about this stuff is Practical Electronics for Inventors. It explains how electronics work using water as an analogy. Electricity flowing through wires is water flowing through tubes. And then there's the mechanical side which is very closely connected with the product design side.  I think a great book in that world is on Growth and Form. I mean the mechanical engineering side is much less my area so I wouldn't even – but let's see, The Machinist's Handbook is a great book.  That’s kind of more advanced.  And then programming – I mean that's probably much closer to your comfort zone. 

BS: That's your area too right? 

JP: Yeah, so, there are lots of books on processing, this kind of thing.  I feel like a good solid foundation for that is like C or C++ and kind of work up from there.  But a lot of people will start something like processing. So I think that the most powerful approach is to try to get good at more than one of those areas and then surround yourself with other people who are also good at more than one of those areas because I think that really, the interesting things happen at the boundaries between the different areas. 

BS: What about some of the things that you can't live and just anything that you love to work with?  Anything that you can't get enough of?  What helps you do your work? 

JP: Let's see, favorite tools – so I guess one of my favorite tools is an oscilloscope.  It's like just a little – it's very similar to a computer but it just shows you an electrical signal over time as a line, like a graph.  But it's something that constantly updates as you work on a circuit.  And a friend of mine designed one that will change electrical signals into sound.  So sometimes, it's much easier to kind of listen to what a circuit's doing rather than seeing what it's doing. Like when we were working on the gravity harps, they're like 25 feet in the air and it was getting this weird interference and I couldn't figure out what it was and I stuck this thing on it and I heard a radio station.  I was like, "Oh, okay."  And then suddenly, it was like, "Oh well," and on the oscilloscope, it would have just looked like a bunch of noise. We have a lot of metal-like, manual metalworking tools like a Bridgeport Mill, Lathe, and different kinds of welders.  These are great for – you know, you have an idea.  You need to very quickly turn it into an object that you can see if your idea's any good or not.  What else do we have? Three-D Printers, yeah, but even still – I still spend the vast majority of my day in front of my Mac which I wish that weren't true but that's how it is. 

BS: Actually we just kind of talked of this in the beginning, so we're coming full-circle.  What motivates you and what was something that you were inspired by?  Maybe to work on a recent project or maybe submitted your plan. 

JP: One of the things I'm really motivated right – right now, one of the things I'm really excited about is the Thumbles, like trying to get this idea out into the world.  I feel like it's really close to getting the technology to a point where it's ready to sort of have other people play with it.  I think that's – that's something I'm really excited about. In terms of inspiration, one of the things I really find inspiring is – so our company is in a shared space with a bunch of other similarly-sized companies and so there's like really amazing work going on around us constantly.  So it's really inspiring to see some of the stuff take shape.  Like a friend of mine, Andy, actually the guy who's working on the Gravity Harp, he's building something now where he's using electromagnets to play a piano.  Where, you know, the strings are made of steel so they're magnetic.  So, by pulsing them with magnets, you can get some sounds out of them that you can't actually get by hitting them with a hammer, like a piano normally does.  And so just sort of being around someone like Andy, lots of people like Andy that we're really lucky to be around on a daily basis and sort of seeing these like crazy ideas that they come up with and sort of see them being realized.  I think it's super inspiring and it kind of makes me feel, "Wow, I've got to up my game here."  So much cool stuff going on.

Read This.