The Post-Screen User Interface.

What emerging interaction models mean for design.

Emily Atwater
March 31, 2015

No matter how interactive or dynamic, our digital experiences are limited by a constraint so fundamental that we barely think about it: the screen. In many ways, our devices have become extensions of ourselves—but the screen remains a boundary between digital and physical realities, limiting the value devices can deliver to users. 

Soon, post-screen user interfaces will make this boundary porous. Recent advances in user interface design are fueling this shift: we began with the Graphical User Interface (GUI), which allows us to manipulate information on screens with a mouse or keyboard. Today, the Tangible User Interface (TUI), better known as the touch screen, allows us to interact physically with digital information — a capability we have come to expect. 

On the horizon is the ability to merge TUI with holographic augmented reality (AR) to create a new kind of experience: Tangible Augmented Reality (TAR). TAR isn’t necessarily about a new must-have device (not yet, anyway); its potential exists at the crossroads of wearables, gesture control, and projection technology. And the field is open for brands to redefine themselves by enabling new kinds of experiences. 


Photo: Microsoft

TAR today.

Several major academic players, including MIT and Carnegie Mellon, are at the forefront of TAR research and development and may become players in the marketplace. Meanwhile, tech companies are forging ahead. Development of Microsoft’s HoloLens is well underway, and the Google-backed Magic Leap has the tech world buzzing. Sony and Facebook also have AR projects in the works, while other companies are developing retail products with TAR capabilities, like wearables that turn the user’s skin into a touchscreen. 

In order to better serve users, we need to anticipate how these technologies will inform new interaction models, which will shape the future of communication and storytelling. Here are some ways that augmented reality might come into play. 

Innovative experiential marketing. 

A realistic augmented or virtual reality experience could mean fewer limitations for experiential marketing campaigns in terms of cost, space, and — most importantly — concept. Imagine, for example, test “driving” a new car with a combination of physical elements (sitting in the car itself, pressing the gas and feeling the engine hum) and digital ones (a windshield that is augmented to display a driving route that reacts to the user’s steering). One simple way AR could affect experiential marketing is through the use of holographic signage, reducing the cost of materials and manufacturing for temporary installations. 

Education & training applications. 

Augmented reality applications are already being used in classroom and learning environments, and three-dimensional hands-on teaching methods like model-building aren’t new, but the possibilities are even greater for tangible augmented reality if we could combine these methods to harness spatial memory and spatial organization for improved recall. 

Augmented reality & the Internet of Things. 

As the volume of user data increases, so too does the potential for establishing meaningful connections among people, places, and things using “smart” objects. In the future, real-time, context-specific, user-specific data could enable augmented reality visualizations that are personalized for the user. For example, in conjunction with a fitness wearable, a runner might see virtual mileposts or street signs that display arrows guiding her along a predetermined route. The trick, of course, will be creating experiences that are meaningful and useful, not adding more noise to our surroundings. 


Photo: Google

While the ultimate potential of TAR is yet to be seen, brands can start preparing now to take advantage of this emerging technology. These are our best practices for TAR today: 

Develop potential use cases and start designing for them. 

This technology’s applications will vary by industry. From healthcare to home entertainment, we need to hypothesize possible use cases and be ready to test our assumptions as new products become possible. UX designers should start designing for 3D interactions now—and that means saying goodbye to static wireframes and embracing prototyping, using tools like CAD or animation software for 3D design. 

Implement standards, like a taxonomy of user interactions. 

There is a lack of standardization around AR (although OASIS, the Organization for the Advancement of Structured Information Standards, is trying to change that). One consequence is that terminology around AR is still ambiguous. For example, “Tangible User Interface” encompasses a broad range of interaction models with no taxonomy to differentiate them: touch, gesture, screen-to-screen, and so on. Establishing a clear language will be invaluable in defining new interactions, whether they fall under the as-yet-hypothetical TAR or something else entirely. To adapt to the changes these technologies will enable, we have to know we’re all talking about the same thing. 

Do your research. 

Remember: plenty of amazing tools are in development, but there’s also lots of competition out there. The impact of any one invention is hard to sustain, let alone predict. Even though TUI and AR have been around for years, the practical applications of these technologies have yet to change our everyday lives. 

Pay attention to technology flops as well as successes; awareness is key to helping clients implement strategies that hinge on emerging technologies successfully. 

Marry content, design, and technology. 

Making great digital experiences requires close interdisciplinary collaboration. We need information strategists to model data for optimized AR experiences; we need interaction designers to create intuitive user flows; we need developers and engineers to be familiar with the implications of our designs and content models. This is nothing new, but will become more important as Tangible Augmented Reality becomes, well… a reality.