The Art of Human Computer Interface Design,
B. Laurel, editor,
NY: Addison Wesley, 1990

Realness and Interactivity

Michael Naimark

Legend has it that a gentleman once approached Picasso on the street and criticized his paintings as distorting reality. Seeming to change the subject, the artist asked the gentleman if he had a girlfriend. He did, and produced a small picture of her from his wallet. "She's beautiful," replied Picasso, "but she's so tiny."

This paper is about how reality is a funny thing, now that we are beginning to simulate it, and where the clues may lie in understanding styles of function and aesthetics about virtual realities.

Virtual Realities Require Realness And Interactivity

Realness and interactivity have various meanings. We usually use the term "realness" to mean our sense-ability: what our senses sense. This "sense of presence" is independent of whether the material is derived from the real world or from a fantasy based one (perhaps "thereness" is a more appropriate word than "realness"). The term "interactivity" usually means our effect-ability: what our effectors affect. Though interactivity always requires information flowing in both directions, it is our input and its effect that distinguishes it from non-interactivity, where the flow is always one-way.

The ultimate virtual reality is, by definition, indistinguishable from reality, where all our effectors are sensed and all our sensors are affected. This is a rather formidable task. Such models exist in the fantasy literature: the family media room in Bradbury's Veldt and cyberspace in Gibson's Neuromancer are popular examples of total immersion in virtual realities (and most certainly include created worlds as well as real ones). Other models, theoretical ones, have been proposed from the points of view of media technology (Negroponte 1981), cybernetic theory (Bricken 1983), and drama (Laurel 1986). The combination of realness with interactivity is a new frontier where experienced pioneers today are few, and come from such diverse worlds as flight simulation, dream research, and experimental theater.

Clearly there is an attraction to experiencing the jaw-dropping impact of a seven story high IMAX movie controlled by a video game-style steering wheel (which is very much what present-day flight simulators really do). Or having a conversation with a Disney-style talking head interacting in a way that dwarfs the ELIZA psychoanalyst program (which very much doesn't exist today in any form of media). We can expect such progress on the technological front, but there is a bit of a communication problem.

The sense-ability experts and the effect-ability experts come from different worlds. And their uses of each others' terms are grossly miscalibrated.

The Movie World Understands Realness But Not Interactivity

If you want a mediated experience to look and sound (and feel and smell) real, go to the motion picture industry. From the beginning, the trend has been toward better sound, color, scale, depth, spatial resolution, and temporal resolution. Today we have various "enhanced-format" theaters at theme parks and worlds' fairs offering "more reality." These include big movies like IMAX and OMNIMAX (horizontally formatted 70mm film), panoramic movies like CircleVision (nine 35mm screens arranged in a cylinder), deep movies like Twin-70mm stereo (two 70mm projectors and polarized glasses), and smooth movies like Showscan (70mm film shot and projected at 60 frames per second).

There are other tricks known by the motion picture industry to achieve even more reality. The use of a concave mirror to achieve infinity focus is used in today's flight simulators to allow the users' eyes to focus at infinity, as they normally do when viewing landscapes. Disney pioneered relief projection, projecting movies on a screen the same shape as the image, such as the "talking heads" in the Haunted Mansion. He also pioneered the use of a single room-size half-silvered mirror to produce 3D ghost-like images seen in the Haunted Mansion's ballroom. Lucasfilm recently introduced the use of a motion platform synchronized with image and sound in their "Star Tours" ride giving the audience a parallel kinesthetic sensory input channel.

What has been restricted to the film world has begun to enter the video world. High definition television, with its various and opposing formats, has forced some of these parameters to be surveyed and formalized (Schrieber 1984).

So when people from the motion picture industry say they can give you a sense of reality, they can, as they say, knock your socks off. But if they say they can give you an interactive experience, don't come running.

The world's first interactive movie (as it was billed) was produced by the Czechs for EXPO '67 in Montreal. The audience had "yes/no" buttons and every few minutes they were given a choice. This database of movie scenes was so sparse that after the end, the audience was asked if they would like to see the other end ("vote yes or no"), which acknowledged what the experience really was: a clever and perhaps prophetic gimmick. (Incidently it was more clever and less technological than the audience may have imagined. Each choice was carefully structured so that either option would end back at the same next choice, where the audience vote signalled the projectionist on which of the two synchronized projectors to put the lens cap.)

Today most of the interactivity known to the motion picture industry is in the form of interactive videodiscs, since they share common production hardware and techniques with conventional cinema. Most interactive videodisc applications are poke-and-see movie snippets with just slightly more options than the Czech's film. In a Sony advertisement a few years ago, a Cuisinart kiosk was shown with a monitor and ten pushbuttons numbered one through ten, presumably allowing the user one of ten short linear programs. (Sardonically, the words on the screen flashed "PUSH BUTTON SIX." One can't help but wonder who's controlling who here.)

You can't conduct a symphony orchestra, or carry on a rich conversation, or drive a car this way.

The Computer World Understands Interactivity But Not Realness

If you want an interactive mediated experience, go to the computer world. But since it is younger and less evolved than the world of cinema, it has addressed interactivity more in the context of here-and-now practicalities than in the context of future sensory-rich virtual realities, where theoretical factors such as simultaneity, memory, and interruptability become essential (Lippman 1985).

All interactive media experiences require user input. With a little imagination, one can envision human-machine interaction beyond a keyboard and mouse to the natural and kinesthetic way we encounter the real world. Input devices for our hands, arms, head, eyes, body, and feet can sense position, gesture, touch, movement, and balance.

Video games are primitive examples of this sort of interactivity. Driving games such as "Pole Position" offer user control as tight linked as real world power steering. Even the first arcade video game, "Pong," had an input channel many times the bandwidth of practically any "interactive" videodisc system today.

But don't ask computer people about a sense of media reality: their perceptions are warped by current hardware constraints.

Go to any computer graphics film show and listen when the audience applauds. A few years ago I was at the annual Siggraph film show, the largest of these events, standing next to a young security guard in a mobbed colosseum. After a couple minutes a shot of a (fractal) mountain scene appeared on the screen and the audience applauded. I noticed the security guard getting tense. Shortly after that, (ray traced) shiney steel balls were shown hovering over a chessboard. More applause. Then a (atmospheric) foggy outdoor forest scene appeared and the audience exploded. At this point the security guard snapped. He grabbed my arm ("something weird is going on here man") and demanded an explanation.

More recently I attended a demo of a new chip capable of unlimited realtime movement through a 3D database of limited cartoon-like quality. A very excited representative of a well-known computer maker kept exclaiming how real this was, and continued to refer to the little green cones as "trees."

Who're we kidding?

Bandwidth Is Not Enough

But if virtual reality-making follows moviemaking (or any other media form), then throwing more bandwidth at the medium doesn't automatically make it better. Being artful with what you have does.

In early motion pictures, shots were long in duration, camera movement, if any, was slow, and scenes were edited for perceptual continuity. Audiences were cautiously led through the virtual space of cinema, careful not to get "lost." As the language of cinema evolved and audiences grew more sophisticated, more inferences could be made. Conceptual continuity replaced perceptual continuity. One might speculate that the Lucas/Spielberg-style fast-paced thriller would have been impossible for early movie watchers to have followed. The evolution of aesthetic style results in massive bandwidth compression.

We have little basis for aesthetic style in virtual realities yet, and we have much to learn. For starters, we still have only a meager understanding of the use of scale in media. Nor do we have more than a hint of a cinematic language of interactivity. We have little understanding of the roles of acculturation and habituation in media. And we have no experience at all in creating behavior as well as image.

We do know that these media issues have something to do with context and its relationship to medium (remember McLuhan), as we are creating media forms with vast new capabilites. This has created a particularly critical role for artists today, if we are to believe that artists of the future will create landscapes that you walk through and portraits that you converse with.

References

Bricken, W. (1983) A Model Interface Model. Atari Research Memo #XX. Sunnyvale: Atari Sunnyvale Research Laboratory.

Laurel, B.K. (1986) Toward The Design Of A Computer-Based Interactive Fantasy System (PhD dissertation). Department of Theater, Ohio State University.

Lippman, A. (1985) Imaging and Interactivity. Media Lab report. Cambridge: Massachusetts Institute of Technology.

Negroponte, N. (1981) Media Room. SID 22, No. 2, 109-113.

Schrieber, W. (1984) Psychophysics and the Improvement of Television Image Quality. SMPTE 93, No. 8, 717-725.