Background Input devices are a classic topic in human-computer interaction. Bill Buxton's unfinished book manuscript: Human Input to Computer Systems: Theories, Techniques and Technology. Bill Verplank's classic lecture on interaction design on paper and in video . Handles Buttons continuous discrete

Continuous actions can be divided into two classes(Gibson, 1979). Explorative Performative Uncover information Achieve, Express

In the HCI context, performative actions have been studied quantitatively. Fitts law (1954) can be considered the "Law of Pointing". Pointing was considered as a discrete act performed using a continuous device, or like sending a message through a communication channel (in Shannon's terms). Pointing as a discrete act has been exploited in Graphical User Interfaces as well as in speech-gesture interaction (Bolt 1980). That this is commonly believed to be the fundamental paradigm for "intuitive" interaction is testified by its wide use in the movie Minority Report. However, since the nineties some people started to look at continuous gestures in GUIs, like crossing targets or traversing a hierarchical cascading menus, and tried to model them as limits of sequences of discrete gestures. The steering law was derived. Some other people looked at what is in between starting position and final target, and discovered that there are kinematic patterns in goal-directed movements (see , from Bootsma, Fernandez, and Mottet, 2004).

Phase trajectories for goal-directed movement So, there is much more behind point-like acts. See for example preparatory movements in drumming. Before industrial revolution, most of human actions in the world were essentially continuous. Supposedly, continuous actions and gestures are more "natural" than triggers. Naturalness here means that control is left to the human manipulator rather than transferred to some machinery. According to the tightness of sensory feedback to the handle, control can be more or less direct/physical. Example: sailing using the tiller or the wheel to control the rudder. In the latter there is a decoupling that allows application of smaller forces. Any potentiometer relies on a (abstract) mental model: a map. It is not a very natural kind of interaction. Examples: audio mixer, fires in the kitchen. Our physical (mechanical) actions control changes in non-mechanical (acoustic, thermal, electromagnetic, etc.) energy whose display is displaced from the locus of action. Triggers can elicit sustained feedback, and the perceived behavior is sometimes that of autonomous life. Conversely, Enaction is based on motor skills. In the closed loop between perception and action the cause-effect relation breaks down. When continuous action is relevant, it is important to understand the "more subtle characteristics about the way a movement is done with respect to inner intention. The difference between punching someone in anger and reaching for a glass is slight in terms of body organization - both rely on extension of the arm." ( Laban movement analysis is a useful tool). Continuous action conveys an emotional content which can be organized in terms of kinematic variables and patterns. "Etymologically, the word emotion is a composite formed from two Latin words. e(x)/out, outward + motio/movement, action, gesture." The circumplex of emotions in (gestural) product design

Human-Computer Interfaces Token+Constraint systems for tangible interaction with digital information (Tangible User Interfaces - TUI) have no notion of function embodied in the objects. They rather stand on: (i) objects as representations (of information), and (ii) physical manipulation. In TUIs, tokens are container+control. Constraints are used to reduce dimensionality and guide the actions.

MVC and MCRit interaction models MVC GUIs obey to the Model-View-Control scheme (see ) MCRit TUIs obey to the Model-Control-Representation (intangible and tangible) scheme Shneiderman's principles of direct manipulation: Continuous (persistent) representation of the object of interest Physical actions or labelled buttons instead of complex syntax Rapid incremental reversible operations whose impact on the object of interest is immediately visible.

Embodiment Indeed, the direct manipulation of GUIs has a level of indirection. Feedback is not where the action is. Conversely, embodied interaction (à la Dourish) tends to be direct and physical. A disembodied interface, as most of existing interfaces are, gives a schizoid perception and action in the world. This is related to the concept of schizophonia, coined by Murray Schafer. He has been reported to say that "It shouldn't be allowed to have sounds without knowing where they come from, so that you can destroy the source if you don't like it". Indeed, distruction seems to be the most compelling outcome of large-scale marketing of a partially-embodied interface such as the Nintendo Wii remote.

The case of Music The emergence of Schizophonia Pre-recorded media and loudspeakers (20th century) Electroacoustic music (1948) Computer music (60’s): form of "non-instrumental composition" that breaks the link between physical human movement and music making. Belá Bartók, The mechanical music, 1937 "The final source of any sound, and thus of the musical sound, is a vibrating body. […] So, the less foreign bodies are interposing themselves between the human body and the vibrating body or, the longest the time during which the human body controls the vibration is, the more the created musical sound will be immediate and, so to speak, human."Musical performance: a guideline for (continuous) interaction

Space-Movement-Matter-Sound Matter and space form also part of the interface between movement and sound. In the same manner, movement and sound form also part of the interface between matter and space. Musical instruments are, thus, situated at the intersection of the two interfaces.

The architecture of an acoustic instrument Traditional acoustic instruments are characterized by an extraordinary integration between the three main parts of the sound production chain: the input interface, the filtering section and the output interface.

The architecture of a Hyper-instrument (HI) In an electronic musical instrument, each stage is split into separate units. Therefore, it is important to fully integrate the different parts in order to recreate the unified functionality of traditional instruments. On the other side: The great interest of splitting the different stages is to significantly expand and develop the potentialities found in each of them. The risks are significant: HI's separate the input control interface from the sound generation -- risk of developing poor mapping strategies Break down of the perceptual linkage between the physical action and the musical reaction However, by means of a proper use of electronic sensors and custom interfaces, together with the computer, one can recreate the link between human gestures and music in the context of computer music. In this case the novel possibilities introduced by HI's are many: An hyper-instrument can take any shape, size or form. For instance, it could occupy a large space. It could be divided into individual parts, which together form a kind of network. Also, the body of the interface can be a source of symbolic content and convey extramusical, meaning. Furthermore, an HI can be designed to require absolutely no previous training or practice, or it can require as much sophisticated skill as playing the violin ("virtuoso"). Some examples1. Live-electronics (Stockhausen, Nono,…): a traditional instrument played by a performer on the stage, expanded and transformed in real time by means of a computer. Good compromise! Performance + technology, using already existing and well known "interfaces".

"A Pierre", Live Electronics scheme.png The scheme of the setup for the performance of "A Pierre, Dell'azzurro silenzio, inquietum (1985)" by Luigi Nono for Counterbass Clarinet, Counterbass Flute and Live Electronics.2. "Manipulating" a sound. What does it mean: "feeling" a material, or "feeling" the form of an object or "feeling" its dimensions when one plays an instrument (object)? The Squeezable is a group of six soft balls filled with jello like material (see also Weinberg and others). Applications include: sound sculpting (shaping sound by physically shaping objects) sequence manipulation (grooving) The Pebblebox "exploits the tacet knowledge of the behaviour of physical systems with well understood auditory and haptic percepts". 3. The ReacTable, an example of TUI (Tangible User Interface). Objects as symbolic representation of sounds (see a Demo).4. Expressive gesture control The radio baton by Max Mathews (1997) "The Radio-Baton operates in two principal modes – the Conductor Mode and the Improv Mode. In the Conductor Mode, the Radio-Baton simulates an orchestra. The musician loads a score of the piece to be played from a Computer into the Radio-Baton. She then uses one baton to beat time and thus to control the tempo of the performance and the other baton to control the dynamics, balance, and timbres of the voices. In the Improv Mode, the Radio-Baton serves as a simple controller which sends triggers and the x,y,z positions of the two batons to a computer. The musician must write a program in the computer to interpret this information and to send MIDI commands to play music on a synthesizer. [...] It is a more general mode than the Conductor Mode, but it requires that the musician write the complete program to make a musical interpretation of her gestures." The Theremin by Leon Theremin (1920)

Tangible Acoustic Interfaces (TAI) PingPongPlus (Ishii and others, 1999) The goals are: "to demonstrate an instance of an athletic-tangible interface, developed on top of existing skills and protocols of familiar competitive/cooperative play." "to develop an underlying technology for an "interactive architectural surface" which can track the activities happening on the surface." Passive Acoustic Sensing (Paradiso and others, 2002) A New Musical Interface (Crevoisier and Polotti, 2001) based on tangible-acoustic interaction Idea: Use acoustic signals generated by mechanical:interaction as control signals From: “No limit in the physical design of the input interface” To: “No limit in the choice of any object as input interface” More specifically, one can think of transforming physical objects, flat or complex surfaces and walls into natural seamless unrestricted touch interfaces.The main idea in TAI is to exploit the natural "nervous system" of any solid object, i.e. its capacity to transmit the "feeling" of any interaction with another object in the form of acoustic waves. The transmitted signal can be delivered to the "brain" of the TAI itself (a computer) through a transducer as a piezo-electric microphoneThe simplest approach consists in augmenting the acoustic response (sonic feedback) of an object via coupled systems of piezoelectric/micro-loudspeakers. More interesting: use the acoustic waves as a control signal. The simplest approach is given by threshold and position (an example of an already existing product on the market is the Touch Screen by the 3M). Even more interesting: becoming able to interpret (decode) the expressive contents of a contact between a human (hand) and an object: not only triggers, (impacts) but also scratching, rubbing, caressing, pressing... In the musical case, the purpose of TAI's is to re-unify the role of the input interface as generator of acoustic energy produced by the movement of the performer and controller of sound output at the same time. Use the acoustic vibrations of the interface to generate control information Use the sound generated by the interaction as sound source Once more the musical metaphor fits: Throughout music history, traditional musical instruments have been the means of transforming the physical movements of a musician (or a dancer) into musical sounds. Music as “sonification of gesture”: musical composition has implicitly been a process of composing and directing physical human gestures on a musical instrument

Approaches in Design The controllable dimensions are many. Sensors and actuators should be considered together. How can we tackle this complexity? One possibility is to think in terms of basic phenomena, constructively. So, we should look for fundamental interaction gestalts (Svanaes, Understanding Interactivity, 1999) that we exploit in "natural" interactions. Examples in perception to be inspired from: figure/ground (in vision and in music) good continuation, streaming transparency (à la Albers o à la Metelli) pre-attentive features (Healey) Interaction gestalts may result from abstraction of actual interactions, in the spirit of the Ramsauer drawing tutor (after Franinovic and Visell, at the HGKZ workshop on Sound in Interaction).

The Ramsauer drawing tutor (1821) How can we proceed in the interactive realm? By constructing workbenches or design patterns. A tentative list of workbenches: Equilibrium (as in the the ballancer) Effort, resistance Warm/Cold Affectionate touch (attractive, repellent) Ensembles of objects (patterns, waves, activity) textures (resistant to spatial or temporal inversion) and their continuous exploration extending into depth, proximity, perspective constraints (visual, mechanical, acoustic) volume inflation/deflation Specific problems should help finding more gestalts. Any hints from designers? Bellotti et al raised five questions for sensing-based interaction. When I address a system, how does it know I am addressing it? When I ask a system to do something how do I know it is attending? When I issue a command (such as save, execute delete), how does the system know what it relates to? How do I know the system understands my command and is correctly executing my intended action? How do I recover from mistakes? Another problem is how to provide invisible affordances. If my tangible object is a bottle, can it emit a sound that makes me thirsty and induces me to drink from it? Kandinsky said "Line is a track made by the moving point". We may say that "Sound is a pressure signal made by interactions with and between objects".