Background

Input devices are a classic topic in human-computer interaction.

Continuous actions can be divided into two classes(Gibson, 1979).

In the HCI context, performative actions have been studied quantitatively. Fitts law (1954) can be considered the "Law of Pointing". Pointing was considered as a discrete act performed using a continuous device, or like sending a message through a communication channel (in Shannon's terms). Pointing as a discrete act has been exploited in Graphical User Interfaces as well as in speech-gesture interaction (Bolt 1980). That this is commonly believed to be the fundamental paradigm for "intuitive" interaction is testified by its wide use in the movie Minority Report. However, since the nineties some people started to look at continuous gestures in GUIs, like crossing targets or traversing a hierarchical cascading menus, and tried to model them as limits of sequences of discrete gestures. The steering law was derived. Some other people looked at what is in between starting position and final target, and discovered that there are kinematic patterns in goal-directed movements (see Figure 1, from Bootsma, Fernandez, and Mottet, 2004).

Figure 1

Phase trajectories for goal-directed movement

Before industrial revolution, most of human actions in the world were essentially continuous. Supposedly, continuous actions and gestures are more "natural" than triggers. Naturalness here means that control is left to the human manipulator rather than transferred to some machinery. According to the tightness of sensory feedback to the handle, control can be more or less direct/physical. Example: sailing using the tiller or the wheel to control the rudder. In the latter there is a decoupling that allows application of smaller forces. Any potentiometer relies on a (abstract) mental model: a map. It is not a very natural kind of interaction. Examples: audio mixer, fires in the kitchen. Our physical (mechanical) actions control changes in non-mechanical (acoustic, thermal, electromagnetic, etc.) energy whose display is displaced from the locus of action.

Triggers can elicit sustained feedback, and the perceived behavior is sometimes that of autonomous life. Conversely, Enaction is based on motor skills. In the closed loop between perception and action the cause-effect relation breaks down.

When continuous action is relevant, it is important to understand the "more subtle characteristics about the way a movement is done with respect to inner intention. The difference between punching someone in anger and reaching for a glass is slight in terms of body organization - both rely on extension of the arm." ( Laban movement analysis is a useful tool). Continuous action conveys an emotional content which can be organized in terms of kinematic variables and patterns. "Etymologically, the word emotion is a composite formed from two Latin words. e(x)/out, outward + motio/movement, action, gesture."

Human-Computer Interfaces

Token+Constraint systems for tangible interaction with digital information (Tangible User Interfaces - TUI) have no notion of function embodied in the objects. They rather stand on: (i) objects as representations (of information), and (ii) physical manipulation. In TUIs, tokens are container+control. Constraints are used to reduce dimensionality and guide the actions.

Figure 2

MVC and MCRit interaction models

Definition 1: MVC

GUIs obey to the Model-View-Control scheme (see Figure 2)

Definition 2: MCRit

TUIs obey to the Model-Control-Representation (intangible and tangible) scheme

Shneiderman's principles of direct manipulation:

Embodiment

Indeed, the direct manipulation of GUIs has a level of indirection. Feedback is not where the action is. Conversely, embodied interaction (à la Dourish) tends to be direct and physical.

A disembodied interface, as most of existing interfaces are, gives a schizoid perception and action in the world. This is related to the concept of schizophonia, coined by Murray Schafer. He has been reported to say that "It shouldn't be allowed to have sounds without knowing where they come from, so that you can destroy the source if you don't like it". Indeed, distruction seems to be the most compelling outcome of large-scale marketing of a partially-embodied interface such as the Nintendo Wii remote.

The case of Music

The emergence of Schizophonia

• Pre-recorded media and loudspeakers (20th century)
• Electroacoustic music (1948)
• Computer music (60’s): form of "non-instrumental composition" that breaks the link between physical human movement and music making.

Belá Bartók, The mechanical music, 1937 "The final source of any sound, and thus of the musical sound, is a vibrating body. […] So, the less foreign bodies are interposing themselves between the human body and the vibrating body or, the longest the time during which the human body controls the vibration is, the more the created musical sound will be immediate and, so to speak, human."

Musical performance: a guideline for (continuous) interaction

Figure 3

Space-Movement-Matter-Sound

Matter and space form also part of the interface between movement and sound. In the same manner, movement and sound form also part of the interface between matter and space. Musical instruments are, thus, situated at the intersection of the two interfaces.

Figure 4

The architecture of an acoustic instrument

Traditional acoustic instruments are characterized by an extraordinary integration between the three main parts of the sound production chain: the input interface, the filtering section and the output interface.

Figure 5

The architecture of a Hyper-instrument (HI)

In an electronic musical instrument, each stage is split into separate units. Therefore, it is important to fully integrate the different parts in order to recreate the unified functionality of traditional instruments. On the other side: The great interest of splitting the different stages is to significantly expand and develop the potentialities found in each of them.

The risks are significant:

• HI's separate the input control interface from the sound generation -- risk of developing poor mapping strategies
• Break down of the perceptual linkage between the physical action and the musical reaction

However, by means of a proper use of electronic sensors and custom interfaces, together with the computer, one can recreate the link between human gestures and music in the context of computer music. In this case the novel possibilities introduced by HI's are many:

• An hyper-instrument can take any shape, size or form. For instance, it could occupy a large space. It could be divided into individual parts, which together form a kind of network.
• Also, the body of the interface can be a source of symbolic content and convey extramusical, meaning.
• Furthermore, an HI can be designed to require absolutely no previous training or practice, or it can require as much sophisticated skill as playing the violin ("virtuoso").

Some examples

1. Live-electronics (Stockhausen, Nono,…): a traditional instrument played by a performer on the stage, expanded and transformed in real time by means of a computer. Good compromise! Performance + technology, using already existing and well known "interfaces".

Figure 6: The scheme of the setup for the performance of "A Pierre, Dell'azzurro silenzio, inquietum (1985)" by Luigi Nono for Counterbass Clarinet, Counterbass Flute and Live Electronics.

"A Pierre", Live Electronics scheme.png

2. "Manipulating" a sound. What does it mean:

• "feeling" a material, or
• "feeling" the form of an object or
• "feeling" its dimensions

when one plays an instrument (object)?

• The Squeezable is a group of six soft balls filled with jello like material (see also Weinberg and others). Applications include:
  • sound sculpting (shaping sound by physically shaping objects)
  • sequence manipulation (grooving)
• The Pebblebox "exploits the tacet knowledge of the behaviour of physical systems with well understood auditory and haptic percepts".

3. The ReacTable, an example of TUI (Tangible User Interface). Objects as symbolic representation of sounds (see a Demo).

4. Expressive gesture control

• The radio baton by Max Mathews (1997) "The Radio-Baton operates in two principal modes – the Conductor Mode and the Improv Mode. In the Conductor Mode, the Radio-Baton simulates an orchestra. The musician loads a score of the piece to be played from a Computer into the Radio-Baton. She then uses one baton to beat time and thus to control the tempo of the performance and the other baton to control the dynamics, balance, and timbres of the voices. In the Improv Mode, the Radio-Baton serves as a simple controller which sends triggers and the x,y,z positions of the two batons to a computer. The musician must write a program in the computer to interpret this information and to send MIDI commands to play music on a synthesizer. [...] It is a more general mode than the Conductor Mode, but it requires that the musician write the complete program to make a musical interpretation of her gestures."
• The Theremin by Leon Theremin (1920)

Approaches in Design

The controllable dimensions are many. Sensors and actuators should be considered together. How can we tackle this complexity? One possibility is to think in terms of basic phenomena, constructively. So, we should look for fundamental interaction gestalts (Svanaes, Understanding Interactivity, 1999) that we exploit in "natural" interactions. Examples in perception to be inspired from:

Interaction gestalts may result from abstraction of actual interactions, in the spirit of the Ramsauer drawing tutor (after Franinovic and Visell, at the HGKZ workshop on Sound in Interaction).

Figure 7

The Ramsauer drawing tutor (1821)

How can we proceed in the interactive realm? By constructing workbenches or design patterns. A tentative list of workbenches:

Bellotti et al raised five questions for sensing-based interaction.

Another problem is how to provide invisible affordances. If my tangible object is a bottle, can it emit a sound that makes me thirsty and induces me to drink from it?

Kandinsky said "Line is a track made by the moving point". We may say that "Sound is a pressure signal made by interactions with and between objects".