Enactive perception

Action In Perception

I just finished Action In Perception by Alva Noë. It’s a very readable introduction to the enactive view of perceptual consciousness, which argues that perception neither happens in us nor to us; rather, it’s something we do with our bodies, situated in the physical world, over time. Our knowledge of the way in which sensory stimulation varies as we control our bodies is what brings experience about. Without sensorimotor skill, a stimulus cannot constitute a percept. Noë presents empirical evidence for his claim, drawing on the phenomenology of change blindness as well as tactile vision substitution systems. I highly recommend the book.

The emphasis on embodied experience leads to the use of touch as a model for perception, rather than the traditional vision-based approach. Here’s an excerpt:

Touch acquires spatial content—comes to represent spatial qualities—as a result of the ways touch is linked to movement and to our implicit understanding of the relevant tactile-motor dependencies governing our interaction with objects. [Philosopher George Berkeley] is right that touch is, in fact, a kind of movement. When a blind person explores a room by walking about in it and probing with his or her hands, he or she is perceiving by touch. Crucially, it is not only the use of the hands, but also the movement in and through the space in which the tactile activity consists. Very fine movements of the fingers and very gross wanderings across a landscape can each constitute excercises of the sense of touch. Touch, in all such cases, is movement. (At the very least, it is movement of something relative to the perceiver.) These Berkeleyan ideas form a theme, more recently, in the work of [Brian O’Shaughnessy’s book “Consciousness and World”]. He writes: “touch is in a certain respect the most important and certainly the most primordial of the senses. The reason is, that it is scarely to be distinguished from the having of a body that can act in physical space”…

But why hold that touch is the only active sense modality? As we have stressed, the visual world is not given all at once, as in a picture. The presence of detail consists not in its representation now in consciousness, but in our implicit knowledge now that we can represent it in consciousness if we want, by moving the eyes or by turning the head. Our perceptual contact with the world consists, in large part, in our access to the world thanks to our possession of sensorimotor knowledge.

Here, no less than in the case of touch, spatial properties are available due to links to movement. In the domain of vision, as in that of touch, spatial properties present themselves to us as “permanent possibilities of movement.” As you move around the rectangular object, its visible profile deforms and transforms itself. These deformations and transformations are reversible. Moreover, the rules governing the transformation are familiar, at least to someone who has learned the relevant laws of visuomotor contingency. How the item looks varies systematically as a function of your movements. Your experience of it as cubical consists in your implicit understanding of the fact that the relevant regularity is being observed.

Virtual and augmented reality interface design practices have already begun to demonstrate these concepts. Head mounted augmented reality displays sense the user’s eye and body movements to construct virtual percepts. Head related transfer functions (HRTFs) synthesize sound as it would be heard by an organism with certain physical characteristics, in a particular environment. It seems to me that an important implication for enactive interface design is that haptic sensory patterns can lead to perceptual experience in all sensory modes (vision, hearing, touch). Thus, all human-computer interaction/user experience can be viewed in a haptic context.

Digital Orchestra concert approaches

Rulers in DCS

The McGill University Digital Orchestra is putting on their first performance on March 5th at 7:30 p.m. in Montreal’s Pollack Hall. The lovely and talented Xenia Pestova will be playing the Rulers, an instrument I invented. The piece she will be playing, which I haven’t heard yet, was composed by D. Andrew Stewart and is the first music to be written for the instrument. There will be a live webcast of the show. To watch it, launch the QuickTime application (free download) a few minutes before the start of the show, select “Open URL in New Player” from the File menu, and enter: rtsp://

Whiskers as haptic sensor arrays

Whiskers provide animals with complex perceptual content. In fact, all the things that whiskers actually do are fascinating.

The dimensionality of the data can be modeled according to how an animal moves them through space:

Rat whiskers move actively in one dimension, rotating at their base in a plane roughly parallel to the ground. When the whiskers hit an object, they can be deflected backwards, upwards or downwards by contact with the object. The mechanical bending of the whisker activates many thousands of sensory receptors located in the follicle at the whisker base. The receptors, in turn, send neural signals to the brain, where a three-dimensional image is presumably generated.

Hartmann and Solomon showed that their robotic whiskers could extract information about object shape by “whisking” (sweeping) the whiskers across a small sculpted head, which was chosen specifically for its complex shape. As the whiskers move across the object, strain gauges sense the bending of the whiskers and thus determine the location of different points on the head. A computer program then “connects the dots” to create a three-dimensional representation of the object.

More on that “three-dimensional image” from the end of the first paragraph — whiskers indeed construct a high resolution spatial map:

Based on discoveries in primates and cats, scientists previously thought that highly refined maps representing the complexities of the external world were the exclusive domain of the visual cortex in mammals. This new map is a miniature schematic, representing the direction a whisker is moved when it brushes against an object.

“This study is a great counter example to the prevailing view that only the visual cortex has beautiful, overlapping, multiplexed maps,” said Christopher Moore, a principal investigator at the McGovern Institute and an assistant professor in the Department of Brain and Cognitive Sciences, where he holds the Mitsui Career Development Chair.

Researchers are now working towards developing code for a whisker-like sensor array to be used for robotics. Could this software have human interface applications as well?

This reminds me of the impressive and thought-provoking Haptic Radar/Extended Skin Project. Although the sensing medium in that case was ultrasound rather than a deformable, physical substrate, and the resolution of the stimulators much lower, the researchers state that they intend to make the system more whisker-like as they develop it.

[via Science Daily]

Compact tactile sensors

Man, would I ever love to get ahold of one of these tactile sensors developed for the Shadow Robot Company. Responding to pressure ranging between 0.1 N and 25 N using a quantum tunneling composite material, each sensor has up to 34 individual sensing units, on-board digital signal conditioning, and is the size of a human fingertip. I would bet that this sensor could be used to crudely model the non-pacinian 3 (np3) psychophysical sensory channel, which constructs an acute neural image of skin deformation.

A development kit is available for £1450, which includes one sensor and some interfacing materials. That’s a bit out of my range, but if someone from Shadow is reading this, I will give you free publicity if you send me a sample!

(via aiGuru)

Rolling ball tactile illusion

Friend and colleague Joe Malloch blogs about a very interesting project in which he uses a pipe with an accelerometer in both ends and a vibration actuator in the middle to create a haptic illusion of a rolling ball inside the pipe. From his technical report about the device:

Since the controller already contains five channels of acceleration sensing, it is simple to link this to virtual physical dynamics consistent with real-life gravity and user interaction. The acceleration signal is integrated to approximate velocity, and the resulting signal is used to control the frequency of a periodic signal mimicking the rolling of a ball of a set circumference. By varying the scaling of acceleration data and the scaling of velocity data, the mass and circumference of the virtual ball may be altered. Performing waveshaping of the final signal (or altering the stored waveform if look-up tables are used) alters the perception of the interaction between the virtual ball and the inside of the tube, creating the impression that the motion is smooth or bumpy, or that the inside of the pipe is ribbed. Integrating a second time approximates the position of the ball; this data is used to stop the virtual motion and set the velocity back to zero when the ball reaches the end of the modeled pipe…

Even lacking appropriate amplification and using somewhat un-physical coefficients, people trying the demonstration were convinced by the model – some would not believe that there was actually nothing rolling inside. Observation of users showed that their gaze would often follow the apparent position of the virtual ball, and perception of mass distribution would change depending on which end of the controller “contained” the virtual ball.

The altered perception of mass distribution shows that vibrotaction can give rise to illusions of force. I think there’s a lot of potential for this concept to be exploited for PxD.

Snippets for a simple C state machine

While developing firmware for an LED-based user feedback system, I needed to implement a very simple state machine in C. After spending a few confused hours sweating in black and white, trying to keep track of state variables in my head “the old way”, I found this blog entry which proposes convenient and concise macro definitions for a simple state machine. Suddenly I was coding in full color, and the structure of the program fell into place quickly. Thanks, Jim!

The way a program like this works is by keeping track of the “previous”, “current”, and “next” states in variables. The code inside one of the states is continuously executed, until the “next” state changes. Each time a state is entered or exited, the variables are updated. But by moving the state variable housekeeping to the header file, our switch statement in main() is much more straightforward.

Here’s the code for the macros in the header file:
#define STATE_ENTRY_ACTION if ( currentState != previousState ) {
#define STATE_TRANSITION_TEST previousState = currentState; } if ( nextState == currentState ) {
#define STATE_TRANSITION_TEST_EXCLUSIVE } else if ( nextState == currentState ) {
#define STATE_EXIT_ACTION } if ( nextState != currentState ) {
#define STATE_EXIT_ACTION_EXCLUSIVE } else if ( nextState != currentState ) {
#define STATE_END currentState = nextState; } break;

The EXCLUSIVE macros are used if you want to ensure that the state machine is run at least three times per state, making the timing of the function call more predictable.

Now adding a state here or there, or inserting functions at the right point in the routine is easy:

void main() {
   while( true ) {
      switch ( state ) {
         case state_n:

I did not include a default case in this example, but for a real application it’s a good idea to always include one.