Wearable technology has long had application in musical expression. A historical example can be seen in the “one-man-band,”¹ a concept that dates back well over a century. Figure 1 shows a modern incarnation in such a rig, with each “instrument” mounted for convenient access, responding to the action of a particular limb or a specific, controllable motion of the wearer. Since the instruments were traditionally acoustic, each made a particular kind of sound, and the “action-to-audio” mapping was essentially static. In order to attain a timbral richness approaching that of a “band,” many such instruments were scattered about the body. Despite the apparent clutter, performers could use these adornments to charm and amuse audiences with occasionally virtuosic (although often acrobatic) musical expression as they appropriately flailed away.

Figure 1

With the dawn of electronics, the situation evolved. Now the instruments themselves did not have to be mounted on the performer's body, since they could be replaced by a set of electronic sensors that picked up the motion cues and controlled a remote music synthesizer. In the 1980s, the MIDI (Musical Instrument Digital Interface) standard and digital synthesis brought these systems even further, since now a computer could be easily placed in the loop, recognizing particular motions from real-time analysis of the sensor signals and producing a more complex, dynamic, and captivating software mapping of sound onto action. This was a very liberating process, because the sensor systems freed the body from bearing the burden of the instruments, and advances in synthesis and data interpretation freed the sounds from being tied to simple causal definitions.

Most projects in such electronic musical “wearables”^2,3 come under the rubric of “interactive dance.”⁴ An early example⁵ is found in the work of composer Gordon Mumma, who adorned dancers with accelerometers to control analog synthesizers in performances of the 1960s. The well-known performance artist Laurie Anderson publicized these concepts in her shows of the 1980s,⁶ using active apparel such as body suits adorned with percussive pickup transducers and neckties with embedded music keyboards. In the 1990s, several systems of this sort appeared. Many, such as Mark Coniglio's MidiDancer,⁷ the Danish Institute of Electronic Music (DIEM) digital dance interface,⁸ and the Yamaha Miburi**,³ were based around placing a set of resistive bend sensors across the dancer's joints to obtain dynamic articulation. Because the Miburi was a commercial product, it was packaged as a complete system, including finger controllers for each hand, a wireless interface, an embedded synthesizer, and a set of shoes with piezoelectric taps at the toe and the heel, with each shoe wired to the central beltpack transmitter.

The foot of a trained dancer is a very expressive, multimodal appendage, capable of articulating much more than simple taps. Shoe interfaces for musical performances, however, were dominated by such tap implementations⁹ and, until now, have not appreciably diversified from the toe-heel piezoelectrics.

Different applications have resulted in the adoption of other technologies for foot sensing, although essentially all of these instances concentrate on sensing only a small set of particular parameters. For example, podiatric treatment centers and product development groups at sports shoe companies use densely pixilated pressure sensors¹⁰ to observe the dynamic pressure distribution on the shoe soles during walking and running. In these applications, the shoe is often tethered to a data acquisition system through a multiconductor cable. Much coarser pressure sensor arrays (e.g., sensing at only a few places) have been used in portable commercial products, such as devices to warn patients with podiatric neuropathy about potentially damaging footfalls¹¹ and shoes to interactively coach a golfer on his or her dynamic balance.¹² A pressure-sensing overshoe has also been incorporated in “Cyberboot,”¹³ developed at the National Center for Supercomputing Applications (NCSA) to incorporate foot gesture into virtual reality installations. The “Fantastic Phantom Slipper”¹⁴ was an installation that used a pressure-sensing shoe with an active IR (infrared) optical system that tracked translational position across a small area, enabling users to step on animated insects that were projected onto the floor. Retrofits to jogging sneakers are now being brought to market that use inertial sensors for quantifying footfalls¹⁵ and estimating elapsed distance (i.e., pedometry).¹⁶

The “expressive footwear” device developed in the MIT Media Lab Responsive Environments Group surpasses these niches by using a diverse sensor suite to measure 16 different parameters at the foot, detecting essentially everything that the foot is able to do, and telemetering the data back to a remote host computer in real time, leaving each shoe entirely untethered. Most human-computer interfaces concentrate on precisely measuring gesture expressed by the hands and fingers, devoting little, if any, attention to the feet. We have developed an interface that breaks this tradition, by measuring many parameters articulated at the foot.

The sensor system and shoe hardware

Figure 2 Figure 3 Figure 4 Figure 5

Shoe design and fabrication. A standard foam insole (represented by a dashed line in Figure 2) is embedded with an array of tactile sensors. Two standard force-sensitive resistors (FSRs)²¹ are placed on the left and right in the forward region of the shoe, yielding continuous pressure there and responding to the dancer's rocking of the foot side-to-side. Another FSR is placed forward of the toes, at right angles to the sole so it responds to downward pressure during pointing, when the shoe is vertical. Originally, this sensor was also inside the shoe compartment, but was moved outside for more reliable operation, since its performance varied considerably across different dancers' feet. For easier integration, a more malleable “FlexiForce**”²² FSR was used here (its foil cable is seen running across the side of the sole in Figure 4). At the heel, where dynamic pressure is more relevant, we placed a strip of PVDF (polyvinylidene fluoride),²³ a piezoelectric foil that responds to changes in force.²⁴ Two back-to-back resistive bend sensors,²⁵ which were placed across the middle of the insole behind the toes, measured the sole's bidirectional bend.

A strip of copper mesh adhering to the bottom of the insole acted as a pickup electrode, capacitively coupling to transmitting electrodes, placed on the stage, that broadcast a constant sinusoidal signal at 55 kHz. When the dancer is above one of these plates, the signal received at the shoe decreases with the distance of the shoe from the plate,²⁶ giving an indication of the height of the shoe above the stage. Another electrode (not shown in Figure 2) is placed above the insole, just below the dancer's foot, and is connected to the local electronics ground. This breaks the symmetry²⁷ between the pickup electrode, isolated below the insole, and the local shoe electronics ground, which is now effectively coupled to the dancer's body. The dancer, in turn, is ambiently coupled to the house ground, enabling current to flow from the transmitter plates into the shoe, hence allowing the shoe system to capacitively receive the transmitted 55 kHz signal. The height of the foot is inferred from the detected signal strength.

A small (2¼" × 3¼") circuit board is affixed to the outside edge of the shoe on a metal mount, containing additional sensors and electronics. In our original design, the orientation of the foot at an angle, , about the vertical when the foot was nearly level was obtained from a model 1525 analog electromechanical compass,²⁸ a small gimbaled magnet with quadrature rotation measured by a pair of Hall sensors, manufactured by the Dinsmore Instrument Corporation in Flint, Michigan. This monitored the orientation of the foot relative to the ambient (Earth's) magnetic field. While the Dinsmore device was adequate for capturing slower motion during initial operation, after several hours of use the mechanics would start to fail and the gimbal would stick. As our development experience often indicated, the large forces and shock impulses encountered at a dancer's foot are quite hostile to any fragile devices. In subsequent versions, the electromechanical compass was replaced with an all-solid-state device using permalloy bridge sensors, the Honeywell HMC2003 3-axis magnetic sensor,²⁹ which we modified³⁰ for 5-volt operation and higher gain. Although this sensor was quite reliable and gave wonderful, prompt 3-axis rotational response (another degree-of-freedom above the Dinsmore), permalloy bridges can drift over time as the sensing elements lose their magnetization. Therefore, a set of “strapping” pins was provided on the shoe card. By momentarily connecting an 18-volt source across these pins, all magnetic bridges would be subject to a brief current pulse that would magnetically saturate the permalloy, strapping it to maximum sensitivity. Over normal usage, this strapping procedure would be adequate for at least several days, if not weeks, of operation.

Because spins are important gestures to detect, we mounted another rotational sensor, a compact gyroscope (a Murata GyroStar** vibrating-reed device³¹), on the sensor board, aligned with the axis of the ankle. This provided a direct measurement of angular rate about the vertical, giving clear response to spins and twists.

A 2-axis, ±2 G (where G is the acceleration of gravity) MEMs (microelectromechanical systems) accelerometer from Analog Devices (the ADXL202)³² measured the tilt of the shoe with respect to the gravity vector and responded to the moderate accelerations of foot swings. Impact shocks and kicks, at higher G levels, were measured in 3-axes by a triple piezoelectric accelerometer (the ACH-04-08-05 from Measurement Specialties).³³

A small (1 centimeter diameter) piezoceramic sonar receiver (e.g., the Polaroid 40KR08³⁴) detects 40 kHz pings sent from as many as four locations around the stage. By timing the reception of their first arrival, the translational position of the shoe can be tracked. The shoe system is able to receive pings across a distance of roughly 20 feet using our current projectors, which are standard 1.5-cm diameter 40 kHz piezoceramic sources ganged in pairs. Additional range can be attained with more powerful emitters. With four independent projectors, at least one shoe is generally able to detect the signals from at least two projectors in our present performance configuration (see the section on dance applications later in this paper), fixing the dancer's position on the plane of the stage.

A PIC** 16C711 microcomputer from Microchip Technology Inc., clocked at 16 MHz, is embedded onto the shoe card to digitize all signals and produce a serial data stream, which is broadcast to a base station through a small radio frequency (RF) transmitter, currently the “TX” series from Radiometrix.³⁵ Each shoe streams data at a separate frequency (418 and 433 MHz). The 20 Kb/s peak transmitter data rate enables a full state update rate from each shoe that approaches 50 Hz. Our shoes use a helical stub antenna that protrudes behind the heel, as seen in Figures 2-4. This enables the shoe's transmissions to easily be received across a normal stage; we have used them successfully beyond 100 feet from their base stations, but this performance depends, of course, on the local RF environment. Although the output of these transmitters is just under a milliwatt, they are still too strong for FCC (Federal Communications Commission) regulations, which are well under 10 microwatts in these bands. In addition, emission at 418 MHz is limited to brief duration. The corresponding European limits, for which these transmitters were designed, are much more liberal. Such rules certainly restrict the carefree operation of our present system. As outlined in the last section of this paper, we are currently developing higher-bandwidth, channel-shared communications hardware that will allow for the legal operation of multiple embedded transmitters that meet our requirements.

All onboard shoe electronics draw a current of about 50 mA (milliampere) at 5 volts. The original shoe system used an onboard ½ AA-size 6.2-volt lithium camera battery, which provided a useful life of a few hours. After the first model, however, we moved to an off-card 9-volt alkaline battery, which provides for at least a half-day of very stable continuous performance. Although the operation could be extended significantly by substituting a switching regulator for the on-card series regulator or only powering the compass module (which consumes nearly half of the board's current) during its readout,³⁶ this battery life span was already sufficient for our performance applications, so the additional design complication was not warranted.

Using the shoe. It is much easier to work with this shoe system than most other types of wearable interfaces. One only needs to put on the shoes and flip their power switches; there are no connectors, tethers, cables, harnesses, etc., to worry about. Although some of the sensor systems (e.g., the sonar) could be well implemented at other locations on the body, having all devices concentrated at the shoes greatly simplified the setup. Many dancers have worked with this system and have encountered few, if any, problems with the mechanics and location of the electronics module or antenna. Of the two, the antenna proved the most restrictive, since it could limit ankle motion. It should be noted, however, that all of our dancers worked in a free-form, interpretive, and improvisational modern genre, as opposed to traditional styles such as tap and ballet that may involve more constraints. With more engineering (e.g., going to an embedded loop antenna and distributing the electronics throughout the shoe), the system can be made much more innocuous. In addition, the current device is largely hardwired into a particular shoe. Additional design work can make such a system modular, perhaps clipping onto a shoe with an adjustable insole that is adaptable across a wide range of foot sizes.

Electronics, base stations, and system integration

Figure 6

Electronics. Signal conditioning for the FSR sensors is simply an emitter follower; because no voltage gain is required, this allows the series gain-setting resistor in the FSR to become as large as needed to provide adequate response to toe pressure, while presenting a low impedance to the analog to digital inputs of the PIC. Likewise, the PVDF signals are buffered by a junction field-effect transistor (JFET) source follower, enabling the PVDF shunt resistance (which limits the low-frequency bandwidth) to be set at 40 M (megohm). The back-to-back bend sensors are fed through a differential amplifier to give bidirectional response. The capacitive pickup signal is first conditioned by a passive LC bandpass filter tuned to the 55 kHz transmitter (rejecting ambient background at other frequencies), then fed through a gain block and half-wave envelope detector that extracts the positive amplitude of the received signal. The three shock accelerometer signals are amplified, then time-stretched with a similar half-wave envelope detector, allowing them to be reliably digitized by the PIC across its data acquisition cycle. Although this loses polarity information, the raw accelerometer signals are too narrow to be detected by the PIC at its 50 Hz sampling updates. The signals coming directly from the Murata gyroscope are perfectly within the 0­5-volt digitization range without further conditioning, as are the 3 output signals from the Honeywell compass after it was modified, as mentioned earlier. In addition, the regulated 3-volt supply used by the RF transmitter module is digitized by the PIC and transmitted with each data set, since it is used to continuously monitor the 5-volt supply, which is used as the A/D reference. The 3-volt input will appear to grow as the 5-volt supply droops.

The latest version of the shoe electronics card has two 8-channel analog multiplexers. With the 4 analog inputs built into the PIC, there are 18 available analog channels. Since the shoe system uses only 14 of these, the extra 4 inputs are brought to a header, where they are available for other devices (e.g., these are useful when the card is embedded in systems other than the shoe, as mentioned in the section on other applications later in this paper).

The two low-G accelerometer outputs are digital 1 kHz pulse trains, with the duty cycle of each pulse corresponding to the detected acceleration along the respective axis. They are thus connected directly to a pair of PIC digital inputs. After the PIC digitizes the analog data, it uses software timing to measure the accelerometer pulse-widths, retaining 8 bits of resolution.

The signal from the sonar receiver is first amplified (since the piezoceramic head is already highly resonant, there is no bandpass filtering), then routed through a half-wave envelope detector and sent to a discriminator with adjustable threshold (setting the sonar sensitivity). The discriminator output is applied to a PIC digital input that can generate an interrupt when the discriminator goes high, executing a tight segment of code that starts the timer of the PIC and sets a “sonar received” flag. When the PIC is about to transmit the byte in the serial data record dedicated to the sonar, it checks this flag to see if a ping was received, and if so, it sends the timer value (otherwise it sends zero). This parameter is thus the latency between the time when the ping was received and the time when the sonar byte was transmitted. Making the sonar threshold manually adjustable allows the user to set the trade-off between sonar sensitivity (e.g., range of operation) and any 40 kHz background noise. Most of this noise is caused when the dancer lands hard from a jump or stomps a foot; because the accelerometers also detect this state nicely, any such spurious sonar spikes that coincidentally occur can be removed in the base station or subsequent PC software.

The primary 5-volt supply for the shoe hardware is conditioned by a low-dropout series regulator that produces a battery-low gate, which is tripped when the battery drops below 5.3 volts. This gate is also read by the PIC and encoded into its data transmission.

Base stations. Figure 7 is a block diagram for the base station. It is much simpler, mainly consisting of a PIC16C73 microcomputer (during the hardware design cycle, it was the smallest PIC available with hardware serial ports) that receives serial input from a Radiometrix RX-series RF receiver, which picks up transmissions from the shoe and sends serial output to an RS-232 driver (for communicating with a personal computer's serial port).

Figure 7

In order to provide an appropriately zero-balanced RF serial stream, the shoe's PIC uses a very simple, brute-force variation of Manchester encoding, in which it first sends all data bytes for a full record of sensor values and then sends their binary complements. Additionally, by comparing each data byte in a record with its transmitted complement, RF reception errors are detected, and individual bad bytes are “failed” and ignored, thus keeping the rest of the record intact. In order to enable the base station to quickly synchronize to the shoe's data cycle, the first byte in a record is marked with a unique code (either 254 or 255, depending on the battery-low gate). This code is not permitted to appear in subsequent values.

System integration. Figure 8 shows a high-level block diagram of the entire expressive footwear system. In the current rendition, two base stations are needed, one for each shoe. Each base station listens for its shoe at a different RF frequency (as mentioned, 418 or 433 MHz). One of the base stations, deemed the “master,” also has an onboard 55 kHz sine wave generator and driver for the electric field transmitter plates, which are detectable by both shoes. The master's PIC additionally generates four gates for the sonar pingers, each of which produces a few-millisecond burst of 40 kHz ultrasound when triggered. The master pulses a sonar gate every tenth of a second, going round-robin through all connected pingers.

Figure 8

The master's PIC uses its timer to measure the interval between sending the ping and receiving a valid byte detected by sonar from the shoe. The value of the sonar byte sent from the shoe (containing the latency in the shoe) is then subtracted from the value of the master's timer (containing the acoustic transit time plus shoe latency), resulting in the amount of time it took the ping to reach the shoe, hence the distance of the shoe from the pinger. This sonar system works satisfactorily, providing 8 bits of position resolution across a 30-foot range. As seen in Figure 8, a pulse from the master synchronizes the slave base station when the master sends each ping. This pulse interrupts the slave's PIC to start its timer, enabling the same sonar algorithm to work there. The master and slave base stations keep track of which sonar head was the last to ping, sending that address along to the host personal computer (PC) with every data record.

The current system produces a pair of 19.2 KBaud serial streams from master and slave base stations that are combined in the analysis and content software running on the host PC. The 50 Hz state-update cycle is primarily limited by the 20 Kb/s RF data rate, which is at the edge of capability for the Radiometrix transmitter and receiver modules that we are currently using. A more efficient zero-balancing scheme would speed up the effective data rate to within a factor of two. The data interpretation algorithm running on the host PC provides another layer of error protection by ignoring any spurious “spikes” on most sensor signals (e.g., data that abruptly jump from the baseline to a significant value then return directly to the baseline on the subsequent sample). This introduces an intrinsic delay of one 20-millisecond data cycle.

System performance

Figure 9

After about 7 seconds, the walking stops, and the foot is moved about more wildly, as can be noted in the drop in the consistency of the pressure signals. At roughly 11 seconds, the foot is rotated perpendicular to the floor, and the wearer presses the front of the shoe against the ground, as seen in the tip pressure plotted in the second graph at left in Figure 9 (because this action is very deliberate and the data very clean, it is a good candidate to use for triggering important events). The sole bending is also seen to be unipolar and modest through most of the test, as expected, since jogging sneaker soles are not easily bent in the reverse direction. An exception is near the end of the test, when the foot was rotated and pushed against the floor.

At 6 seconds, two steps were made on the 55 kHz electric field transmitter plate, as clearly seen in the “capacitive height” signal, which gives an extremely simple and reliable signature.

The gait dynamics also leave some traces in the twist gyro (lower left of Figure 9) and low-G accelerometer signals (top right), which can be seen to jump more in the latter part of this test, when the foot was moved about more wildly. Both of these channels have still additional headroom to respond to a dancer's very fast twists and foot swings. Some muted traces of gait can also be seen in the shock accelerometer traces (middle right), but three sets of spikes, corresponding to foot stomps, stand out clearly after 8 seconds, primarily in the second and third sensor axes. The directional difference in the foot strike acceleration is evident from the balance between amplitudes. The low-G accelerometer is relatively insensitive to impacts, because they are too transient. If the accelerometer responds at all, it tends to produce a very narrow and modest spike (as seen in these data), which the analysis software rejects as noise. As designed, the accelerometers are complementary systems: the low-G channels indeed pick up foot swings and tilts, while the high-G sensors nicely detect the shocks.

The magnetic field (“compass”) signals, at lower right of the same figure, are likewise seen to follow the gait, gyrating as the foot pitches cyclically during walking, or giving a different response when the foot rotates during a turn, twist, or swing. At the end of the test, compasses 1 and 2 show the foot pitching up when the tip was pressed against the floor.

Dance applications

Figure 10

Figure 10a - Yuying Chen, dancer (MIT Wearables Fashion Show, 1997) (MPEG, 2.5Mb)
Figure 10b - Mia Keinanen, dancer (MIT Media Lab, 1998) (QuickTime, 5.0Mb)
Figure 10c - Brian Clarkson, gymnast (NICOGRAPH, 1998) (QuickTime, 3.6Mb)
Figure 10d - Dawn Kramer, dancer (IDAT, 1999) (QuickTime, 5.0Mb)
Figure 10e - Takei Minoru, juggler (Tokyo Toy Fair, 1999) (QuickTime, 5.0Mb)
Figure 10f - Byron Suber, choreographer (ADF, 1999) (QuickTime, 5.1Mb)
Figure 10g - Mark Haim, choreographer/dancer (Sens*bles, 1999) (QuickTime, 5.1Mb)

Our musical mappings are essentially “direct manipulation,”³⁹ in which sonic events are tied to sets of simple gestures under direct control of the dancer, as opposed to trying to garner more sophisticated interpretation from a higher level analysis of the data. Although interesting from a research viewpoint, this can risk removing some of the dancer's immediate control. Our approach gives improvisational dancers a “palette” of action-to-sound rules and relationships that they can exploit to evolve a compelling performance after practicing with the system.

Wearable computing fashion show. Because our prototype shoe (Figure 2) was designed to be exhibited at the first IEEE conference on wearable computing¹⁷ in October 1997, it was pressed into actual stage performance a few days later at the subsequent “Beauty and the Bits” event,⁴⁰ the world's first wearable computing fashion show held at the MIT Media Laboratory. Because only one base station had been constructed, this mapping used only a single shoe. All subsystems, except for the sonar, provided usable data. Since it was intended for a brief on-stage walkthrough, the mapping, as described below, was very simple and literal, easily mastered but very limited in scope.

The music itself consisted of three voices: a drum voice, a bass voice, and a melody voice. The drum voice ran steadily throughout the whole piece and gave a rough “techno feel” to the music, fitting the mood of the fashion show. The volume of the bass drum and the bass voice were controlled by the pitch tilt () accelerometer, and the electric field height sensor controlled the volume of the other drum instruments. The tempo was adjusted slightly by the bend sensor. The bass voice and melody voices were switched on and off in various combinations as impulses were detected by the shock accelerometer. The bass voice produced a harmony effect, and the specific harmony was selected by rotating the shoe in (yaw). The bass voice was articulated by changing its octave upon detecting impulses from the rear PVDF sensor. The melody voice played harmonizing tones in upper registers. The range of the melody voice was controlled by the front pressure sensors. Panning and flanging of both voices were controlled by the detected compass direction. Also attached to the high-G accelerometer was an explosion sound, triggered by heavy stomps and kicks. Finally, a panning wind sound was produced with quick rotations, as detected by the twist gyro. An MIT student dancer, Yuying Chen, practiced with these mappings and performed at the fashion show (Figure 10A).

Colloquium on Musical Informatics. ¹⁸ The next application used athletic shoes, and was built around the Nike Air Terra Kimbia of Figure 4. This mapping has no pedestrian rhythm, but is entirely freeform, triggering and modifying simple sounds in accordance with the dancer's gestures and hence providing the audience with a more direct causal link. We tied random melodic notes to toe and heel pressure, and the bend of the sole selected a harmony chord that was changed in volume in accordance with the pitch tilt () of the shoe. Roll tilt () faded up (in the sense of sound systems) an additional harmony chord. The height of the shoe above an electric field transmitter embedded in the carpet dictated the volume of the bass pedal point notes. These increased in volume as the shoe approached the floor. A fast movement or stomp, as detected by the shock accelerometers, introduced percussive sounds, and fast twists, as indicated by the rate gyro, introduced a stream of random piano notes. Our performer in this sequence (Figure 10B) was another MIT semi-professional dancer, Mia Keinanen from the Gesture and Narrative Language Group of the Media Lab.

NICOGRAPH show. Our next mapping was with an athlete, Brian Clarkson, a trained gymnast and student in the Media Lab's Vision and Modeling Group. The performance took place during the Wearable Computing Fashion Show at NICOGRAPH'98 in Tokyo (Figure 10C). Here, we went back to having the performer's actions embellish a background pedestrian music sequence, as in our original example. Notes were likewise tied to toe pressures and transposed with bend. Glissandos were attached to spins and tilts, sharp sonic transients to jumps, and panning/crossfading to the compass. When the performer approached the portion of the stage where the electric field sensor was embedded, a percussion track faded up. Additional effects were introduced specifically for the performer's gymnastic style: a sharp blast of noise for very fast spins and detection (via the low-G accelerometers) of handstands, at which point a drumroll would begin, finishing in a loud crescendo when he landed.

IDAT and Tokyo Toy Fair. At this point, the shoe system was reliable enough to use in performance with dance professionals, so we enlisted the collaboration of Byron Suber, a choreographer from Cornell University's dance department. Taking the gymnast's mapping as a starting point, we worked with Byron to make it dance-relevant, going through the musical response of each sensor, one by one. This resulted in a complex, yet very controllable demonstration piece that again dispensed with the background sequence and enabled the dancer to launch and modify a variety of sounds, using all of the sensor systems simultaneously. In this mapping, the right/left toes and heels produced various melodic tones in an assigned harmony. Pressure sensor response from both feet must be present for these tones, thus ensuring that they are both on the ground. Bend of the sole transposed these toe melodies by an octave up or down, depending on the bend direction, and pressure at the toe tip sensors triggered cymbal crashes. The gyro picked up twirls, again launching a cascading glissando (and burst of white noise for very fast right-spins). The shock sensors launched orchestra hits, and the shock of the left foot also turned off all notes and changed the harmony played by the toes. Forward pitch tilt () launched “sparkling” notes for the left foot and ambient string sounds in the right foot, while sideways roll () tilt would adjust the octave ranges of sounds controlled by the corresponding pitch tilt. This version of the system had functioning sonar. As the shoes approached the single sonar pinger used in this mapping, a cymbal/snare rhythm would start, growing louder with the dancer's proximity. If the dancer stepped on the electric field transmitter, all sounds would stop, and a droning chord would fade up, increasing in volume as the foot was lifted away from the electrode. There was a different sound for each foot. Chord voicing changed as the foot was rotated (as derived from the compass signal).

This mapping had its debut in a live demonstration at the International Dance and Technology (IDAT) '99 conference in Tempe, Arizona.¹⁹ It was tried by many dancers (e.g., Figure 10D shows Boston dancer Dawn Kramer performing with it for a television newscast). The sonic palate was rich and controllable, allowing a dancer to acclimate to it within an hour or so of practice. The sonic palate gave complex, causal, and appropriate response to the dancer's motion. It also succeeded outside of the dance community. For example, we worked with a juggler, Takei Minoru (Figure 10E), who used it to sonically embellish his mime/juggling routine in scores of regular public performances at the 1999 Tokyo Toy Fair.

American Dance Festival. Our last mapping was the most complex. It was developed for a performance and demonstration at the July 1999 American Dance Festival with a pair of dancers, Rachel Boggia and Lena Rose Magee, each with one active shoe (Figure 10F). It was choreographed by Byron Suber and refined further for New York choreographer/dancer Mark Haim's solo performance (Figure 10G) during the Sens*bles conference at MIT's Kresge Auditorium in October 1999. The stage setup used in this performance is shown in Figure 11. The sonar system was now fully operational, hence all four projectors were deployed at the corners of the square capacitive transmitter plate. (They can be seen in the Figure 10G photo of Mark Haim.) The resultant tracking was used to divide the stage into discrete regions, as labeled in Figure 11. This quantization was used to generate a dynamic musical mapping, where the musical mode would depend on the regions in which the dancer was located and/or had visited.

Figure 11

By default, there is no quiescent background sound over this mapping. The dancer(s), however, can select a looped music sample to start playing by staying relatively stable for several seconds in regions A₃₁ or A₁₃, with the left foot facing the center square. There are five samples of different musical excerpts in all (30-second loops, ranging from Cajun music to classical music), and these are selected by the region and range at which the trigger occurred. The sample will continue playing until the dancer toe taps (i.e., points the front of the shoe down and presses) with the left foot. In contrast, toe tapping at any time with the right foot plays a brief portion (e.g., 5 seconds) of a minimalist MIDI sequence written by Cornell composer David Borden. Each portion has three parts that the dancers can add and subtract with additional toe tapping. When a shoe is lifted and tilted (or swung), continuous audio effects will be proportionally added to a voice in the sequence or to the music sample (e.g., filtering, flanging, reverb, crossfading, or vibrato, depending on what is currently playing). Both angles (, ) of tilt are used, allowing one shoe to control the mix of two different effects. The pressure sensors in the sole again sound notes (bass on the heel, melody on the toes), except when a music sample is playing, at which point they trigger soft percussive sounds. These (together with the backing sequence, if appropriate) will transpose up and down with bend of the sole.

If a dancer is in zones A₁₁ or A₃₃, the sole's pressure sensors will play different pitched speech phrases pronounced by a computer-generated voice, allowing the dancer to put musical sentences together with their movement. These phonemes are also transposed by the sole's bend, and effects are similarly introduced with tilt. Throughout the dance, different sonic events are tied to the shock accelerometer signals (e.g., jumps, leaps) and rate gyro, as in the previous mapping. The shock sensor response is also processed to determine the intensity of the dancer's activity. If the activity is relatively smooth, the shock sounds are distinctive, but not dominant. As the dancer's motion becomes more vigorous and sustained, the shock sounds become progressively more intense. When a shoe enters the electric field transmitter at stage center, the same effect is produced as in the previous mapping; e.g., any background notes or samples are silenced, and orientation-dependent drones are produced. Fast foot swings in this region are detected by the low-G accelerometer, and launch swooshing sounds. Every time a dancer enters and leaves the central electric field region, the voices playing the notes triggered by the insole pressure sensors change (there are seven voice loads defined, which cycle round-robin with each visit to the transmitter pad).

This mapping has advanced the shoe system well beyond the “demo” stage. It has sufficient depth and variation for professional dancers to work with in many different ways, entertaining an audience throughout an extended performance.

Machine perception. In an entirely different project⁴¹ using the prototype shoes, real-time classification algorithms have been developed that detect certain dance styles from the shoe data stream, e.g., discriminating between a waltz and a tango. After exposing the analysis to several seconds of the real-time dance data, the appropriate musical accompaniment would fade up once the decision was completed. Because the data streaming from the shoe system provides a rich description of foot activity, this example represents only the first step in a promising trajectory of applying more sophisticated analysis for extracting higher-level features that may be useful in dance and sports training or podiatric therapy.

Other applications

Figure 12

The right-hand photograph in Figure 12 shows an evolution of our shoe sensor concept into a different form factor: a wireless, 6-axis inertial measurement unit (IMU) that fits inside a common bread bun, complete with microcomputer, RF transmitter, loop antenna, and a battery that lasts for at least two days of continuous operation while streaming data at 65 Hz. A pair of these devices⁴⁸ was built for the void* installation⁴⁹ shown at SIGGRAPH 99. In this exhibit, a user could control one of three semi-autonomous virtual characters, causing them to dance. Drawing inspiration from Charlie Chaplin's famous “buns and forks” scene in The Gold Rush,⁵⁰ we created a pair of input devices whose outer casings were two bread rolls, each with a fork stuck near the end, thereby mimicking a pair of legs. The IMU was placed inside the buns. A variety of gestures (kicks, twirls, etc.) were recognized (using a similar gesture-recognition algorithm to that in Swamped!) and used as controls for the virtual characters. These buns also transmitted a coded, low-frequency RF signal that enabled them to be identified when placed near capacitive receivers embedded in the table top, under a set of dinner plates.

Conclusions and future directions

The RF solution taken here, devoting one fixed frequency to streaming data from a particular shoe (see Figure 8), is inefficient and often illegal for operation in several parts of the world. Rather than devoting a separate base station to each shoe, a superior strategy is shown in Figure 13, where a higher-bandwidth channel-sharing scheme (e.g., code division multiple access [CDMA] or time division multiple access [TDMA]) is used to address several such wireless sensor packages. Although this involves significantly more complication in the RF hardware and data management/handshaking schemes, it scales much better, allowing us to more easily instrument a full ensemble of dancers and/or place many discrete wireless sensor packages around a dancer's body to access additional gestures beyond the feet. Commercial packages are now making an appearance that have the potential for solving this problem, e.g., the single-chip⁵¹ Bluetooth⁵² transceivers. Although appealing to the many consumer applications for which they are being developed, Bluetooth devices are limited to 7 nodes per base station and devote considerable overhead to dynamic network management. This capability is not necessary (and potentially detrimental) for performance applications of the sort that are described in this paper. We are thus developing a new system,⁵³ diagrammed in Figure 13, that uses a static network topology that can be a priori programmed into a “heavy” base station, which communicates with a set of lightweight, low-power, fast TDMA transceivers at each sensor package.

Figure 13

Although placing so many sensors at the feet was novel and technically challenging, it was an adjustment for many modern dancers who were used to gesturing more equitably across their entire body. By contrast, most current interactive dance systems are based on video tracking (e.g., see Reference 54), which only monitors the body proper and does not adequately address the feet. After some hours of practice, our dancers acclimated and learned to direct most gestures through their feet, where they would get appropriate musical response.

This system presents a different environment from a standard tap shoe, which produces its output only when in contact with the floor. Although our system is somewhat slow for precise tap performance (limited by the 50 Hz state update rate coupled with any processing delay incurred at the host PC), the foot dynamics were continuously measured by the inertial, tracking, and tactile systems when elevated above the stage, enabling other expressive degrees of freedom.

We have barely begun to subject the rich, descriptive data streaming from our shoe system to significant gestural analysis, an area promising to bear fruit for automated dance training, sports coaching, podiatric therapy, or biomotion research. Such data acquisition systems also promise to open new vistas into professional sports broadcasting, especially in the upcoming era of digital television. Exploiting similar dense sensor packages, data can stream directly from the figure skater's blades or boxer's glove to the viewer's entertainment system, where it is mapped onto different information representations, potentially including parametrically adjustable musical mappings, such as those explored here.

This shoe music system blends music composition with dance, as aptly articulated by Merce Cunningham.⁴ Since our recent mappings give performers access to a complicated musical palette linked to their motion, virtuosity with such devices logically requires some level of musical talent. It is therefore no coincidence that dancers who are also adept musicians seem to do the best work with our shoe system.

Acknowledgments

**Trademark or registered trademark of Yamaha Corporation, Ballet Makers, Inc., E. I. du Pont de Nemours and Company, Microchip Technology Inc., Tekscan Incorporated, or Murata Manufacturing Company, Limited.

Cited references