Wearable Sensor Badge & Sensor Jacket for Context Awareness

Jonny Farringdonα, Andrew J. Mooreα, Nancy Tilburyβ, James Churchβ & Pieter D. Biemondα

Philips Research LaboratoriesCross Oak Lane, Redhill, Surrey, RH1 5HA, UK.

αSoftware Engineering & Applications Group,βPhilips Designfarringdon@bcs.org.uk Andrew.Moore, Nancy.Tilbury @philips.com

james.church@btinternet.com pieterb@acm.org

Abstract

The addition of sensors to wearable computers allowsthem to adapt their functions to more suit the activitiesand situation of their wearers. Here a wearable sensorbadge is described constructed from (hard) electroniccomponents, which can sense perambulatory activitiesfor context-awareness. A wearable sensor jacket isdescribed that uses advanced knitting techniques to form(soft) fabric stretch sensors positioned to measure upperlimb and body movement. Worn on-the-hip, or worn asclothing, these unobtrusive sensors supply abstractinformation about your current activity to your otherwearable computers.

Keywords accelerometers, context-aware computing,gesture recognition, mobile computing, soft sensors,smart clothing, smart fabrics, technical textiles,wearables.

1. Introduction

Wearable computers and devices are becomingincreasingly aware of their wearer's current activities.Current personal and mobile devices, such as personaldigital assistants (PDA's) and mobile phones havelittleor no knowledge of their owner's situation. They executetheir functions with no regard for appropriateness, andthus ring, vibrate, and bleep when they think they should,no matter what.

The addition of sensors to a wearable device allows forthe wearable's behaviour to be related to the wearer'scurrent activity or situation, [9]. Two sensors commonlyused with wearable computers are small cameras andchorded keyboards. The idea of a sensor unit able toderive a person's activity is established as a useful

addition to other wearable devices. Van de Velde'sTechnology for Enabling Awareness (TEA) projectunderway at Starlab with Nokia is constructing such adevice for use with a mobile phone [12]. The first TEAprototype contains the following sensors, “light,temperature, CO gas, microphone, two accelerometers formovement, a pressure sensor, and an infrared (IR) sensorfor motion detection.”

The use of biometric sensors with wearable computersis discussed by Picard and Healey [7]. Such sensors allowfor new interactions between the wearable and the wearerbased upon affect detection, prediction, and synthesis [6].Biometric monitoring for personal health in a wearable isdescribed by Mann, using commercial sensors andalgorithms such as signalling danger when the ratio ofheart rate and walking speed goes outside a certainrange, [3].

Here we describe wearable hard and soft sensors.Other wearable devices can exploit sensing informationin order to mediate their behaviour, making themsomewhat aware of, and thus behave more appropriatelyto, their user's current activities. Using commerciallyavailable electronic components and custom software asensor badge detecting six perambulatory activities hasbeen developed. Using bespoke knitwear and garmentdesign a sports top detecting eleven degrees of limb andupper body movement is described. The relative merits ofeach sensor system are also discussed. In addition, thebenefit to the user of wearing sensors is the improvedbehaviour of their wearable computer now related to theircurrent activity.

2. The Sensor Badge

The sensor badge uses stock electronics componentshoused in a belt worn unit. Novel software to derive highlevel meaning from the sensor data runs on amicrocontroller chip, and the current activity of thewearer is displayed on the front of the badge as well as

output to any wearable device via an RS-232 serialconnection.

2.1. Sensor Selection

Determining a person's activity within the sensorbadge is achieved by measuring the accelerations,constant and varying, exerted on the badge by themovements of the user. The sensor chosen to measureacceleration was the Analogue DevicesADXL05, anaccelerometer that can measure up to 5g accelerations inthe direction of one axis. It was chosen due to itsmeasurement of constant and varying accelerations andits relative low cost, around 20 US$ in small quantities.Two directions of acceleration were chosen as the mostfruitful, forward horizontal and vertical acceleration,relative to the wearer of the badge. This made necessarythe use of two ADXL05 accelerometers. The twoaccelerometers were calibrated to a full scale of +/- 2g.

2.2. Sitting, Standing & Lying

Identifying when a wearer of the sensor badge wassitting, standing or lying was a simple exercise. Theseare states where the wearer of the sensor badge is notsubjecting it to any varying accelerations, thereforedetermining the state of the user is simply a matter ofexamining the magnitude of the voltages produced by theaccelerometers. Figure 1 shows the different orientationsof the sensor badge in standing, sitting and lyingpositions. These all result in different voltage levels fromthe horizontal and vertical accelerometers that reflect theg-force and direction of the users current state.

Figure 1. Orientation of the sensor badgein standing, sitting and lying positions

To reliably detect one of the four states above, bothaccelerometer output voltages were sampled 20 times persecond, the average for each was calculated together withthe difference between the maximum and minimumvalues. If this difference was small enough as to suggestthat the wearer of the badge was stationary, the averagevalues were examined to determine the users currentstate.

2.3.Walking & Running

A slightly more complicated algorithm was employedfor detection of walking and running. The sampling rateof 20 samples per second was found to be satisfactory forrepresentation of the walking and running accelerometertraces as shown below in Figure 2 and Figure 3. Thesetraces were captured during an initial data collectionperiod using a Xybernaut wearable computer connectedto a Pico Technology ADC11 parallel port analogue-to-digital converter, [10], connected to two similarly alignedaccelerometers.

Figure 2. Horizontal and verticalaccelerometer walking traces

Figure 3. Horizontal and verticalaccelerometer r unning t races

The difference between maximum and minimum, i.e.the heights of the traces are used as characteristics todetect walking and running. The horizontal heights ofthe walking and running traces can be seen as similar,while the running vertical height is much larger than the

Walking (20sps, 2sec)

0

100

200

300

400

500

600

700

800

1 4 7 10 13 16 19 22 25 28 31 34 37 40

Sample

Val

ue Horizontal

Vertical

Running (20sps, 2 sec)

0

200

400

600

800

1000

1200

1 4 7 10 13 16 19 22 25 28 31 34 37 40

Sample

Val

ue Horizontal

Vertical

walking vertical height. These magnitudes are used as afirst indicator of either walking or running. The averagevalues of the above traces are similar to the averagevalues obtained when the wearer of the badge is standingstill. This information is used as a second indicator.

The frequency of the traces is also important toidentify whether the user is walking or running orneither. As the sensor badge uses a PIC microcontrollerwith limited memory, a crude measure of frequency wasused. By counting the number of times each trace crossedthe average value, an approximate measure of frequencywas obtained. Using this information together with thetwo measures described above, the detection of walkingand running was achieved.

Figure 4. Front and inside views of thesensor badge

2.4. The Sensor Badge

The prototype “Sensor Badge” as shown in Figure 4 isthe first physical implementation of the abovealgorithms. It uses six LED’s to show the users currentstate from walking, running, standing, sitting, lying faceup and lying face down. The serial port is designed tomake the sensor badge a source of context information.Another wearable device may be able to use the contextdata supplied by the sensor badge to modify its ownbehaviour in a manner more befitting the user.

3. The Sensor Jacket

The aim of the “Sensor Jacket” is to detect the postureand movements of the user by using knitted stretchsensors and knitted conductivetracking (described later).Designing technical textiles into a jacket has resulted in aunique completely fabric worn sensor. This newunobtrusive sensor can be connected to a wearablecomputer. Indeed it becomes part of the sensoryapparatus of the wearable computers.

Most current systems for the analysis of humanmotion cannot be used for wearable applications becausethey use external devices mainly suitable for researchlabs or other “Smart Rooms”,[5]. Cheap and wearablesoft sensors can be used as an alternative to expensiveand often very obtrusive sensors and trackers. If thesensors are integrated in everyday clothing then thegesture recognition isn’t bound to a specific location andcan be used at any place at any time.

3.1. Technical Textiles & Smart Clothing

Using the electrical properties of soft materials andadvanced knitting techniques new fabric sensors andtracking have been created. These technical textiles canbe seamlessly integrated in clothes, making them ‘smart’or ‘intelligent’. Smart clothing is considered asempowering the wearer with smart devices for theirpersonal use. Mann [2, 3] addresses other forms of smartclothes and smart uniforms. Post and Orth [8] discuss theuse of smart fabric switches and circuits by usingconductive yarn. Smart fibres are currently of particularinterest in the textile trade who consider “anti-stress,calm-inducing properties, or pamper with fibresimpregnated with fragrant oils or vitamins, or even beingable to electronically monitor the wearer's health or stateof mind through interactive wearable computers.” [13].

Additional benefits of fabric sensors are that they:• can be integrated into smart clothes in a non-

obtrusive way• can be manufactured using standard garment

industry production techniques• are washable and ironable, unlike some hardware-

enhanced clothing.

3.2. Knitted Stretch Sensors

The sensors used measure stretch from resistivechanges in knitted strips. Strips 10mm wide areprepared, typically a 10 meter length. The sensor gives alinearly increasing asymptotic resistance with stretch, upto almost maximum stretch. Stretches of over 50% arepossible with these strips. This curve is shown in Figure5.

Figure 5. Resistance of a 10mm wide100mm long knitted sensor strip as it isstretched from its 100mm length

3.3. Knitted Conductive Tracking

A knit containing a conductive fibre for the purpose oftransmitting an electrical signal we refer to as tracking.Knitted tracking is used in place of wiring to theterminals of the sensors. Conductive threads chosen fortheir stable properties are introduced to this knit. Thetracking used is 10 mm wide and exhibits a 3 ohmsresistance over 100 mm. This particular knit can bestretched by approximately 5%. When the sensor andtracking materials are slack the sensors exhibit aresistance 25 times greater than that of the tracking, andwhen stretched this ratio increases to 100 times. Thesensor and tracking material is shown in Figure 6.

Figure 6. A strip of stretch sensor knitwearand a 10 meter roll of knitted conductivetracking

Figure 7. The initial toile – sensor j acketprototype made from toiling fabric.Sensor strips are sewn onto toilingfabric and connected by wires

Resistance with stretch

0100200300400500600

100 110120 130

140150 160 length mm

koh

ms

3.4. Jacket Design

A jacket has been designed that positions 11 knittedsensor strips over the elbow and shoulder joints. In theinitial toile (the first cut of our garment using toilingfabric), six sensors are distributed about the right side.Right elbow movement is measured by one sensor alongthe joint. Three sensors measure the position andmovements of the right upper arm. Two of them are usedfor measuring (upper) arm-movements in the frontal(vertical) plane. They are positioned on top of theshoulder and at the armpit. A sensor strip on the upperback and shoulder measures position and movements ofthe upper arm in the transverse (horizontal) plane. Asensor positioned diagonally across the centre of the backdetects bending forward, with an additional sensorpositioned horizontally across the right breast, as shownin Figure 7.

During development only the sensor strips aredesigned into the jacket, with wires and crocodile clipstaking the place of the knitted tracking. The tracking hassimilar stable properties to the wires and this plays littlepart in the positioning of the sensors themselves.

After a second toiling, positioning the sensors andtracking on left and right sides, a bespoke sensor jacketwas manufactured to commercial standards (patterncutting, manufacture and materials). When gesture andarm position recognition was shown to be successful thepattern was graded up to a size large (chest 47.25 inch,120 cm), Figure 8.

3.5. Signal Processing

A wire umbilical attaches the sensor jacket to aanalogue to digital converter and a PC. A PicotechADC11 analogue to digital converter is used, giving 11channels, 11-bit precision and a sample rate of up to10kHz [10]. The sensor signals are used as an input to akinematic geometry model. This mathematical modelrepresents the mobility of the user’s arms and neglects allother aspects (e.g. mass distribution, force), [14]. Eacharm is simplified as being a two-link kinematic chain andthe shoulders are considered fixed in space. This model iscapable of detecting the position of the elbows and wristsof the wearer.

Each time the jacket is worn, it has to be calibrated bythe wearer because the users under clothing (shirt, t-shirt,etc.) and body shape (e.g. shoulder and chest breadth,length of the arms) influence the sensor readings. Duringthe calibration process the user is asked to put his armsdown, outstretched, forward and then up. The softwarecan detect these states automatically regardless of theusers body shape by using pre-calibrated ranges. These

Figure 8. The final sensor j ackets. Sensorsare shown in bl ack, and are c onnected ateach end by covered c onductivetracking

poses make the various sensors stretch and theinformation about the minima and maxima of eachsensor are used in the algorithms for detecting the anglesof the joints.

After the calibration process, demonstration softwareon the PC illustrates the positioning of the arms. Motionsand gestures are recognised and displayed in real-time,sampled at 50 Hz.

4. Hard & Soft Sensors

It may be possible to have a sensor garment thatdetects limb and upper body motion using only hardaccelerometers positioned throughout the garment. Abody suit using soft sensor fabric may also be able to

detect perambulatory activities such as walking andrunning. The merits of electronic and fabric sensors arehighly dependent upon their application, with theavailability and cost of the components highly significantfor current applications. Particular benefits of the sensorbadge are its small size and its suitability to be worn withany other clothing. The sensor jacket has the particularbenefits of being both socially acceptable and comfortableto wear.

5. Future Work

Work continues on identifying additional activitiesfrom the sensor badge. For example distinguishingtravelling in a car or by train.

Two potential applications of the sensor badge can beapplied to mobile phones as well as wearable computers.The first application is for using the sensor badge todetect whether you are driving in your car. If you are,your devices can detect this automatically and go to ahands-free mode. A second application is for people whowant to instruct their devices not to bother them whenit’s not convenient. For example, you might want toinstruct your devices not to bother you with unimportantmessages when you are active (e.g. running for a bus or atrain), or lying down (resting, sleeping). As soon as thesystem thinks you are ready, it will give you the messagesit has saved.

To complement the sensor-jacket, trousers and shortsare being considered to allow for full-body sensing. Awireless to PC link and on-jacket data recording are alsobeing investigated.

Future interfaces are to be expected to combine severalmodalities (e.g. gesture and speech) in order to providemore natural ways of communicating without the use of akeyboard or a mouse, [4]. Context-aware devices like thesensor badge and smart clothing like the sensor jacketcan be used for future multimodal human-computerinterfaces, [4, 11], which can be used at any time and atany place.

6. Conclusions

A variety of sensing technologies have been developedfor wearable devices to enable limited awareness of auser's ongoing activities and context. One sensor solutionis composed of hard electronics, the other soft fabrics.

Commercially available electronics can be readilyassembled into a belt wearable sensing device. Howeverthe development of software (or other means) forinterpreting the sensor data into high level abstractions isnot trivial. The sensor badge described here uses twoaccelerometers and software running on a PIC

microcontroller, outputting identified activities to otherwearable devices through an RS-232 port.

The sensor jacket includes knitted fabrics which haveelectrical properties suited for either sensing stretch orfor use as non-sensing conductive tracking. The first toileof this jacket measures movement through six sensors onthe right hand side and back and the final jackets uses 11sensors over both sides. A connections port on the jacketcan be connected to other wearable devices supplyingdata regarding the current limb movement and bodyposition of the wearer.

Both the belt worn sensor badge and the sensor jacketare suitable for continuous wearing. The benefit for theuser of wearing such sensors is in the situationalawareness their wearable computer gains. The interfacebetween wearable computer and wearer can besympathetic to the user's activities, which can alsomediate wearable software applications.

Acknowledgements

With particular thanks to Dr. Frances Geesin, seniorresearch fellow at the London College of Fashion, for hercontinuing support and original work initiated at theRoyal College of Art, London, on the technical use ofindustrial and conductive fibres and cloth.

Thanks also to Philippa Wagner MA RCA (textiledesigner), Juliette Allen (pattern cutting, machining) andCanan Geyik (production assistant) for the production ofthe sensor jackets.

References[1] F. Geesin, “Interactive Fabrics Show Potential”,Technical

Textiles International, Elsevier Science Ltd., October1996.

[2] S. Mann, “Smart Clothing: The Shift to WearableComputing”,Communications of the ACM, Vol. 39, No. 8,August 1996, pp. 23–24.

[3] S. Mann, “Smart Clothing: The Wearable Computer andWearCam.”,<http://www.wearcam.org/personaltechnologies/>. Lastvisited: Aug. 4, 1999. Also appeared inPersonalTechnologies, Vol. 1, No. 1, Springer, 1997, March 1997.

[4] J. Nielsen, Noncommand user interfaces ,<http://www.useit.com/papers/noncommand.html>. Lastvisited: Aug. 4, 1999. Revised version of the paper thatappeared inCommunications of the ACM, Vol. 36, No. 4,April 1993, pp. 83–99.

[5] A. Pentland, “Smart Rooms, Smart Clothes”,ProceedingsFourteenth International Conference on PatternRecognition, pp. 949–953, 1998.

[6] R.W. Picard,Affective Computing, MIT Press, Cambridge,MA, 1997.

[7] RW. Picard and J. Healey, “Affective Wearables”,Proceedings of the First International Symposium onWearable Computers, IEEE, pp. 90–97. October 13–14,1997.

[8] R.E. Post and M. Orth, “Smart Fabric, or WashableComputing.”,<http://www.media.mit.edu/physics/people/rehmi/cloth/>.Last visited: Aug. 4, 1999. Appears abridged as “SmartFabric, or "Wearable Clothing"”,Proceedings of the FirstIEEE International Symposium on Wearable Computers,Cambridge, MA, October 13–14, 1997, pp. 167–168(abridged).

[9] B. Rhodes, Context-Aware Computing (or, why contextneeds wearables and wearables need context),<http://wearable.www.media.mit.edu/projects/wearables/context.html>. Last visited: Aug. 4, 1999.

[10] Pico Technology Limited, <http://www.picotech.com>,Cambridge, UK.

[11] R. Sharma, V.I. Pavlovic, T.S. Huang, “TowardMultimodal Human-Computer Interface”,Proceedings ofthe IEEE, Vol. 86, No. 5, pp. 853–869, May 1998.

[12] W. van de Velde, TEA: Technology for EnablingAwareness,<http://www.starlab.org/bits/smartdevices/tea.html>.Official TEA-site: <http://www.omega.it/tea/>. Lastvisited: Aug. 4, 1999.

[13] P. Watkins, “Get Smart.”,Drapers Record Textiles, pp.33–34, September 1998.

[14] V.M. Zatsiorsky, Kinematics of Human Motion, HumanKinetics, 1997.