Activity Recognition from Accelerometer Data

Nishkam Ravi and Nikhil Dandekar and Preetham Mysore and Michael L. Littman
Department of Computer Science
Rutgers University
Piscataway, NJ 08854
{nravi,nikhild,preetham,mlittman}@cs.rutgers.edu

Abstract Bussmann et al. 2001). The most successful and exhaus-

tive work in this regard is that of Bao & Intille (2004). In
Activity recognition fits within the bigger framework their experiments, subjects wore 5 biaxial accelerometers on
of context awareness. In this paper, we report on our different body parts as they performed a variety of activi-
efforts to recognize user activity from accelerometer
ties like walking, sitting, standing still, watching TV, run-
data. Activity recognition is formulated as a classifica-
tion problem. Performance of base-level classifiers and ning, bicycling, eating, reading etc. Data generated by the
meta-level classifiers is compared. Plurality Voting is accelerometers was used to train a set of classifiers, which
found to perform consistently well across different set- included decision trees (C4.5), decision tables, naive Bayes
tings. classifier and nearest-neighbor algorithm found in the Weka
Machine Learning Toolkit (Witten & Frank 1999). Decision
tree classifiers showed the best performance, recognizing ac-
Introduction tivities with an overall accuracy of 84%.
We have attempted to recognize activities using a single
A triaxial accelerometer is a sensor that returns a real val- triaxial accelerometer worn near the pelvic region. Activity
ued estimate of acceleration along the x, y and z axes from recognition is formulated as a classification problem. In ad-
which velocity and displacement can also be estimated. Ac- dition to analyzing the performance of base-level classifiers
celerometers can be used as motion detectors (DeVaul & (Bao & Intille 2004), we have studied the effectiveness of
Dunn 2001) as well as for body-position and posture sens- meta-level classifiers (such as boosting (Freund & Schapire
ing (Foerster, Smeja, & Fahrenberg 1999). Apple’s iLife 1996), bagging (Breiman 1996), plurality voting, stacking
Fall Detection sensor which embeds an accelerometer and a using ODTs, and stacking using MDTs (Todorovski & Dze-
microcomputer to detect falls, shocks or jerky movements is roski 2003)) in improving activity recognition accuracy. We
a good example. Active research is being carried out in ex- have tried to answer the following questions: (1) Which are
ploiting this property for determining user context (Randell the best classifiers for recognizing activities; is combining
& Muller 2000). Advances in miniaturization will permit classifiers a good idea? (2) Which among the selected fea-
accelerometers to be embedded within wrist bands, bracelets tures/attributes are less important than others? (3) Which
and belts and to wirelessly send data to a mobile computing activities are harder to recognize?
device that can use the signals to make inferences. User con-
In the following sections, we describe our data collection
text can be utilized for ambulatory monitoring (Makikawa et
methodology and our approach to recognize activity from
al. 2001; Foerster, Smeja, & Fahrenberg 1999) and is the key
accelerometer data, followed by results.
to minimizing human intervention in ubiquitous computing
applications.
Making devices aware of the activity of the user fits into
Data Collection
the bigger framework of context awareness. Ubiquitous Data from the accelerometer has the following attributes:
computing is centered around the idea of provisioning ser- time, acceleration along x axis, acceleration along y axis and
vices to the user in a seamless manner. Provisioning ser- acceleration along z axis. We used a triaxial accelerometer
vices to the user based on his location and/or activity is an CDXL04M3 marketed by Crossbow Technologies, which is
active research area. While the research thrust is on au- capable of sensing accelerations up to 4G with tolerances
tomatically determining user location (Want et al. 1992; within 2%. The accelerometer is mounted on a hoarder
Harter & Hopper 1994; Priyantha, Chakraborty, & Balakr- board (which samples at 50Hz), as shown in Figure 1. The
ishnan 2000), determining user activity is getting a lot of accelerometer was worn near the pelvic region while the
attention lately. Attempts have been made at recognizing subject performed activities. The data generated by the ac-
user activity from accelerometer data (Lee & K.Mase 2002; celerometer was transmitted to an HP iPAQ (carried by the
subject) wirelessly over Bluetooth. The Bluetooth transmit-
Copyright
c 2005, American Association for Artificial Intelli- ter is wired into the accelerometer. A Bluetooth enabled HP
gence (www.aaai.org). All rights reserved. iPAQ running Microsoft Windows was used. The Windows’
 Figure 1: Data Collection Apparatus

Figure 2: Data Lifecycle

Bluetooth library was used for programming Bluetooth. The Feature extraction
data was then converted to ASCII format using a Python Features were extracted from the raw accelerometer data us-
script. ing a window size of 256 with 128 samples overlapping
We collected data for a set of eight activities: between consecutive windows. Feature extraction on win-
• Standing dows with 50% overlap has demonstrated success in previ-
ous work (Bao & Intille 2004). At a sampling frequency
• Walking of 50Hz, each window represents data for 5.12 seconds. A
• Running window of several seconds can sufficiently capture cycles
in activities such as walking, running, climbing up stairs
• Climbing up stairs etc. Furthermore, a window size of 256 samples enabled
fast computation of FFTs used for one of the features.
• Climbing down stairs
Four features were extracted from each of the three axes
• Sit-ups of the accelerometer, giving a total of twelve attributes. The
features extracted were:
• Vacuuming
• Mean
• Brushing teeth.
• Standard Deviation
The activities were performed by two subjects in multiple
rounds over different days. No noise filtering was carried • Energy
out on the data. • Correlation.
Label-generation is semi-automatic. As the users per- The usefulness of these features has been demonstrated
formed activities, they were timed using a stop watch. The in prior work (Bao & Intille 2004). The DC component of
time values were then fed into a Perl script, which labeled the signal over the window is the mean acceleration value.
the data. Acceleration data collected between the start and Standard deviation was used to capture the fact that the range
stop times were labeled with the name of that activity. Since of possible acceleration values differ for different activities
the subject is probably standing still or sitting while he such as walking, running etc.
records the start and stop times, the activity label around The periodicity in the data is reflected in the frequency
these times may not correspond to the actual activity per- domain. To capture data periodicity, the energy feature was
formed. calculated. Energy is the sum of the squared discrete FFT
Figure 2 shows the lifecycle of the data. To minimize mis- component magnitudes of the signal. The sum was divided
labeling, data within 10 s of the start and stop times were by the window length for normalization. If x1 , x2P, ... are the
discarded. Figure 3 shows the x-axis readings of the ac- |w|
|xi |2
celerometer for various activities. FFT components of the window then, Energy = i=1
|w| .
 Figure 3: X-axis readings for different activities

Correlation is calculated between each pair of axes as the than that of base-level classifiers, base-level-classifiers are
ratio of the covariance and the product of the standard devia- known to outperform meta-level-classifiers on several data
tions corr(x, y) = cov(x,y)
σx σy . Correlation is especially useful
sets. One of the goals of this work was to find out if com-
for differentiating among activities that involve translation bining classifiers is indeed the right thing to do for activity
in just one dimension. For example, we can differentiate recognition from accelerometer data, which to the best of
walking and running from stair climbing using correlation. our knowledge, has not been studied earlier.
Walking and Running usually involve translation in one di- Meta-level classifiers can be clustered into three frame-
mension whereas Climbing involves translation in more than works: voting (used in bagging and boosting), stack-
one dimension. ing (Wolpert 1992; Dzeroski & Zenko 2004) and cascad-
ing (Gama & Brazdil 2000). In voting, each base-level clas-
Data Interpretation sifier gives a vote for its prediction. The class receiving the
most votes is the final prediction. In stacking, a learning
The activity recognition algorithm should be able to recog- algorithm is used to learn how to combine the predictions
nize the accelerometer signal pattern corresponding to every of the base-level classifiers. The induced meta-level clas-
activity. Figure 3 shows the x-axis readings for the different sifier is then used to obtain the final prediction from the
activities. It is easy to see that every activity does have a predictions of the base-level classifiers. The state-of-the-art
distinct pattern. We formulate activity recognition as a clas- methods in stacking are stacking with class probability dis-
sification problem where classes correspond to activities and tributions using Meta Decision Trees (MDTs) (Todorovski
a test data instance is a set of acceleration values collected & Dzeroski 2003), stacking with class probability distribu-
over a time interval and post-processed into a single instance tions using Ordinary Decision Trees (ODTs) (Todorovski &
of {mean, standard deviation, energy, correlation}. We eval- Dzeroski 2003) and stacking using multi-response linear re-
uated the performance of the following base-level classifiers, gression (Seewald 2002). Cascading is an iterative process
available in the Weka toolkit: of combining classifiers: at each iteration, the training data
• Decision Tables set is extended with the predictions obtained in the previ-
• Decision Trees (C4.5) ous iteration. Cascading in general gives sub-optimal results
compared to the other two schemes.
• K-nearest neighbors
To have a near exhaustive set of classifiers, we chose the
• SVM following set of classifiers: Boosting, Bagging, Plurality
• Naive Bayes. Voting, Stacking with Ordinary-Decision trees (ODTs) and
We also evaluated the performance of some of the state- Stacking with Meta-Decision trees (MDTs).
of-the-art meta-level classifiers. Although the overall per- • Boosting (Meir & Ratsch 2003) is used to improve the
formance of meta-level classifiers is known to be better classification accuracy of any given base-level classifier.
 Boosting applies a single learning algorithm repeatedly All the above meta-level classifiers, except MDTs, are
and combines the hypothesis learned each time (using available in the Weka toolkit. We downloaded the source
voting), such that the final classification accuracy is im- code for MDTs and compiled it with Weka.
proved. It does so by assigning a certain weight to each Alternate approaches to activity recognition include use
example in the training set, and then modifying the weight of Hidden Markov Models(HMMs) or regression. HMMs
after each iteration depending on whether the example would be useful in recognizing a sequence of activities to
was correctly or incorrectly classified by the current hy- model human behavior. In this paper, we concentrate on rec-
pothesis. Thus final hypothesis learned can be given as ognizing a single activity. Regression is normally used when
T a real-valued output is desired, otherwise classification is a
natural choice. Signal processing can be helpful in automat-
X
f (x) = αt ht (x),
t=1
ically extracting features from raw data. Signal processing,
however, is computationally expensive and not very suitable
where αt denotes the coefficient with which the hypothe- for resource constrained and battery powered devices.
sis ht is combined. Both αt and ht are learned during the
Boosting procedure. (Boosting is available in the Weka
toolkit.)
Results
All the base-level and meta-level classifiers mentioned
• Bagging (Breiman 1996) is another simple meta-level above were run on data sets in four different settings:
classifier that uses just one base-level classifier at a time.
It works by training each classifier on a random redistri- Setting 1: Data collected for a single subject over
bution of the training set. Thus, each classifier’s training different days, mixed together and cross-validated.
set is generated by randomly drawing, with replacement,
N instances from the original training set. Here N is the Setting 2: Data collected for multiple subjects over
size of the original training set itself. Many of the origi- different days, mixed together and cross-validated.
nal examples may be repeated in the resulting training set
while others may be left out. The final bagged estimator, Setting 3: Data collected for a single subject on one day
hbag (.) is the expected value of the prediction over each of used as training data, and data collected for the same subject
the trained hypotheses. If hk (.) is the hypothesis learned on another day used as testing data.
for training sample k,
M Setting 4: Data collected for a subject for one day used
1 X
hbag (.) = hk (.). as training data, and data collected on another subject on
M another day used as testing data.
k=1

• Plurality Voting selects the class that has been predicted

by a majority of the base-level classifiers as the final pre- Data for settings 1 and 2 is independently and identically
dicted class. There is a refinement of the plurality vote al- distributed (IID), while that for settings 3 and 4 is not. Run-
gorithm for the case where class probability distributions ning classifiers on both IID and non-IID data is important
are predicted by the base-level classifiers. In this case, the for a thorough comparison.
probability distribution vectors returned by the base-level We did a 10-fold cross-validation for each of the clas-
classifiers are summed to obtain the class probability dis- sifiers in each of the above settings. In a 10-fold cross-
tribution of the meta-level voting classifier: validation, the data is randomly divided into ten equal-sized
pieces. Each piece is used as the test set with training done
1 X on remaining 90% of the data. The test results are then av-
PM L (x) = Pc (x).
|C| eraged over the ten cases.
c∈C
Table 1 shows the classifier accuracies for the four set-
• Stacking with ODTs is a meta-level classifier that uses the tings respectively. It can be seen that Plurality Voting per-
results of the base-level classifiers to predict which class forms the best in the first three settings, and second best in
the given instance belongs to. The input to the ODTs are the fourth setting. Boosted/Bagged Naive Bayes, SVM and
the outputs of the base-level classifiers i.e. class probabil- kNN perform consistently well for the four settings. Boosted
ity distributions (CPDs) — pCj (ci |x), as predicted over SVM outperforms the other classifiers by a good margin in
all possible class values ci , by each of the base-level clas- the fourth setting. In general, meta-level classifiers perform
sifiers Cj . The output of the stacked ODT is the class- better than base level classifiers.
prediction for the given test instance. The scatter-plot in Figure 4 shows the correlation in the
• Stacking with MDTs (Todorovski & Dzeroski 2003) performance of each classifier on IID and non-IID data. Val-
learns a meta-level decision tree whose leaves consist of ues on x-axis correspond to the accuracy of classifiers aver-
each of the base level classifiers. Thus, instead of speci- aged over settings 1 and 2, while the values on y-axis corre-
fying which class the given test instance belongs to, as in spond to the accuracy of classifiers averaged over settings 3
a stacked ODT, an MDT specifies which classifier should and 4. Plurality Voting has the best performance correlation
be used to optimally classify the instance. The MDTs are (0.78).
also induced by a meta-level data set that consists of the Plurality voting combines multiple base-level classifiers
CPDs — pCj (ci |x). as opposed to boosting and bagging which use a single
Table 1: Accuracy of classifiers for the four different settings Figure 4: Performance correlation for IID and non-IID data

100
Classifier Accuracy(%)
Setting1 Setting2 Setting3 Setting4
Naive Bayes(NB) 98.86 96.69 89.96 64.00
90
Boosted NB 98.86 98.71 89.96 64.00
Bagged NB 98.58 96.88 90.39 59.33 Stacking (MDTs)
SVM 98.15 98.16 68.78 63.00 Boosted NB
Plurality Voting
Boosted SVM 99.43 98.16 67.90 73.33 80
Naive Bayes(NB)

Non−IID Data Accuracy

Bagged SVM 98.15 98.53 68.78 60.00 Bagged NB
kNN 98.15 99.26 72.93 49.67 Stacking (ODTs)

Boosted kNN 99.15 99.26 72.93 49.67 70 Bagged DTr Boosted SVM
Bagged kNN 99.15 99.26 70.52 46.67 Boosted DTr
Decision Tree (DTr)
Decision Table(DT) 92.45 91.91 55.68 46.33 Bagged SVM
Boosted DT 97.86 98.53 55.68 46.33 SVM
60 Boosted kNN
Bagged DT 93.30 94.85 55.90 46.67 Decision Bagged DT Bagged kNN
Decision Tree(DTr) 97.29 98.53 77.95 57.00 Table (DT) kNN

Boosted DTr 98.15 98.35 77.95 57.00

Bagged DTr 97.29 95.22 78.82 63.33 50
Plurality Voting 99.57 99.82 90.61 65.33 Boosted DT
Stacking (MDTs) 99.00 99.26 89.96 64.00
Stacking (ODTs) 98.86 98.35 84.50 64.00 40
90 92 94 96 98 100 102
IID Data Accuracy

base-level classifier. Voting can therefore outperform boost-

ing/bagging on certain datasets. From our results, it is clear
ities with fairly high accuracy, we did not explore the possi-
that plurality voting does better than boosting and bagging
bility of adding more features/attributes.
consistently, although by a small margin. Plurality voting
outperforming MDTs and ODTs is not very intuitive. A In order to find out which activities are relatively harder
careful analysis however explains this finding. (Todorovski to recognize, we manually analyzed the confusion matrices
& Dzeroski 2003) showed that MDTs and ODTs usually obtained for different data sets for different classifiers. The
outperform plurality voting on datasets where the error di- confusion matrix gives information about the actual and pre-
versity of base-level classifiers is high. Plurality Voting on dicted classifications done by the classifiers. The confusion
the other hand outperforms MDTs and ODTs on datasets matrix in Table 2 is a representative of the commonly ob-
where base-level classifiers have high error correlation (low served behavior in setting 3. It shows that climbing stairs
error diversity), the cutoff being approximately 50%. The er- up and down are hard to tell apart. Brushing is often con-
ror correlation between a pair of classifiers is defined as the fused with standing or vacuuming and is in general hard to
conditional probability that both classifiers make the same recognize.
error given one of them makes an error:
Conclusions and Future work
φ(Ci , Cj ) = p(Ci (x) = Cj (x)|Ci (x) 6= c(x)∨Cj (x) 6= c(x)),
We found that activities can be recognized with fairly high
where Ci (x) and Cj (x) are the predictions of classifiers Ci accuracy using a single triaxial accelerometer. Activities
and Cj for a given instance x and c(x) is the true class of x. that are limited to the movement of just hands or mouth (e.g
We calculated error correlation between all the base-level brushing) are comparatively harder to recognize using a sin-
classifiers (which is defined as the average of pairwise er- gle accelerometer worn near the pelvic region. Using meta-
ror correlations) for all the four settings. The error corre- classifiers is in general a good idea. In particular, combin-
lation came out to approximately 52%. This high value of ing classifiers using Plurality Voting turns out to be the best
error correlation may explain why Plurality Voting does bet- classifier for activity recognition from a single accelerome-
ter than MDTs and ODTs on accelerometer data. ter, consistently outperforming stacking. We also found that
We wanted to find out which features/attributes among the energy is the least significant attribute.
selected ones are less important than the others. To this end, An interesting extension would be to see whether ”short
we ran the classifiers on the data with one attribute removed activities” (e.g opening the door with a swipe card) can be
at a time. Table 3 shows the average number of misclassi- recognized from accelerometer data. These could be instru-
fications for data of setting 2, with one attribute dropped at mental in modeling user behavior. Along similar lines, it
a time. The Energy attribute turns out to be the least sig- would be interesting to find out how effective an ontology
nificant. There is no significant change in accuracy when of activities could be in helping classify hard-to-recognize
Energy attribute is dropped. Since we could recognize activ- activities.
 Table 2: Representative Confusion Matrix for Setting 3
Activity Classified As
Standing Walking Running Stairs Up Stairs Down Vacuuming Brushing Situps
Standing 63 0 0 0 0 0 0 0
Walking 0 44 0 1 0 0 0 0
Running 0 0 17 16 20 0 0 0
Stairs Up 0 0 0 9 12 0 0 0
Stairs Down 0 0 0 19 0 0 0 0
Vacuuming 0 0 0 0 0 45 0 0
Brushing 18 0 0 0 0 15 0 0
Situps 0 0 0 0 0 7 0 24

Makikawa, M.; Kurata, S.; Higa, Y.; Araki, Y.; and Tokue, R.
Table 3: Effect of dropping an attribute on classification ac- 2001. Ambulatory monitoring of behavior in daily life by ac-
curacy celerometers set at both-near-sides of the joint. In Proceedings of
Average no. of misclassi- MedInfo, 840–843.
Attribute
fications Meir, R., and Ratsch, G. 2003. An introduction to boosting and
Drop None 14.05 leveraging. 118–183.
Drop Mean 21.83 Priyantha, N. B.; Chakraborty, A.; and Balakrishnan, H. 2000.
Drop Standard Deviation 32.44 The cricket location-support system. In Mobile Computing and
Drop Energy 14.72 Networking, 32–43.
Drop Correlation 28.38
Randell, C., and Muller, H. 2000. Context awareness by analysing
accelerometer data. In MacIntyre, B., and Iannucci, B., eds., The
Fourth International Symposium on Wearable Computers, 175–
176. IEEE Computer Society.
Acknowledgments
Seewald, A. K. 2002. How to make stacking better and faster
Our sincere thanks to Amit Gaur and Muthu Muthukrishnan while also taking care of an unknown weakness. In Proceedings
for lending us the accelerometer. of the Nineteenth International Conference on Machine Learning,
554–561. Morgan Kaufmann Publishers Inc.
References Todorovski, L., and Dzeroski, S. 2003. Combining classifiers
Bao, L., and Intille, S. S. 2004. Activity recognition from user- with meta decision trees. Machine Learning 223–249.
annotated acceleration data. In Proceceedings of the 2nd Interna- Want, R.; Hopper, A.; Falcao, V.; and Gibbons, J. 1992. The
tional Conference on Pervasive Computing, 1–17. active badge location system. Technical Report 92.1, ORL, 24a
Breiman, L. 1996. Bagging predictors. Machine Learning 123– Trumpington Street, Cambridge CB2 1QA.
140. Witten, I., and Frank, E. 1999. Data Mining: Practical Machine
Bussmann, J.; Martens, W.; Tulen, J.; Schasfoort, F.; van den Learning Tools and Techniques with Java Implementations. Mor-
Bergemons, H.; and H.J.Stam. 2001. Measuring daily behavior gan Kauffman.
using ambulatory accelerometry: the activity monitor. Behavior Wolpert, D. H. 1992. Stacked generalization. Neural Networks
Research Methods, Instruments, and Computers 349–356. 241–259.
DeVaul, R., and Dunn, S. 2001. Real-Time Motion Classifi-
cation for Wearable Computing Applications. Technical report,
MIT Media Laboratory.
Dzeroski, S., and Zenko, B. 2004. Is combining classifiers with
stacking better than selecting the best one? Machine Learning
255–273.
Foerster, F.; Smeja, M.; and Fahrenberg, J. 1999. Detection of
posture and motion by accelerometry: a validation in ambulatory
monitoring. Computers in Human Behavior 571–583.
Freund, Y., and Schapire, R. E. 1996. Experiments with a
new boosting algorithm. In International Conference on Machine
Learning, 148–156.
Gama, J., and Brazdil, P. 2000. Cascade generalization. Machine
Learning 315–343.
Harter, A., and Hopper, A. 1994. A distributed location system
for the active office. IEEE Network 8(1).
Lee, S., and K.Mase. 2002. Activity and location recognition
using wearable sensors. IEEE Pervasive Computing 24–32.