Nineteenth National Conference on Artificial Intelligence (AAAI-04), July 25-29, 2004, San Jose, California

Common Sense Data Acquisition for Indoor Mobile Robots

Rakesh Gupta and Mykel J. Kochenderfer∗

Honda Research Institute USA, Inc.
800 California Street, Suite 300
Mountain View, CA 94041
rgupta@[Link], [Link]@[Link]

Abstract This paper describes the Open Mind Indoor Common

Sense (OMICS) project that became publicly available at the
Common sense knowledge can be efficiently collected from beginning of August 20034. In the next section we describe
non-experts over the web in a similar fashion to the Open the knowledge base framework and templates used to cap-
Mind family of distributed knowledge capture projects. We ture data. We then report on our results and experiences with
describe the collection of common sense data through the
Open Mind Indoor Common Sense (OMICS) website. We
working with online contributors. The next section of the
restrict the domain to indoor home and office environments paper discusses two applications that use this data. We dis-
to obtain dense knowledge. The knowledge was collected cuss inference based on anticipating the desires of the users.
through sentence templates that were generated dynamically We also describe a room-labeling application based on sim-
based on previous user input. Entries were converted into ulated object recognition and simulated room and object la-
relations and saved into a database. We discuss the results bels given by a user. Finally, we discuss our conclusions and
of this online collaborative effort and describe two applica- future work.
tions of the collected data to indoor mobile robots. We dis-
cuss active desire selection based on current beliefs and com-
mands and a room-labeling application based on probability
Data Collection Methods
estimates from the common sense knowledge base. In this section we first describe the framework of the knowl-
edge base and the relations necessary to capture those types
of common sense most useful to an indoor robot. We then
Introduction describe how we built a website to capture this data in a
user-friendly manner and how we converted the data into
The objective of this research is to enhance the intelligence machine-understandable relations.
of mobile robots so that they can autonomously accomplish
tasks in a home or office environment. For these tasks, the Knowledge Representation
robots must posses some common sense including knowl-
edge about human desires, objects and their locations, and The framework of this work is object-centric. Robot activ-
causality. Since common sense does not require expert ities involve perceiving the environment and acting upon it.
knowledge, the data may be collected as part of a public What the robot knows about the environment must include
online collaborative effort over the Internet. what objects are present and their state. The robot must ma-
nipulate these objects in such a way so as to put them in
Distributed online knowledge acquisition projects, such
some desired state to accomplish its goals. Hence, every-
as those associated with the Open Mind Initiative (Stork
thing that the robot knows about the world and can do in the
1999; 2000), have become quite popular. The Open Mind
world is grounded in objects and their properties (or state).
Common Sense project, led by Push Singh at the MIT Media
The robot can observe properties of objects in its vicinity
Lab, has accumulated a corpus of 700,000 pieces of knowl-
and it can perform actions that change the properties of ob-
edge from 14,000 users (as of January 2004) over the past
jects. In this system, a statement is a pair φ = (o, p) where
three years1 . Other projects such as Open Mind Word Ex-
o is some object and p is an adjective describing the prop-
pert2 and Open Mind 1001 Questions3 , have also been suc-
erty. Statements may be thought of as assertions about the
cessful.
property of an object in the world or actions to be taken on
∗
Currently at Institute of Perception, Action and Behaviour, an object to achieve a particular effect (actions are referred
School of Informatics, University of Edinburgh, Edinburgh, United to by the effect they achieve.) For example, the statement
Kingdom. (cup-of-coffee, hot) can mean “a cup of coffee is hot” or rep-
Copyright
c 2004, American Association for Artificial Intelli- resent the action “make a cup of coffee hot.” Using the same
gence ([Link]). All rights reserved. notation allows us to make connections between beliefs, de-
1
[Link] sires, and intentions.
2
[Link]
3 4
[Link] [Link]

1
 Figure 1: A template sentence.

Our representation allows us to capture such common

sense knowledge as: (o1 , p1 ) causes (o2 , p2 ). For exam- Figure 2: A word sense disambiguation form.
ple, the statement (fan, on) causes (room, cool). We also
wish to capture knowledge about human desires such as
(o1 , p1 ) indicating the human desire (o2 , p2 ). For example, from a collection of stock photography.
the perception (cup-of-coffee, cold) indicates that the desire
(cup-of-coffee, hot) be fulfilled. Statements In the ‘statements’ activity, the user is
At any given point in time, the robot observes a set of prompted with a question such as, “You often want a fan
statements that are true and can execute a set of statements. to be .” This activity pairs objects with properties in
Using the common sense knowledge about causality and de- the knowledge base. The objects that appear in these sen-
sires as previously described, the general problem is to de- tence templates come from the objects entered by users in
cide which statements to execute in order to achieve per- the other activities such as the ‘objects’ activity.
ceived goals. Uses This activity associates objects with their uses. For
Indeed, this object-centric representation is limited. It example, the user might be prompted with the form, “A
cannot express everything that first order logic with a large hanger is used to .” Again the objects come from
array of predicates can represent. In addition to making in- user input.
ference simpler, this “object-centric” representation makes
it much easier to collect data from sentence templates. Causes This activity captures causality. For exam-
ple, a form might ask, “A computer is off when
Activity Sentence Templates a is .” If the user enters a new object or a
We need some way to convert the common sense in the new object-property pair, it will be entered into the object or
minds of non-expert users into relations in a knowledge statement table. The object and property that makes up the
base. Following the style of the Open Mind Common Sense first part of the sentence is formed dynamically by selecting
website, we decided to use sentence templates. Users are a random object from the knowledge base.
prompted to fill in the blanks of sentences with words, as
shown in figure 1. Desires This activity helps the robot determine what needs
Once a user logs on with their account, they are presented to be done in various situations. A template form might
with a random activity. After a random number of entries ask, “You might want a fan to be blowing if you notice that
for a particular activity, the system prompts them with a new your has become .”
activity. Users may also manually switch between the activ- Locations This activity associates objects with the rooms
ities. where they are typically found. For example, the user might
Different activities capture different types of knowledge. be prompted with, “A room where you generally find a din-
Below is a summary of some of the activities: ner table is the .”
Objects In this activity, the user is asked to identify the
Proximity This activity associates objects with each other
types of objects commonly found in a home or office. The
based on proximity. A sample form would be, “You gener-
user may be prompted to type in an object name that comes
ally find a frying pan in the same room as a .”
to mind or simply label an image of an object. It is important
to allow a user to simply type in any indoor object that comes Senses This activity disambiguates the intended sense of
to mind because we want to include all relevant objects in the various objects entered into the database by other users. Fig-
database even if we do not have their picture. ure 2 shows a sample form. The objects to disambiguate are
Image labeling can link multiple labels to the same object selected from previous user entries and the senses are from
(e.g. the labels “sofa” and “couch” might be both associated WordNet (Miller 1995).
with “[Link]”). The images themselves can be used for
training the object recognition system of the robot. When People This one describes the activities of people in a
the website became public, the database initially contained home or office (e.g. People eat food when they are hungry).
a set of over 400 images of indoor objects selected by hand The template form is, “People when they .”
Paraphrase This activity tries to capture multiple ways of
interacting with the robot in natural language to accomplish
a task. A sample template is “Another way to say heat the
food in the microwave is .”
Tasks This activity tries to capture the steps required to
accomplish a task like making coffee, answering the phone
etc. We prompt the user with 7 short natural language steps
to accomplish the task. A sample template is “The task water
indoor plants involves the steps: .”
Generalization This activity tries to generalize upon pre-
vious entries in the knowledge base. Instead of having to
individually specify that bananas, oranges, apples, etc. are
commonly found in the kitchen, it would be useful to know
that all kinds of fruit are found in the kitchen. If we know
that fruit is found in the kitchen, we can use the WordNet
hierarchy to infer that all of the hyponyms of fruit are found
in the kitchen. An example of a prompt for this activity is
the following: “Are all types of writing implement (a gener-
alization of ‘marker’) commonly found in the study?”
Freeform This activity allows users to type in any form of
common sense knowledge that might not be captured by any
of the other activities. Although it is quite difficult to convert
freeform sentences into useful relations, it provides us with
a sense of the types of knowledge the general public would
like an indoor robot to understand. Analysis of freeform Figure 3: The review form used for administrators to com-
sentences will later lead to the creation of new activities. mit, uncommit, or reject entries.

Data Quality Review

It is important that there be some way of ensuring data qual- Mind projects have used similar contests to help motivate
ity since the data (such as names of objects and their proper- submissions. Winners were listed on the front page of the
ties) are used to generate new sentence templates. Sentence site.
templates containing misspelled objects or objects that do
not appear in a home or office environment would propagate Observations
errors in the knowledge base. The OMICS site was publicly announced on August 5th and
The completed sentence templates are stored in the the first t-shirt prize was awarded August 6th. A signifi-
database as raw sentences pending administrator review (see cant portion of the entries were submitted within two days
figure 3). It generally takes an administrator roughly half of the announcement. Prizes were also awarded on August
a minute to scan through a page with fifteen submissions. 12, August 19, and August 26. In general, submissions were
Once an administrator approves a set of entries, they are greater close to contest deadlines.
parsed into relations immediately. There is currently no The quality of the data was actually quite good. About
need for part-of-speech tagging or lemmatization (as with 10% of the submissions were rejected. Entries that were
the Open Mind Common Sense project) since the sentence rejected tended to fall within one of the following categories:
templates are structured and designed in such a way that they • Misspelling: e.g. “A room where you generally find a
implicitly cue the user as to what part-of-speech and tense exercise bike is the bym.”
they should use.
• Unclear or loose statements: e.g. “People cry when they
Data Collection Results can’t get it.”
To advertise the Open Mind Indoor Common Sense website, • Outside the scope of home and office environments: e.g.
a message was sent to the Open Mind Initiative mailing list “A trap is set when a predator is hunting.”
on August 5th. With no other direct advertising, within three • Nonsense: e.g. “You generally want a green light to be
weeks we had 190 users and 18,000 submissions with about addressed to you.”
17,000 of them accepted. As of March 2004 we have over
• Inappropriate: e.g. suggestive or obscene
400 users with 29,000 submissions with over 26,000 of them
accepted. The ‘causes’ activity had the highest rejection rate. De-
We have had two weekly contests (lasting four weeks ciding how one object affects another object proved to be
each) in August 2003 and February 2004 where the top con- difficult for some users. Interestingly, almost all word sense
tributor was awarded an Open Mind t-shirt. Other Open activities were answered correctly. Even users that entered
 Activity Count observed by robot
Objects 5804
Uses 3517 (coke-can, warm)
Locations 3400
Statements 3394 indicates
Proximity 2547 action taken by robot desire
Freeform 1696 causes
People 1667 (coke-can, refrigerated) (coke-can, chilled)
Desires 1792
Causes 1558
Senses 2349 Figure 5: Example of using the OMICS knowledge base for
Generalization 718 active desire selection.
Paraphrase 351
Tasks 120
Images 55 I still don’t really know what to put in the freeform cat-
egory, or what needs to be worked through to be the
Figure 4: The number of submissions for each activity. most help in the project (I’m guessing something like
computer & mouse gets overused, while other objects
go ignored?).
rogue data in other activities generally entered the correct The general tone of the e-mails we received were positive
sense of the words in the ‘sense’ activity. and indicated a desire by the users to see the project succeed
Users that appeared to have hostile intentions, indicated in making robots more intelligent.
by sentence completions that were of a crude or sexual na-
ture, also submitted useful data. Surprisingly, a few users Data Applications
that might be classified as malicious were among the top The data collected from the Open Mind family of projects
contributors of good data. have been applied to a wide variety of problems. Open Mind
Figure 4 shows the ranking of the various activities and Word Expert has been applied to word sense disambigua-
the number of submissions. Not surprisingly, users spent tion (Liu, Lieberman, & Selker 2003) and Open Mind Com-
significantly more time on the ‘objects’ activity. This is mon Sense has been applied to textual affect sensing (Liu &
probably because the ‘objects’ activity requires the least Singh 2003). In this section, we describe two ways in which
amount of thought and because it was the only activity that the data collected as part of OMICS is being used.
involved images. Although users were allowed to submit
their own images of indoor objects, very few users actually Active Desire Selection
did. The second contest in February 2004, with more em- Our knowledge base can be used for common sense and
phasis on Generalization, Paraphrase, and Tasks activities practical reasoning using Belief-Desires-Intentions (BDI)
had much fewer submissions (about 2000). Fewer submis- theory. BDI was originally developed by Bratman (1987)
sions may be due to higher degree of difficulty, thought, and and is founded upon established theory of rational action in
typing required for submissions. humans.
Given causality relations, observations, and human com-
Feedback mands, the robot can use the desires relations from the
Although we have received little feedback on the OMICS OMICS knowledge base to deduce active desires (goals).
website, comments thus far have been largely positive. One These desires can then be used in action selection using the
of the weekly winners entered data with her seven-year-old Belief-Desire-Intention (BDI) architecture (Rao & Georgeff
son. She had the following to say about the site: 1995; Wooldridge 1999).
Figure 5 shows an example of active desire selection.
As a teacher I think it is really great and put my son on Here the robot observes from its sensors that a coke can
it with me here at home—It was a great mom and kid is warm. From the desires relations, the robot knows that
project for several days. My little one who is 5 will be (coke-can, warm) →d (coke-can, chilled), and hence that it
working on it too this week. It really forces us to do should pursue an action that causes (coke-can, chilled). The
some critical thinking—the relationships and location robot, however, does not know how to directly cause (coke-
section were great for him as were the free questions can, chilled), so it looks into its list of causality implica-
that he could come up with on his own. tions. The robot sees that (coke-can, refrigerated) → (coke-
Some users were concerned about their spelling errors and can, chilled). Thus it can infer that by refrigerating the warm
the spelling errors that were part of their sentence templates. coke-can, it can be chilled, and therefore the robot can add
Most grievous spelling errors were filtered out by the ad- that desire to its list of active desires.
ministrators, but some minor spelling errors were allowed
into the database. Other users were concerned that the data Topological Map with Room Labeling
they entered already existed in the database. One user com- Space can be labeled using terms that people typically use
mented: such as Large Conference Room, Small Conference Room,
 kitchen
dinning room

sink dinning room

dinning table
kitchen
“This is the
master bedroom”
living room

couch

“This is the bed”

master bedroom
coffee table

piano couch living room

Figure 6: Room labeling showing input of simulated room Figure 7: Room labeling showing output of our algorithm
and object labels at specific locations. with most likely room labels and locations.

and Library. Information on the extent and connectivity of P (sink|D) = 0.70 (from object recognition)
these regions can be used to generate a topological map with P (microwave|D) = 0.90 (from speech recognition)
spatial labeling for providing natural interaction. This map P (sink|kitchen) = 0.13 (estimated from OMICS)
can be used to plan a path to a specific room based on the P (microwave|kitchen) = 0.11 (estimated from OMICS)
current task. P (sink|bedroom) = 0.003 (estimated from OMICS)
We simulated data for a laser scanner to provide extent P (microwave|bedroom) = 0.002 (estimated from OMICS)
and connectivity 2D map information. Dots in figure 6 rep-
resent individual laser scan readings from a merged map Further work will involve using laser data instead of sim-
generated during environment exploration. Humans provide ulated 2D maps. This data can also be extracted from cam-
a running description of the immediate surroundings as the eras by building a sparse map of the unknown environment.
robot explores a new home or office. These labels can be Simulated place and object labels supplied by the user will
room types (e.g., This is a kitchen or This is the living room), be replaced by real data from a speech recognition system.
or about objects (e.g., This is a chair or This is a computer). Simulated object labels will be replaced by an object recog-
In addition, the robot might use an object recognition sys- nition system with associated confidence values.
tem to recognize objects with a confidence value given by Probability Estimation from common sense knowledge
the recognition system. base The robot’s sensory perceptions are combined with
These simulated objects and room labels with associated the priors and conditional probabilities for object location
probabilities (from a speech/object recognition system) and estimated from the common sense knowledge base to deter-
the 2D map are input to our system. We then use probability mine the most likely room label.
estimates given by statistical analysis of object location data Given a set ω ∈ Ω of rooms, objects xi , and sensory
from our knowledge base to label different rooms and open data D, the robot sensory perception provides P (xi |D), and
areas. We ran simulations to output topologically labeled the location data in our knowledge base is used to estimate
maps of indoor home and office environments with different P (xi ) and P (xi |ω). The robot collects information about
inputs. the objects in the room, perhaps through speech recognition
In related work, Myers and Konolige (1992) generated (e.g. a human says “this is a chair”) or through an object
and modified map layouts to reflect sentential information recognition system. These observations D induce a con-
and common sense constraints for example: ditional probability distribution P (xi |D) over the objects
• individuals own offices xi ∈ X. We wish to combine these probability distribu-
tions with P (xi |ω) and P (xi ) probability estimations from
• galleries or walkways are not owned by individuals
our knowledge base to calculate the room that is most likely,
• reception is located at the entrance to the office namely:
As shown in figure 6, we are given the walls and labels ω
= arg max P (ω|D)
ω∈Ω
such as This is the bed and some objects recognized in the Let x denote a vector indicating the presence or absence
environment or pointed to by a user like sink, dining table, of the objects xi . Assuming a generative Bayesian model
coffee table, piano and couch with an associated probability where ω influences x and x influences D, we calculate the
value. From this information, sensory data D, and the Open joint distribution:
Mind Indoor Common Sense database we may compute the
location probabilities. P (ω, x, D) = P (ω)P (x|ω)P (D|x)
These numbers can be used to compute the probability of P (ω)P (x|ω)P (x|D)P (D)
the most likely room using Bayes’ formula. Figure 7 shows =
P (x)
the output of our algorithm. For example we have deter-
mined that based on available information, the most likely We use this to calculate the most likely room, ω
:
room label for the top-left room is the kitchen. arg max P (ω|D)
ω∈Ω


= arg max P (ω, x|D) not yet intelligent enough to actually select and execute the
ω∈Ω
x mid-level actions that accomplish these desires. It remains to

P (ω)P (x|ω)P (x|D) be seen how common sense can be used in the actual execu-
= arg max
ω∈Ω P (x) tion of various tasks, such as cleaning a bathtub. One might
x

P (x1 |ω)P (x1 |D)
P (xn |ω)P (xn |D) use the teleo-reactive program framework, as proposed by
= arg max P (ω)
P (x1 )
···
P (xn )
Nils Nilsson (1992; 1994), to accomplish such basic tasks
ω∈Ω
x1 xn as make a cup of coffee hot or pop some popcorn.


n

P (xi |ω)P (xi |D)
= arg max ln P (ω) +
ω∈Ω
ln
P (xi )
Acknowledgments
i=1 xi This work was done while Mykel Kochenderfer was a sum-
mer intern at Honda Research Institute USA, Inc. The au-
Since the xi ’s are binary valued, the computational com- thors would like to thank David Stork and Push Singh for
plexity is linear in the number of possible objects. their helpful comments and suggestions. Nils Nilsson re-
Our knowledge base contains a collection of tuples of ob- viewed an earlier version of this paper and provided invalu-
jects and rooms. We estimate P (xi ) by counting the num- able feedback. Thanks are also due to anonymous reviewers
ber of times xi is mentioned in the database and dividing by and the users of the Open Mind Indoor Common Sense web-
the number of entries in the database. We estimate P (xi |ω) site for their data and feedback.
by counting the number of times the tuple (xi , ω) appears
and dividing by the number of tuples that mention ω, i.e. References
P (xi |ω) = C(xi , ω)/C(ω). However, this assigns zero
Bratman, M. 1987. Intention, Plans, and Practical Reason.
probability to P (xi |ω) in cases where the database never
mentions the tuple (xi , ω). This is rectified by using Lid- Cambridge, MA: Harvard University Press.
stone’s law to redistribute some of the probability mass as- Liu, H., and Singh, P. 2003. OMCSNet: A commonsense
signed to the observed tuples to the unobserved tuples. Lid- inference toolkit. Technical Report SOM02-01, MIT Me-
stone’s law uses a parameter λ < 1 to control how much dia Lab Society Of Mind Group.
probability is distributed to unseen tuples. We then have Liu, H.; Lieberman, H.; and Selker, T. 2003. Exploiting
C(xi , ω) + λ agreement and disagreement of human annotators for word
P (xi |ω) = sense disambiguation. In Proceedings of the Seventh In-
C(ω) + λn ternational Conference on Intelligent User Interfaces (IUI
Unseen instances are assigned probability λ/n, where n is 2003), 125–132.
the number of objects. Miller, G. A. 1995. WordNet: A lexical database for en-
glish. Communications of the ACM 38(11):39–41.
Conclusions and Future Work Myers, K., and Konolige, K. 1992. Reasoning with analog-
The Open Mind Indoor Common Sense project has success- ical representations. In Nebel, B.; Rich, C.; and Swartout,
fully captured thousands of pieces of common sense knowl- W., eds., Principles of Knowledge Representation and Rea-
edge about home and office environments. Our contribu- soning: Proceedings of the Third International Conference
tions to common sense data collection include the restric- (KR92). San Mateo, CA: Morgan Kaufmann Publishers
tion of the domain to enhance the density of the knowledge, Inc.
dynamic prompting of data based on prior data in the knowl-
Nilsson, N. J. 1992. Towards agent programs with circuit
edge base, and object-centric data collection focusing on ob-
semantics. Technical Report STAN–CS–92–1412, Depart-
jects and their properties. We use comprehensive manual
ment of Computer Science, Stanford University, Stanford,
data review to ensure the quality of the collected knowledge.
CA 94305.
The indoor home and office focus of our data collection
and the structured activities have lead to a dense knowledge Nilsson, N. J. 1994. Teleo-reactive programs for agent
base. This knowledge base was useful in determining ac- control. Journal of Artificial Intelligence Research 1:139–
tive desires from which actions could be selected. We com- 158.
puted probability estimates from our common sense knowl- Rao, A. S., and Georgeff, M. P. 1995. BDI-agents: from
edge base and used them in a Bayes formulation to compute theory to practice. In Proceedings of the First Intl. Confer-
the most likely room label given room and object labels with ence on Multiagent Systems.
confidence levels. These labels were combined with a 2D Stork, D. G. 1999. The Open Mind Initiative. IEEE Expert
map to build topologically labeled maps. Systems and Their Applications 14(3):19–20.
Distributed knowledge capture results in messy knowl- Stork, D. G. 2000. Open data collection for training intel-
edge and one of the challenges is to convert this into a us- ligent software in the open mind initiative. In Proceedings
able form. Further research will be required to clean up the of the Engineering Intelligent Systems (EIS2000).
data using statistical probability estimation techniques. Re-
search is also required to represent, maintain and update this Wooldridge, M. 1999. Intelligent agents. In Weiss, G., ed.,
knowledge on a real robot. Multiagent Systems: A Modern Approach to Distributed
Although the robot can use the common sense knowledge Artificial Intelligence. Cambridge, MA, USA: The MIT
at a very high level to determine which desires to pursue, it is Press. 27–78.