{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T05:22:10Z","timestamp":1784784130065,"version":"3.55.0"},"reference-count":47,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2005,1,1]],"date-time":"2005-01-01T00:00:00Z","timestamp":1104537600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2005,1]]},"abstract":"<jats:p>As robots enter the human environment and come into contact with inexperienced users, they need to be able to interact with users in a multimodal fashion\u2014keyboard and mouse are no longer acceptable as the only input modalities. In this paper we introduce a novel approach for programming robots interactively through a multimodal interface. The key characteristic of this approach is that the user can provide feedback interactively at any time\u2014during both the programming and the execution phase. The framework takes a three-step approach to the problem: multimodal recognition, intention interpretation, and prioritized task execution. The multimodal recognition module translates hand gestures and spontaneous speech into a structured symbolic data stream without abstracting away the user\u2019s intent. The intention interpretation module selects the appropriate primitives to generate a task based on the user\u2019s input, the system\u2019s current state, and robot sensor data. Finally, the prioritized task execution module selects and executes skill primitives based on the system\u2019s current state, sensor inputs, and prior tasks. The framework is demonstrated by interactively controlling and programming a vacuum-cleaning robot. The demonstrations are used to exemplify the interactive programming and the plan recognition aspect of the research.<\/jats:p>","DOI":"10.1177\/0278364904049250","type":"journal-article","created":{"date-parts":[[2005,1,26]],"date-time":"2005-01-26T07:28:38Z","timestamp":1106724518000},"page":"83-104","source":"Crossref","is-referenced-by-count":42,"title":["Interactive Multimodal Robot Programming"],"prefix":"10.1177","volume":"24","author":[{"given":"Soshi","family":"Iba","sequence":"first","affiliation":[{"name":"The Robotics Institute, Carnegie Mellon University, Pittsburgh,                         Pennsylvania 15213-3890, USA,"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christiaan J. J.","family":"Paredis","sequence":"additional","affiliation":[{"name":"Systems Realization Laboratory, G. W. Woodruff School of Mechanical                         Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0405,                         USA,"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pradeep K.","family":"Khosla","sequence":"additional","affiliation":[{"name":"The Robotics Institute and Electrical and Computer Engineering, Carnegie                         Mellon University, Pittsburgh, Pennsylvania 15213-3890,"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2005,1,1]]},"reference":[{"key":"atypb1","doi-asserted-by":"crossref","unstructured":"Agah, A. and Tanie, K. 1996. Human-machine interaction through an intelligent user interface based on contention architecture . IEEE International Workshop on Robot and Human Communication RO-MAN\u201996, Tsukuba, Japan, pp. 537-542 .","DOI":"10.1109\/ROMAN.1996.568894"},{"key":"atypb2","doi-asserted-by":"crossref","unstructured":"Bahlmann, C. and Burkhardt, H. 2001. Measuring HMM similarity with the Bayes probability of error and its application to online handwriting recognition . Proceedings of the 6th International Conference on Document Analysis and Recognition, Seattle, WA, pp. 406-411 .","DOI":"10.1109\/ICDAR.2001.953822"},{"key":"atypb3","doi-asserted-by":"crossref","unstructured":"Batavia, P. H. and Nourbakhsh, I. 2000. Path planning for the Cye personal robot . Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Takamatsu, Japan, October 30-November 5, pp. 15-20 .","DOI":"10.1109\/IROS.2000.894575"},{"key":"atypb4","doi-asserted-by":"crossref","unstructured":"Boehme, H. J.et al. 1997. Neural architecture for gesture-based human-machine interaction. Gesture and Sign Language in Human-Computer Interaction, Bielefeld, Germany , pp. 219-232.","DOI":"10.1007\/BFb0053002"},{"key":"atypb5","doi-asserted-by":"crossref","unstructured":"Fong, T., Conti, F., Grange, S., and Baur, C. 2000. Novel interfaces for remote driving: gesture, haptic and PDA. SPIE Telemanipulator and Telepresence Technologies VII, Boston, MA .","DOI":"10.1117\/12.417314"},{"key":"atypb6","unstructured":"Forsyth, D. and Ponce, J. 2003. Computer Vision: A Modern Approach, Prentice-Hall, Upper Saddle River, NJ ."},{"key":"atypb7","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008856824126"},{"key":"atypb8","doi-asserted-by":"crossref","unstructured":"Gertz, M., Stewart, D., and Khosla, P. K. 1993. A software architecture-based human-machine interface for reconfigurable sensor-based control systems . IEEE International Symposium on Intelligent Control, Chicago, IL, pp. 75-80 .","DOI":"10.1109\/ISIC.1993.397653"},{"key":"atypb9","doi-asserted-by":"crossref","unstructured":"Ghidary, S. S.et al. 2001. Multimodal human-robot interaction for map generation . Proceedings of the IEEE\/RSJ International Conference on Intelligent Robot and Systems, Maui, Hawaii, pp. 2246-2251 .","DOI":"10.1109\/IROS.2001.976404"},{"key":"atypb10","doi-asserted-by":"publisher","DOI":"10.1006\/csla.1993.1007"},{"key":"atypb11","doi-asserted-by":"crossref","unstructured":"Iba, S., Vande Weghe, J. M., Paredis, C. J. J., and Khosla, P. K. 1999. An architecture for gesture-based control of mobile robots . Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Kyongju, Korea, pp. 851-857 .","DOI":"10.1109\/IROS.1999.812786"},{"key":"atypb12","doi-asserted-by":"publisher","DOI":"10.1109\/70.294211"},{"key":"atypb13","doi-asserted-by":"crossref","unstructured":"Ishida, T.et al. 2001. Motion entertainment by a small humanoid robot based on OPEN-R . Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Piscataway, NJ, pp. 1079-1086 .","DOI":"10.1109\/IROS.2001.976312"},{"key":"atypb14","doi-asserted-by":"publisher","DOI":"10.1109\/70.554349"},{"key":"atypb15","unstructured":"Kawamura, K., Alford, A., Hambuchen, K., and Wilkes, M. 2000. Towards a unified framework for human-humanoid interaction . Proceedings of the 1st IEEE-RAS International Conference on Humanoid Robots, Boston, MA."},{"key":"atypb16","doi-asserted-by":"crossref","unstructured":"Kimura, H., Horiuchi, T., and Ikeuchi, K. 1999. Task-model based human robot cooperation using vision . Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Kyongju, Korea, pp. 701-706 .","DOI":"10.1109\/IROS.1999.812762"},{"key":"atypb17","unstructured":"Kortenkamp, D., Huber, E., and Bonasso, R. P. 1996. Recognizing and interpreting gestures on a mobile robot . National Conference on Artificial Intelligence, Portland, OR, pp. 915-921 ."},{"key":"atypb18","doi-asserted-by":"crossref","unstructured":"Kortenkamp, D., Bonasso, R. P., and Subramanian, D. 2001. Distributed, autonomous control of space habitats . IEEE Aerospace Conference, Piscataway, NJ, pp. 2751-2762 .","DOI":"10.1109\/AERO.2001.931296"},{"key":"atypb19","unstructured":"Koyama, T. 2002. On combining sampled statistics without prior samples. Private communication (September 27)."},{"key":"atypb20","doi-asserted-by":"crossref","unstructured":"Kuno, Y., Murashima, T., Shimada, N., and Shirai, Y. 2000. Interactive gesture interface for intelligent wheelchairs . International Conference on Multimedia and Expo, New York, NY, pp. 789-792 .","DOI":"10.1109\/ICME.2000.871479"},{"key":"atypb21","doi-asserted-by":"crossref","unstructured":"Lee, C. and Xu., Y. 1996. On-line, interactive learning of gestures for human\/robot interfaces . Proceedings of the IEEE International Conference on Robotics and Automation, Minneapolis, MN, pp. 2982-2987 .","DOI":"10.1109\/ROBOT.1996.509165"},{"key":"atypb22","unstructured":"Mardia, K. V. 1972. Statistics of Directional Data, Academic, New York ."},{"key":"atypb23","doi-asserted-by":"crossref","unstructured":"Mascaro, S. and Asada, H. H. 1998. Hand-in-glove human-machine interface and interactive control: task process modeling using dual Petri nets . Proceedings of the IEEE International Conference on Robotics and Automation, Leuven, Belgium, pp. 1289-1295 .","DOI":"10.1109\/ROBOT.1998.677281"},{"key":"atypb24","unstructured":"Matsumoto, Y. and Zelinsky, A. 2000. An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement . Proceedings of the 4th International Conference on Automatic Face and Gesture Recognition, Grenoble, France, pp. 499-504 ."},{"key":"atypb25","doi-asserted-by":"crossref","unstructured":"Matsumoto, Y., Ino, T., and Ogasawara, T. 2001. Development of intelligent wheelchair system with face and gaze based interface . Proceedings of the 10th IEEE International Workshop on Robot and Human Communication (ROMAN 2001), Bordeaux-Paris, France, pp. 262-267 .","DOI":"10.1109\/ROMAN.2001.981912"},{"key":"atypb26","doi-asserted-by":"crossref","unstructured":"Morrow, J. D. and Khosla, P. K. 1997. Manipulation task primitives for composing robot skills . Proceedings of the IEEE International Conference on Robotics and Automation, Albuquerque, NM, pp. 3354-3359 .","DOI":"10.1109\/ROBOT.1997.606800"},{"key":"atypb27","doi-asserted-by":"crossref","unstructured":"Musser, G. 2003. Robots that suck . Scientific American 288(2): 84-86 .","DOI":"10.1038\/scientificamerican0203-84"},{"key":"atypb28","unstructured":"Nishimura, T., Mukai, T., Nozaki, S., and Oka, R. 1998. Adaptation to gesture performers by an on-line teaching system for spotting recognition of gestures from a time-varying image. Transactions of the Institute of Electronics, Information and Communication Engineers D-II J81DII(8):1822-1830."},{"key":"atypb29","unstructured":"Ogawara, K.et al. 2000. Acquiring hand-action models in task and behavior levels by a learning robot through observing human demonstrations . Proceedings of the IEEE-RAS International Conference on Humanoid Robots, Boston, MA."},{"key":"atypb30","doi-asserted-by":"publisher","DOI":"10.7210\/jrsj.18.979"},{"key":"atypb31","doi-asserted-by":"crossref","unstructured":"Oviatt, S. 2000. Taming recognition errors with a multimodal interface . Communications of the ACM 43(9): 45-51 .","DOI":"10.1145\/348941.348979"},{"key":"atypb32","doi-asserted-by":"crossref","unstructured":"Perzanowski, D.et al. 2001. Building a multimodal human-robot interface . IEEE Intelligent Systems 16(1): 16-21 .","DOI":"10.1109\/MIS.2001.1183338"},{"key":"atypb33","doi-asserted-by":"crossref","unstructured":"Perzanowski, D.et al. 2002. Communicating with teams of cooperative robots. Multi-Robot Systems: From Swarms to Intelligent Automata, A. C. Schultz and L. E. Parker, editors, Kluwer, Dordrecht , pp. 185-193.","DOI":"10.1007\/978-94-017-2376-3_20"},{"key":"atypb34","doi-asserted-by":"publisher","DOI":"10.1109\/5.18626"},{"key":"atypb35","doi-asserted-by":"crossref","unstructured":"Rybski, P. E. and Voyles, R. M. 1999. Interactive task training of a mobile robot through human gesture recognition . Proceedings of the IEEE International Conference on Robotics and Automation, Detroit, MI, pp. 664-669 .","DOI":"10.1109\/ROBOT.1999.770051"},{"key":"atypb36","doi-asserted-by":"crossref","unstructured":"Skubic, M., Perzanowski, D., Schultz, A., and Adams, W. 2002. Using spatial language in a human-robot dialog . Proceedings of the International Conference on Robotics and Automation, Washington, DC, pp. 4143-4148 .","DOI":"10.1109\/ROBOT.2002.1014398"},{"key":"atypb37","unstructured":"Stallman, R. M. 1984. Emacs: the extensible, customizable, selfdocumenting display editor. Interactive Programming Environments, D. R. Barstow, H. E. Shrobe, and E. Sande-wall, editors, McGraw-Hill, New York , pp. 300-325."},{"key":"atypb38","doi-asserted-by":"crossref","unstructured":"Starner, T. and Pentland, A. 1995. Real-time American sign language recognition from video . Proceedings of the IEEE International Symposium on Computer Vision, Coral Gables, FL, pp. 265-270 .","DOI":"10.1109\/ISCV.1995.477012"},{"key":"atypb39","doi-asserted-by":"crossref","unstructured":"Thrun, S.et al. 1999. MINERVA: a second-generation museum tour-guide robot . Proceedings of the IEEE International Conference on Robotics and Automation, Piscataway, NJ, pp. 1999-2005 .","DOI":"10.1109\/ROBOT.1999.770401"},{"key":"atypb40","doi-asserted-by":"crossref","unstructured":"Todd, D. J. 1986. Fundamentals of Robot Technology: An Introduction to Industrial Robots, Teleoperators, and Robot Vehicles, Wiley, New York .","DOI":"10.1007\/978-94-011-6768-0_7"},{"key":"atypb41","unstructured":"Virtual Technologies Inc. 1998. CyberGlove Reference Manual."},{"key":"atypb42","doi-asserted-by":"crossref","unstructured":"Voyles, R. M., Agah, A., Khosla, P. K., and Bekey, G. A. 1997. Tropism-based cognition for the interpretation of context-dependent gestures . Proceedings of the IEEE International Conference on Robotics and Automation, Albuquerque, NM, pp. 3481-3486 .","DOI":"10.1109\/ROBOT.1997.606874"},{"key":"atypb43","doi-asserted-by":"publisher","DOI":"10.1109\/5254.809564"},{"key":"atypb44","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008918401478"},{"key":"atypb45","doi-asserted-by":"crossref","unstructured":"Wren, C. R., Clarkson, B. P., and Pentland, A. P. 2000. Understanding purposeful human motion . Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France, pp. 378-383 .","DOI":"10.1109\/AFGR.2000.840662"},{"key":"atypb46","doi-asserted-by":"crossref","unstructured":"Yamada, Y., Morizono, T., Umetani, Y., and Yamamoto, T. 2002. Human error recovery for a human\/robot parts conveyance system . Proceedings of the International Conference on Robotics and Automation, Washington, DC, pp. 2004-2009 .","DOI":"10.1109\/ROBOT.2002.1014835"},{"key":"atypb47","unstructured":"Young, S. J.et al. 2000. HTK: Hidden Markov Model Toolkit V3.0, Microsoft Corporation, Redmond, WA ."}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364904049250","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364904049250","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:17:12Z","timestamp":1777457832000},"score":1,"resource":{"primary":{"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/journals.sagepub.com\/doi\/10.1177\/0278364904049250"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2005,1]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2005,1]]}},"alternative-id":["10.1177\/0278364904049250"],"URL":"https:\/\/2.zoppoz.workers.dev:443\/https\/doi.org\/10.1177\/0278364904049250","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2005,1]]}}}