IEEE-RAS International Conference on Humanoid Robots (Humanoids 2007)

Benchmarking in Humanoid Robotics

Thursday, Nov. 29th, Pittsburgh, USA


Objectives Program Organizers and contact


Benchmarking robotics research and systems is a difficult task. In the past, different competitions and benchmarks have been suggested, e.g. RoboCup Soccer, RoboCup Rescue, DARPA grand challenge etc. Recently, the attention of the robotics community has been increased to the definition of clear and reproducible evaluation performance metrics and benchmarking procedures in terms of methodology and test data sets, which are both representative for the different aspects of a specific robotic task and have the chance to be accepted by a wide community.

This workshop will provide a forum for researchers working in various areas pertaining to humanoid robots to interact and exchange ideas toward the definition of benchmarks in the core humanoid robotics areas.

Topics of Interest:

• Benchmarking Learning for Humanoid robots
• Benchmarking Natural Human-Humanoid Interaction
• Benchmarking Motion Planning
• Benchmarking Object and Action Recognition
• Benchmarking Grasping and Dexterous Manipulation
• Benchmarking Middleware standard
• Benchmarking Humanoid System Integration

Time Title
Welcome and Overview
Rüdiger Dillmann
University of Karlsruhe, Karlsruhe, Germany
Bechmarking Humanoid Robots: Scenarios, Tools and Metrices
Tamim Asfour, and Rüdiger Dillmann
University of Karlsruhe, Karlsruhe, Germany

Benchmarking in humanoid motion planning

Kensuke Harada, Eiichi Yoshida, and Kazuhito Yokoi
Humanoid Research Group, Intelligent Systems Research
Institute National Institute of Advanced Industrial Science and Technology (AIST)


Benchmarking Integrative Humanoid Systems.

Gordon Cheng, NiCT (National Institute of Information and Communications Technology)
ATR International, Kyoto, Japan/JST-ICORP, Saitama, Japan


Benchmarking of Dexterous Manipulation and Assembly Tasks

Robert Platt


Integration and standardization: experience in open-source development
Giorgio Metta
Italian Institute of Technology & University of Genova
Evaluation of perception and dialogue components of a human-friendly robot
Rainer Stiefelhagen, and Alex Waibel
University of Karlsruhe, Germany
On Benchmarking Benchmarking

Christian Goerick
Honda Research Institute Europe GmbH

Humanoid Robot Benchmarks

Alex Waibel
University of Karlsruhe,
Carnegie Mellon University




End of workshop

Organizers and contact

Tamim Asfour
Gordon Cheng
Rüdiger Dillmann

James Kuffner
Aleš Ude
Alex Waibel

For any further information, please contact the organizers at:
Tamim Asfour, University of Karlsruhe, Haid-und-Neu-Str. 7, 76131 Karlsruhe, Germany
Phone: +49 (0)721 608 7379, Fax: +49 (0)721 608 8270, Email: asfour at