Robots bootstrapped through learning from Experience (scientific version)
This video shows an integrated demonstration of central results of the Xperience project. In the example task of preparing a salad and setting a table together with a human, the robot ARMAR-III uses its knowledge gained from previous experience to plan and execute the necessary actions towards its goal. The demonstration highlights the aspects of the realization of integrated complete robot systems, and emphasizes the concept of structural bootstrapping on the levels of human-robot communication and physical interaction, sensorimotor learning, learning of object affordances, and planning in robotics.
Robots bootstrapped through learning from Experience (short version)
This video shows an integrated demonstration of central results of the Xperience project. In the example task of preparing a salad and setting a table together with a human, the robot ARMAR-III uses its knowledge gained from previous experience to plan and execute the necessary actions towards its goal. The demonstration highlights the aspects of the realization of integrated complete robot systems, and emphasizes the concept of structural bootstrapping on the levels of human-robot communication and physical interaction, sensorimotor learning, learning of object affordances, and planning in robotics.
Multi-Purpose Natural Language Understanding Linked to Sensorimotor Experience in Humanoid Robots
This video demonstrates our approach to the domain description generation and natural language understanding works on the humanoid robot ARMAR-III in a scenario requiring planning and plan recognition based on human-robot communication. In this scenario, the robot is requested to perform different tasks in the kitchen scenario. The video demonstrates a human-robot collaboration. The robot not only generates and executes a plan given a human command, but adjusts it's plan during execution given new descriptions of the world and human actions.
Learning Where to Grasp Unknown Knifes for Cutting
We developed a demonstrator that allows for the transfer of grasp for cutting to unknown knifes. The video shows how we use our simulation setup to generate experiences of grasping a knife and use it for cutting, these are used to learn which visual constellations predict grasp positions that lead to successful grasp-cut action pairs. We implemented these methods on the MARVIN platform at SDU where they enable us to grasp previously unseen knifes. For that a RGB-D camera is used to get a representation oft the current scene. The video shows how the learned model is used to predict good grasp positions and how once the knife is grasped, a cutting trajectory based on human demonstration is used to cut the fruit.
Analysis of Human Cutting Behaviour
To more accurately study trajectories from human cutting we developed a setup at SDU with a small 6D F/T sensor, a 6D magnetic pose tracker and printed parts that allow the mounting of the blade. In the video we show how this setup is used to log data from demonstrated cutting trajectories of several different fruits and vegetables and show how these trajectories vary between different human demonstrations. This provides data for formalising the space in which successful cutting is possible and in which the robot later will compute successful cutting trajectories.
Transferring Object Grasping Skills and Knowledge Across Different Robotic Platforms
This video demonstrates the transfer of object grasping skills between two different humanoid robots, iCub and ARMAR-III, with different software frameworks. These two robots have different kinematics and are programmed using different middlewares, YARP and ArmarX. A bridge system has been developed to allow for the execution of grasping skills of ARMAR-III on iCub. As the embodiment differs, the known feasible grasps for the one robot are not always feasible for the other robot. Thus a reactive correction behavior has been used to detect failure of a grasp during its execution, to correct it until it is successful, and thus adapt the known grasp definition to the new embodiment.
Visual Collision Detection for Corrective Reactions during Grasp Execution on a Humanoid Robot
We present an approach for visually detecting collisions between a robot's hand and an object during grasping. This allows to detect unintended premature collisions between parts of the hand and the object which might lead to failure of the grasp if they went unnoticed. Our approach is based on visually perceiving that the object starts to move, and is thus a good complement for force-based contact detection which fails on (e.g. light) objects that don't resist the applied force but are just pushed away. Our visual collision detection tracks the hand in the robot's camera images and analyzes the optical flow near it. When a collision is perceived, the most probable part of the hand to have caused it is estimated, and a corrective motion is executed. We evaluate the detection together with different reaction strategies on the humanoid robot ARMAR-III. The results show that the detection of failures during grasp execution and their correction allow the robot to successfully finish the grasp attempts in almost all of the cases in which it would otherwise have failed.
Action sequencing in complex scenarios
Here we present action sequencing in a relatively complex scenario where robot is supposed to take a zucchini, put in on a cutting board, take a knife, cut zucchini, pour zucchini into a bowl, take a spoon and stir. It is a fully integrated system (ROS-based) where we have different modules working together in real-time: 1) planner, 2) scene segmentation, 3) object recognition, 4) object tracking and 5) action execution. In the video we show KUKA LWR robot performing above mentioned scenario.
Interaction learning for dynamic movement primitives used in cooperative robotic tasks (1/2)
Since several years dynamic movement primitives (DMPs) are more and more getting into the center of interest for flexible movement control in robotics. In this study we introduce sensory feedback together with a predictive learning mechanism which allows tightly coupled dual-agent systems to learn an adaptive, sensor-driven interaction based on DMPs. The coupled conventional (no-sensors, no learning) DMP system automatically equilibrates and can still be solved analytically allowing us to derive conditions for stability. When adding adaptive sensor control we can show that both agents learn to cooperate.Interestingly, all these mechanisms are entirely based on low level interactions without any planning or cognitive component. In videos we show two interaction demos (human-robot and robot-robot) where agents learn to interact and cooperate in order to help each other to ovoid obstacles on a way.
Interaction learning for dynamic movement primitives used in cooperative robotic tasks (2/2)
Since several years dynamic movement primitives (DMPs) are more and more getting into the center of interest for flexible movement control in robotics. In this study we introduce sensory feedback together with a predictive learning mechanism which allows tightly coupled dual-agent systems to learn an adaptive, sensor-driven interaction based on DMPs. The coupled conventional (no-sensors, no learning) DMP system automatically equilibrates and can still be solved analytically allowing us to derive conditions for stability. When adding adaptive sensor control we can show that both agents learn to cooperate.Interestingly, all these mechanisms are entirely based on low level interactions without any planning or cognitive component. In videos we show two interaction demos (human-robot and robot-robot) where agents learn to interact and cooperate in order to help each other to ovoid obstacles on a way.
Enhancing Software module reusability using port plug-ins: an iCub Experiment
Systematically developing high-quality reusable software components is a difficult task and requires careful design to find a proper balance between potential reuse, functionalities and ease of implementation. Extendibility is an important property for software which helps to reduce cost of development and significantly boosts its reusability. This work introduces an approach to enhance components reusability by extending their functionalities using plug-ins at the level of the connection points (ports). Application dependent functionalities such as data monitoring and arbitration can be implemented using a conventional scripting language and plugged into the ports of components. The main advantage of our approach is that it avoids to introduce application dependent modifications to existing components, thus reducing development time and fostering the development of simpler and therefore more reusable components. Another advantage of our approach is that it reduces communication and deployment overheads because extra functionalities can be added without introducing additional modules. The video demonstrates the port plug-in in a clean the table scenario on the iCub Humanoid robot.
Physical Interaction for Segmentation of Unknown Textured and Non-textured Rigid Objects
We present an approach for autonomous interactive object segmentation by a humanoid robot. The visual segmentation of unknown objects in a complex scene is an important prerequisite for e.g. object learning or grasping, but extremely difficult to achieve through passive observation only. Our approach uses the manipulative capabilities of humanoid robots to induce motion on the object and thus integrates the robots manipulation and sensing capabilities to segment previously unknown objects. We show that this is possible without any human guidance or pre-programmed knowledge, and that the resulting motion allows for reliable and complete segmentation of new objects in an unknown and cluttered environment.
We extend our previous work, which was restricted to textured objects, by devising new methods for the generation of object hypotheses and the estimation of their motion after being pushed by the robot. These methods are mainly based on the analysis of motion of color annotated 3D points obtained from stereo vision, and allow the segmentation of textured as well as non-textured rigid objects. In order to evaluate the quality of the obtained segmentations, they are used to train a simple object recognizer. The approach has been implemented and tested on the humanoid robot ARMAR-III, and the experimental results confirm its applicability on a wide variety of objects even in highly cluttered scenes.
Affordance based grasp exploration based on two grasping OACs
The video shows the execution of two explorative grasping behaviors. Grasping is done with the SDH dexterous three finger hand. We show how the two processes are used on the MARVIN platform in the SDU environment to build up grasping experience in terms of ’experiments’. Grasping attempts are shown and internal processes are visualized, e.g., storing the episodic memory and the development reliability measure M. This demo will show the application of the OAC formalism in a cognitive architecture in which ’experiments’ are permanently produced in the exploration process delivering data as an ’outside–in’ processes. This data is then the basis for the structural bootstrapping as an ’inside–out’ processes.
Execution of Pushing Action with Semantic Event Chains: iCub First Integration
Here we present the first integration of the framework for manipulation execution based on the so called "Semantic Event Chain" on the iCub robot. The Semantic Event Chain is an abstract description of relations between the objects in the scene. It captures the change of those relations during a manipulation and thereby provides the decisive temporal anchor points by which a manipulation is critically defined.
Action Sequence Reproduction
Teaching robots object manipulation skills is a complex task that involves multimodal perception and knowledge about processing the sensor data. We show a concept for humanoid robots in household environments with a variety of related objects and actions. Following the paradigms of Programming by Demonstration (PbD), we provide a flexible approach that enables a robot to adaptively reproduce an action sequence demonstrated by a human. The obtained human motion data with involved objects is segmented into semantic conclusive sub-actions by the detection of relations between the objects and the human actor. Matching actions are chosen from a library of Object-Action Complexes (OACs) using the preconditions and effects of each sub-action. The resulting sequence of OACs is parameterized for the execution on a humanoid robot depending on the observed action sequence and on the state of the environment during execution. The feasibility of this approach is shown in an exemplary kitchen scenario, where the robot has to prepare a dough.
Discovery, Segmentation and Reactive Grasping of Unknown Objects
Learning the visual appearance and physical properties of unknown objects is an important capability for humanoid robots that are supposed to be working in an open environment. We present an approach that enables a robot to discover new, unknown objects, segment them from the background and grasp them. This gives the robot full control over the object and allows its further multimodal exploration.
In order to discover an unknown object in a cluttered scene and segment it from the (likewise unknown) background, we generate hypotheses based on visual input and try to verify one of them by pushing it. The induced motion solves visual ambiguities and allows a clear object-background segmentation.
The acquired estimation of the object position and extent allows the robot to try grasping it. As we do not have exact shape information, we apply a reactive grasping approach. Based on tactile sensor feedback of the hand, we execute correction movements until the object can be grasped in a stable manner.
Interactive Segmentation of Unknown Objects by a Humanoid Robot
In order to learn about or manipulate unknown objects, a humanoid robot first needs to segment them from their direct surroundings. We present an approach that leverages physical interaction of the robot with the unknown object to allow for an unambiguous segmentation, which serves as a prerequisite for object learning and grasping. We demonstrate the successfull realization of this idea on the humanoid robot ARMAR-III.
Gaze Selection during Manipulation Tasks
A major strength of humanoid platforms consists in their potential to perform a wide range of manipulation tasks in human-centered environments thanks to their anthropomorphic design. Further, humanoid platforms offer active head-eye systems which allow to extend the observable workspace by employing active gaze control. In order to exploit these two key capabilities in an integrated manner, the question where to look during manipulation tasks is addressed in this work.
A solution to this gaze selection problem is proposed, which takes into account constraints derived from the manipulation tasks. Thereby, three different subproblems are addressed: the representation of the acquired visual input, the calculation of saliency based on this representation, and the selection of the most suitable gaze direction.
Learn to Wipe: A Case Study of Structural Bootstrapping from Sensorimotor Experience
In this work, we address the question of generative knowledge construction from sensorimotor experience, which is acquired by exploration. We show how actions and their effects on objects, together with perceptual representations of the objects, are used to build generative models which then can be used in internal simulation to predict the outcome of actions. Specifically, the paper presents an experiential cycle for learning association between object properties (softness and height) and action parameters for the wiping task and building generative models from sensorimotor experience resulting from wiping experiments. Object and action are linked to the observed effect to generate training data for learning a nonparametric continuous model using Support Vector Regression. In subsequent iterations, this model is grounded and used to make predictions on the expected effects for novel objects which can be used to constrain the parameter exploration. The cycle and skills have been implemented on the humanoid platform ARMAR-IIIb. Experiments with set of wiping objects differing in softness and height demonstrate efficient learning and adaptation behavior of action of wiping.
Encoding of Periodic and their Transient Motions by a Single Dynamic Movement Primitive
Present formulations of periodic dynamic movement primitives (DMPs) do not encode the transient behavior required to start the rhythmic motion, although these transient movements are an important part of the rhythmic movements (i.e. when walking, there is always a first step that is very different from the subsequent ones). An ad-hoc procedure is then necessary to get the robot into the periodic motion. In this contribution we present a novel representation for rhythmic Dynamic Movement Primitives (DMPs) that encodes both the rhythmic motion and its transient behaviors. As with previously proposed DMPs, we use a dynamical system approach where an asymptotically stable limit cycle represents the periodic pattern. Transients are then represented as trajectories converg- ing towards the limit cycle, different trajectories representing varying transients from different initial conditions. Our ap- proach thus constitutes a generalization of previously proposed rhythmic DMPs. Experiments conducted on the humanoid robot ARMAR-III demonstrate the applicability of the approach for movement generation.
