Dexterous Functional Grasping

Abstract

While there have been significant strides in dexterous manipulation, most of it is limited to benchmark tasks like in-hand reorientation which are of limited utility in the real world. The main benefit of dexterous hands over two-fingered ones is their ability to pickup tools and other objects (including thin ones) and grasp them firmly in order to apply force. However, this task requires both a complex understanding of functional affordances as well as precise low-level control. While prior work obtains affordances from human data this approach doesn't scale to low-level control. Similarly, simulation training cannot give the robot an understanding of real-world semantics. In this paper, we aim to combine the best of both worlds to accomplish functional grasping for in-the-wild objects. We use a modular approach. First, affordances are obtained by matching corresponding regions of different objects and then a low-level policy trained in sim is run to grasp it. We propose a novel application of eigengrasps to reduce the search space of RL using a small amount of human data and find that it leads to more stable and physically realistic motion. We find that eigengrasp action space beats baselines in simulation and outperforms hardcoded grasping in real and matches or outperforms a trained human teleoperator.

All videos play at 1x speed

Hammering

The grasping policy generalizes to hammers of different surface properties, weights and shape. The blue hammer is heavy (1 kg) while the wooden one is smooth and hard to grip. It is robust to heavy pushes during the grasping and post-grasp phase.

Drilling

The same policy as above generalizes to drills which differ signficantly in shape from hammers. The red YCB drill is smaller while the green industrial drill is very large and has a highly skewed weight distribution since the weight is much heavier than the base.

Screwing

The grasping policy generalizes to screwdrivers of different types. The orange screwdriver is narrower and longer while the red one is fatter and shorter.

Objects placed upright

We add cameras along the x and y axes of our setup and use the one with the highest affordance matching score. This allows the policy to grasp objects kept in any orientation.

Emergent Behavior

Here we showcase some emergent behaviors learnt by our policy. We see that the grasp is adapted on-the-fly based on interaction with the object.

Since the bottle is wide, the thumb gets trapped between the palm and bottle initially but the policy moves it back to wrap around the bottle.

The thumb position is altered when the orientation of the drill changes to prevent it from falling out.

Thin small objects

Our policy generalizes to thin and small objects such as a knife (blade retracted for safety) and a small carrot. Thin objects are challenging because it is hard to get fingers around them. They are also small and light and can easily be pushed away by the fingers.

Other objects

The same policy can also be used to functionally manipulate a range of other objects. We show examples of kitchen implements such as a spatula and two saucepans, office supplies such as a stapler, deformable objects like a teddy bear and soft dinosaur toy among others.

Failure Cases

The arm locks out when a heavy object is being lifted due to motor current overload and fails to complete post-grasp motion.

Because the hand is large, it has trouble grasping small objects like the screwdrivers which fall out upon perturbation.