Mastering physical contact with a new algorithm for robots

Penn Engineers has developed a new algorithm that allows robots to respond to complex physical contact in real time, enabling autonomous robots to successfully complete previously impossible tasks, such as controlling the movement of a sliding object.

The algorithm, known as consensus complementarity control (C3), could prove to be an essential part of future robots, translating instructions from the results of artificial intelligence tools such as large language models , or LLM, in appropriate actions.

“Your big language model might say, ‘Go chop an onion,'” says Michael Posa, assistant professor of mechanical engineering and applied mechanics (MEAM) and core faculty member in the General Robotics, Automation, Sensing Laboratory and perception (GRASP). . “How do you move your arm to hold the onion in place, hold the knife, slice it in the right direction, reorient it if necessary?”

One of the biggest challenges in robotics is control, a catch-all term for the intelligent use of robot actuators, the parts of a robot that move or control its limbs, such as motors or hydraulic systems. Controlling the physical contact that a robot establishes with its environment is both difficult and essential.

“This type of lower- and mid-level reasoning is really fundamental to making anything work in the physical world,” Posa says.

Since the 1980s, artificial intelligence experts have recognized that, paradoxically, the first skills humans learn – how to manipulate objects and move from one place to another, even in the face of obstacles – are the hardest to teach to robots, and vice versa.

“Robots work great until they start touching things,” Posa says. “Artificial intelligence machines can currently solve International Mathematical Olympiad-level math problems and beat experts at chess. But they have at best the physical capabilities of a 2 or 3 year old.”

Essentially, this means that every robot interaction that involves touching something – picking up an object, moving it somewhere else – must be carefully choreographed. “The main challenge is the contact sequence,” says William Yang, a recent graduate of Posa’s Dynamic Autonomy and Intelligent Robotics (DAIR) Lab. “Where do you put your hand in the environment? Where do you put your foot in the environment?”

Of course, humans rarely have to think twice about how they interact with objects. Part of the challenge for robots is that something as simple as picking up a cup actually involves many different choices, from the correct approach angle to the appropriate amount of force.

“All of these choices are not so terribly different from those around them,” Posa points out. But until now, there was no algorithm that allowed robots to evaluate all these choices and make an appropriate decision in real time.

To solve the problem, the researchers essentially designed a way to help robots “hallucinate” the different possibilities that can arise when contacting an object. “By imagining the benefits of touching objects, you get gradients in your algorithm that correspond to that interaction,” Posa explains.

“And then you can apply some style of gradient-based algorithm and, in the process of solving that problem, physics gradually becomes more and more precise over time, to the point where you no longer just imagine, “What if I touch it?” but you actually intend to go out and touch him.

Over the past year, Posa and the DAIR Lab have authored a series of award-winning articles on the topic, most recently published on the arXiv preprint server for which Yang was lead author, which won the Outstanding Student Paper Award at the 2024 Robotics: Science and Systems conference in the Netherlands.

This article shows how C3 can enable robots to control sliding objects in real time. “Sliding is notoriously difficult to control in robotics,” says Yang. “Mathematically, it’s difficult, but you also have to rely on the feedback from the objects.”

But, using C3, Yang demonstrated how a robotic arm can safely manipulate a tray, similar to what servers might use in a restaurant. In videotaped experiments, Yang asked the robotic arm to pick up the tray and put it down, with and without a coffee cup, and rotate it against a wall. “Previous work thought, ‘We just want to avoid sliding,'” says Yang, “but the algorithm includes sliding as a possibility for robots to consider.”

In the future, Posa and his group hope to make the algorithm even more robust to different situations, such as when objects handled by a robot weigh slightly more or less than expected, and expand the project to more open scenarios than C3 currently does. . can’t manage.

“This is a building block that can take a fairly simple specification (make this part over there) and distill it down to the motor torque that the robot will need to achieve it,” Posa explains. “Going from a very, very complicated and messy world to the key sets of objects, features or dynamic properties that are important for a given task, that’s the open question we’re interested in.”