artificial intelligence, robotics

Researchers manipulate elasto-plastic objects into target shapes from visual cues. Credit: MIT CSAIL

Robots manipulate soft, deformable material into various shapes from visual inputs in a new system that could one day enable better home assistants.

Many of us feel an overwhelming sense of joy from our inner child when stumbling across a pile of the fluorescent, rubbery mixture of water, salt, and flour that put goo on the map: play dough. (Even if this rarely happens in adulthood.)

While manipulating play dough is fun and easy for 2-year-olds, the shapeless sludge is quite difficult for robots to handle. With rigid objects, machines have become increasingly reliable, but manipulating soft, deformable objects comes with a laundry list of technical challenges. One of the keys to the difficulty is that, as with most flexible structures, if you move one part, you’re likely affecting everything else.

Recently, scientists from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Stanford University let robots take their hand at playing with the modeling compound, but not for nostalgia’s sake. Their new system called “RoboCraft” learns directly from visual inputs to let a robot with a two-fingered gripper see, simulate, and shape doughy objects. It could reliably plan a robot’s behavior to pinch and release play dough to make various letters, including ones it had never seen. In fact, with just 10 minutes of data, the two-finger gripper rivaled human counterparts that teleoperated the machine — performing on-par, and at times even better, on the tested tasks.

“Modeling and manipulating objects with high degrees of freedom are essential capabilities for robots to learn how to enable complex industrial and household interaction tasks, like stuffing dumplings, rolling sushi, and making pottery,” says Yunzhu Li, CSAIL PhD student and author of a new paper about RoboCraft. “While there’s been recent advances in manipulating clothes and ropes, we found that objects with high plasticity, like dough or plasticine — despite ubiquity in those household and industrial settings — was a largely underexplored territory. With RoboCraft, we learn the dynamics models directly from high-dimensional sensory data, which offers a promising data-driven avenue for us to perform effective planning.”

When working with undefined, smooth materials, the whole structure must be taken into consideration before any form of efficient and effective modeling and planning can be done. RoboCraft uses a graph neural network as the dynamics model and transforms images into graphs of tiny particles together with algorithms to provide more precise predictions about the material’s change in shape.

RoboCraft just employs visual data instead of complicated physics simulators, which researchers often use to model and understand the dynamics and force acting on objects. Three components work together within the system to form soft material into, say, an “R,” for example.

Perception — the first part of the system — is all about learning to “see.” It employs cameras to gather raw, visual sensor data from the environment, which are then turned into little clouds of particles to represent the shapes. This particle data is used by a graph-based neural network to learn to “simulate” the object’s dynamics, or how it moves. Armed with the training data from many pinches, algorithms then help plan the robot’s behavior so it learns to “shape” a blob of dough. While the letters are a little sloppy, they’re unquestionably representative.

Besides creating cutesy shapes, the team of researchers is (actually) working on making dumplings from dough and a prepared filling. It’s a lot to ask at the moment with only a two-finger gripper. A rolling pin, a stamp, and a mold would be additional tools required by RoboCraft (much as a baker requires various tools to work effectively).

A further in the future domain the scientists envision is using RoboCraft for assistance with household tasks and chores, which could be of particular help to the elderly or those with limited mobility. To accomplish this, given the many obstructions that could take place, a much more adaptive representation of the dough or item would be needed, as well as an exploration into what class of models might be suitable to capture the underlying structural systems.

“RoboCraft essentially demonstrates that this predictive model can be learned in very data-efficient ways to plan motion. In the long run, we are thinking about using various tools to manipulate materials,” says Li. “If you think about dumpling or dough making, just one gripper wouldn’t be able to solve it. Helping the model understand and accomplish longer-horizon planning tasks, such as, how the dough will deform given the current tool, movements and actions, is a next step for future work.”

Li wrote the paper alongside Haochen Shi, Stanford master’s student; Huazhe Xu, Stanford postdoc; Zhiao Huang, PhD student at the University of California at San Diego; and Jiajun Wu, assistant professor at Stanford. They will present the research at the Robotics: Science and Systems conference in New York City. The work is in part supported by the Stanford Institute for Human-Centered AI (HAI), the Samsung Global Research Outreach (GRO) Program, the Toyota Research Institute (TRI), and Amazon, Autodesk, Salesforce, and Bosch.

TECH NEWS RELATED

Apple warns its suppliers about China following Pelosi Taiwan visit

Ripple effects are continuing to multiply after the controversial visit of US House Speaker Nancy Pelosi to Taiwan just a few days ago. That visit sparked furious saber-rattling from China, for example, and even had some people fretting about WWIII. Fortunately, the worst-case fears about the Pelosi Taiwan visit ...

View more: Apple warns its suppliers about China following Pelosi Taiwan visit

A brief history of Esperanto, the 135-year-old language hated by Hitler and Stalin

In the late 1800s, the city of Białystok – which was once Polish, then Prussian, then Russian, and is today again part of Poland – was a hub of diversity, with large numbers of Poles, Germans, Russians and Yiddish-speaking Ashkanazi Jews. Each group spoke a different language and viewed ...

View more: A brief history of Esperanto, the 135-year-old language hated by Hitler and Stalin

James Gunn says Guardians of the Galaxy Vol 3. will be ‘incredibly emotional’

Guardians of the Galaxy Vol. 3 is one of the most anticipated movies of Phase 5, and the more we hear about its mysterious plot, the more exciting the sequel becomes. Vol. 3 was already exciting without any of the teasers from James Gunn or the leaked Comic-Con trailer ...

View more: James Gunn says Guardians of the Galaxy Vol 3. will be ‘incredibly emotional’

Samsung’s Galaxy Z Fold 4 Won’t Have an S Pen Slot

Samsung via Equal Leaks Rumors suggested that the Galaxy Z Fold 4 could feature a dedicated S Pen slot, much like the Galaxy S22 Ultra or older Galaxy Note devices. But an accidental Amazon listing proves that the phone cannot hold an S Pen without an add-on case. The Amazon ...

View more: Samsung’s Galaxy Z Fold 4 Won’t Have an S Pen Slot

Harvard scientists closer to solving centuries-old heart mystery

Researchers at Harvard’s John A. Paulson School of Engineering and Applied Sciences (SEAS) believe they have taken a step towards solving a centuries-old heart muscle mystery — and towards the ability to eventually fabricate entire human hearts for transplantation. Using a new form of textile manufacturing, akin to a ...

View more: Harvard scientists closer to solving centuries-old heart mystery

Fintech platform Scripbox makes strategic investment in Pune-based Wealth Managers

Online wealth management platform Scripbox has made a strategic investment in Pune-based wealth advisory platform Wealth Managers.Scripbox declined to comment on the size of the deal.This is expected to bolster its geographical presence, expand customer base and offer digital solutions to Wealth Managers’ clientele.Over the past six months, Scripbox has ...

View more: Fintech platform Scripbox makes strategic investment in Pune-based Wealth Managers

Russians Building a Satellite-Blinding Laser – An Expert Explains the Ominous Technology

A powerful enough laser beam could blind spy satellites. According to a recent report in The Space Review, Russia is building a new ground-based laser facility for interfering with satellites orbiting overhead. The basic idea is simple: flooding the optical sensors of other nations’ spy satellites with laser light ...

View more: Russians Building a Satellite-Blinding Laser – An Expert Explains the Ominous Technology

Podcast #688 – Intel & AMD Financials, Ryzen 7000 Date, be quiet! Pure Base 500 FX, Sonos, 0-Day Hacks + MORE!

Josh wanders back in for this 688 Attack Sub sized podcast. We explore another week of tech news, including about 17 straight minutes of Intel and AMD Q2 financial analysis from Josh. If that isn’t stimulating enough for you we also talk about upcoming CPU launches, a mandatory look ...

View more: Podcast #688 – Intel & AMD Financials, Ryzen 7000 Date, be quiet! Pure Base 500 FX, Sonos, 0-Day Hacks + MORE!

Samsung Galaxy Watch 5 Rumors: Will It Have Better Battery Life?

Apple Delays iPadOS 16 Software Update To Focus on iPhone 14

40-Year Mystery: Doctors Solve a Case of Unexplained Finger Pain

Spotify Finally Fixes Its Stupid Play Button

ISRO says rocket carrying small satellites placed them in wrong orbit, rendering them unusable

Save up to 41% on Already-Affordable SoundPEATS Earbuds Today Only

Netflix releases: Everything coming and going the week of August 7

Apple may launch at least four new smart home products by early 2024

Hands-on photos and features of Samsung Galaxy Z Fold4 leaked

These $269 Noise Cancelling Sleep Earbuds Provide a Nearly Silent Slumber

New Cancer Drug Combination Reduced Risk of Death by 31%

People are saying this big Marvel plot hole might ruin the MCU

OTHER TECH NEWS

Top Car News Car News