If you’ve used a smart voice assistant such as Alexa, Siri and whatever-Google’s-smart-assistant-is-called, you’ll probably have noticed that the tech is getting smarter every day. Google can wait on hold for you, Siri can speak in a gender-neutral voice and Alexa can read you bedtime stories in your dead grandmother’s voice. Robotics is evolving in leaps and bounds as well, as we explored in our Robotics event last month. The gap between the two — voice commands and autonomous robotics — has been vast, for a number of reasons. Last week, we went to Google’s robotics labs in Mountain View to see how that’s set to change in the near future.

Teaching robots what to do for repetitive tasks in controlled spaces where humans aren’t allowed isn’t easy, but it’s more or less a solved problem. Rivian’s recent factory tour was a great reminder of that, but the use of industrial robotics is everywhere in manufacturing.

General-purpose robots that are able to solve lots of different tasks based on voice commands in spaces where humans also exist, is a lot harder. You might say, “But what about Roomba,” but everyone’s favorite robo-vacuum is generally programmed to avoid touching things other than the floor, and whatever’s on the floor — much to some owners’ chagrin.

google makes robots smarter by teaching them about their limitations

Table tennis is a game where the robot can easily self-determine whether a task was successful and learn from its mistakes. Here, one of the robotics researchers at Google is taking a break, showing the robot what’s what. Image Credits: Haje Kamps (opens in a new window) / TechCrunch(opens in a new window)

“You may wonder why ping-pong. One of the big challenges in robotics today is this intersection of being fast, precise and adaptive. You can be fast and not adaptive at all; that’s not a problem. That’s fine in an industrial setting. But being fast and adaptive and precise is a really big challenge. Ping-pong is a really nice microcosm of the problem. It requires precision and speed. You can learn from people playing: it’s a skill that people develop by practicing,” Vincent Vanhoucke, Distinguished Scientist and head of robotics at Google Research told me. “It’s not a skill where you can read the rules and become a champion overnight. You have to really practice it.”

Speed and precision is one thing, but the nut Google is really trying to crack in its robotic labs, is the intersection between human language and robotics. It is making some impressive leaps in the level of robotic understanding natural language that a human might use. “When you have a minute, could you grab me a drink from the counter?” is a pretty straightforward request that you might ask a human. To a machine, however, that statement wraps a lot of knowledge and understanding into a seemingly single question. Let’s break it down: “When you have a minute” could mean nothing at all, just meant as a figure of speech, or it could be an actual request to finish what the robot is doing. If a robot is being too literal, the “correct” answer to “could you grab me a drink” could just be the robot saying “yes”. It can, and it confirms that it is able to grab a drink. But, as the user, you didn’t explicitly ask the robot to do it. And, if we’re being extra pedantic, you didn’t explicitly tell the robot to bring you the drink.

These are some of the issues that Google is tackling with its natural language processing system; the Pathways Language Model — or PaLM among friends: Accurately processing and absorbing what a human actually wants, rather than literally doing what they say.

The next challenge is recognizing what a robot is actually able to do. A robot may understand perfectly well when you ask it to grab a bottle of cleaner from the top of the fridge, where it is safely stored out of the way of children. The problem is, the robot can’t reach that high. The big breakthrough is what Google is calling “affordances” — what can the robot actually do with some reasonable degree of success. This might include easy tasks (“move a meter forward”), slightly more advanced tasks (“Go find a can of Coke in the kitchen”), to complex, multi-step actions that require the robot to show quite a bit of understanding of its own abilities and the world around it. (“Ugh, I spilled my can of Coke on the floor. Could you mop it up and bring me a healthy drink?”).

Google’s approach uses the knowledge contained in language models (“Say”) to determine and score actions that are useful for high-level instructions. It also uses an affordance function (“Can”) that enables real-world-grounding and determines which actions are possible to execute in a given environment. Using the PaLM language model, Google is calling that PaLM-SayCan.

google makes robots smarter by teaching them about their limitations

Google’s robotics lab is using a number of these robots from Everyday Robots. These chaps are taking a well-deserved R&R (rest and recharge), and they’ve even learned how to plug themselves in for recharging. Image Credits: Haje Kamps (opens in a new window) / TechCrunch(opens in a new window)

To solve the more advanced command above, the robot has to break it down into a number of individual steps. One example of that might be:

Come to the speaker. Look at the floor, find the spill, remember where it is. Go through the drawers, cabinets and kitchen counters looking for a mop, sponge or paper towel. Once a cleaning tool (there is a sponge in the drawer) is found, pick it up. Close the drawer. Move to the spill. Clean up the spill, monitoring whether the sponge can absorb all the liquid. If not, go wring it out in the sink, and come back. Once the spill is cleaned, wring the sponge one more time. Turn on the tap, rinse the sponge, turn off the tap, wring the sponge one last time. Open drawer, put sponge away, close drawer. Identify what drinks are in the kitchen, and somehow determine which drinks are “healthier” than a Coke. Find a bottle of water in the fridge, pick it up, bring it to the person who asked for it — who may have moved since they asked the question, because you’re a slow-poke little robot that had to roll back and forth to the sink 14 times, because instead of using paper towels, you thought it’d be a brilliant idea to use a little kitchen sponge to mop up 11 ounces of liquid.

Anyway — I’m poking fun here, but you get the gist; even relatively simple-sounding instructions can, in fact, include a large number of steps, logic and decisions along the way. Do you find the healthiest drink around, or is the goal to get anything that’s healthier than Coca-Cola? Might it make sense to get the drink first, and then mop up the mess, so the human can have their thirst quench while you figure out the rest of the task?

google makes robots smarter by teaching them about their limitations

This screenshot shows how the robot might evaluate a query and figure out what it should do at every step of the way. In this case, the robot determined that it should find a sponge, and that it has a high degree of success in finding, identifying and manipulating sponges. It also shows that this particular robot is really good at “going to drawers,” but also concludes that that wouldn’t be all that helpful in this instance. Image Credits: Google

The crucial thing here is to teach the robots what they can and can’t do, and what makes sense in various situations. Touring the Google robotics lab, I saw 30-odd robots, both from Everyday Robots and more purpose-built machines, playing table tennis, catching lacrosse balls and learning to stack blocks, open fridge doors and “being polite” while operating in the same space as humans.

google makes robots smarter by teaching them about their limitations

Nice catch! Image Credits: Haje Kamps (opens in a new window) / TechCrunch

The interesting challenge that robotics faces is that language models aren’t inherently grounded in the physical world. They are trained on huge libraries of text, but text libraries don’t interact with their environments, nor do they have to worry too much about causing issues. It’s kinda funny when you ask Google to direct you to the nearest coffee shop, and Maps accidentally maps out a 45-day hike and a three-day swim across a lake. In the real world, silly mistakes have real consequences.

For example, when prompted with “I spilled my drink, can you help?” the language model GPT-3 responds with “You could try using a vacuum cleaner.” It makes sense: For some messes, a vacuum cleaner is a good choice, and it stands to reason that a language model associates a vacuum cleaner with, well, cleaning. If the robot actually did that, it would likely fail: Vacuums aren’t great at spilled drinks, and water and electronics don’t mix, so you might end up with a broken vacuum at best, or an appliance fire at worst.

Google’s PaLM-SayCan-enabled robots are placed in a kitchen setting, and are trained to get better at various aspects of being helpful in a kitchen. The robots, when given an instruction, are trying to make a determination. “What is the likelihood of me being successful at the thing I’m about to try,” and “how helpful is this thing likely to be.” Somewhere in the space between those two considerations, robots are getting significantly smarter by the day.

google makes robots smarter by teaching them about their limitations

The triumphant return of the sponge-fetching robot. Image Credits: Haje Kamps (opens in a new window) / TechCrunch

Affordances — or the ability to do something — isn’t binary. Balancing three golf balls on top of each other is very hard, but not impossible. Opening a drawer is almost impossible for a robot that hasn’t been shown how drawers work — but once they are trained, and are able to experiment with how to best open a drawer, they can get a higher and higher degree of confidence in a task. An untrained robot, Google suggests, might not be able to grab a bag of potato chips from a drawer. But give it some instructions and a few days to practice, and the chance of success goes up significantly.

Of course, all of this training data is scored as the robot is trying things out. From time to time, a robot may “solve” a task in a surprising way, but it may actually be “easier” to do it that way for a robot.

By divorcing the language models from the affordances, it means that the robot can “understand” commands in a number of different languages. The team demonstrated that in the kitchen, too, when head of robotics Vincent Vanhoucke asked the robot for a can of Coke in French; “We got the language skills for free,” the team said, highlighting how the neural networks that are being used to train the robots are flexible enough to open new doors (literally and figuratively) for accessibility and universal access.

google makes robots smarter by teaching them about their limitations

Most robots that touch, open, move and clean things aren’t generally invited to operate this close to humans. We were encouraged to keep our distance, but the researchers seemed very at home with the robots operating autonomously within inches of their non-armored human bodies. Image Credits: Haje Kamps (opens in a new window) / TechCrunch

None of the robots or technologies are currently available, or necessarily even destined, for commercial products.

“Right now, it’s entirely research. As you can see from the skill level we have today, it’s not really ready to be deployed in a commercial environment. We are research outfits, and we love to work on things that don’t work,” quips Vanhoucke. “That’s the definition of research in some ways, and we’re going to keep pushing. We like to work on things that don’t need to scale because it’s a way of informing how things scale with more data and more computer abilities. You can see a trend of where things might go in the future.”

It’s going to take Google’s robotics lab a while to figure out what, if any, commercial impacts of its experiments will be in the long run, but even in the relatively simple demos shown in Mountain View last week, it’s obvious that natural language processing and robotics both win as Google’s teams build deeper skills, knowledge and vast datasets in how to train robots.


Google Chrome is the most vulnerable browser in 2022

Google Chrome icon AppleInsider is supported by its audience and may earn commission as an Amazon Associate and affiliate partner on qualifying purchases. These affiliate partnerships do not influence our editorial content. New data reveals that Google Chrome users need to be careful when browsing the web, but Safari users ...

View more: Google Chrome is the most vulnerable browser in 2022

Giant Monster War, a real-time strategy game where you conquer the world, is out now on Android and iOS

X-Legend Entertainment and Audere Gaming have come together to release a new strategy game called Giant Monster War on mobile today. Taking place in a vast world, players will lead a squad of warships and legendary Titans and Heroes who have been tasked with exploring and conquering the Titan ...

View more: Giant Monster War, a real-time strategy game where you conquer the world, is out now on Android and iOS

Idle Asterix, a brand new idle game based on the legendary French comic series, launches onto Android today

German Developer gameXcite GmbH has launched its latest project, an idle game based on the classic Asterix comic book series. This casual idle game will see you taking care of a village and its denizens while also defending against attacks from Roman soldiers as they attempt to tear down ...

View more: Idle Asterix, a brand new idle game based on the legendary French comic series, launches onto Android today

SCOTUS Takes Case Examining Section 230 Protection for YouTube

This site may earn affiliate commissions from the links on this page. Terms of use. (Photo: Adam Szuscik/Unsplash)The US Supreme Court has taken on a case that will test the true limitations of Section 230 of the Communications Decency Act. The case, Reynaldo Gonzalez et al v. Google, questions whether ...

View more: SCOTUS Takes Case Examining Section 230 Protection for YouTube

Google Pixel Watch Photos Leak on Twitter a Day Before the Launch

Leaked photographs of Google’s new Pixel Watch have surfaced online ahead of the company’s Made By Google launch event on Oct. 6, Thursday, in New York City. During the Google I/O conference in May, the tech giant initially provided a sneak peek of its upcoming wearable smartwatch. What To ...

View more: Google Pixel Watch Photos Leak on Twitter a Day Before the Launch

Android 12 has arrived for the unlocked Galaxy A21 in the USA

A week after the US carrier-locked Galaxy A21 started getting the Android 12 update, Samsung is now rolling out the new OS version to the unlocked version for the United States. The unlocked Galaxy A21 is finally making the jump from Android 11 to 12, but the security patch ...

View more: Android 12 has arrived for the unlocked Galaxy A21 in the USA

Grab a Google Pixel 6a for $349, Its Lowest Price Yet

Justin Duino / How-To Geek The Google Pixel 6a is one of the best budget Android phones around, and even though it has only been available for a few months, Google is already slashing prices. It dropped to a new low of $370 at Amazon in August, and now the phone is even ...

View more: Grab a Google Pixel 6a for $349, Its Lowest Price Yet

Pixel Watch Could Keep You From Dumping Android

The Apple Watch has helped lock people into Apple's universe of products and services. Google needs to do the same.

View more: Pixel Watch Could Keep You From Dumping Android

All Android Versions Operating Systems: 19 Version History

Founded by Google’s former head of ads, Neeva brings its ad-free search engine to Europe

Pixel 7 Pro Updates That I'm Hoping Google Announces Tomorrow

CMS integration and 2 other new Google tag feature updates

The Pixel 6 gave Google momentum — now the Pixel 7 has to keep it

Google Pixel 7 looks less appealing after latest spec leak

GMC Hummer EV Malfunctions, Leaves YouTuber Stranded In Traffic

Google Cloud Medical Imaging Suite is here to help medical staff diagnose illnesses with AI

Google Pixel Launch Event: How To Watch It Online & Timings

Google Pixel launch event: how to watch the Pixel 7, Pro & Pixel Watch reveal

Google's Answer to the Apple Watch Leaks in a Closer Look Ahead of Official Reveal

Pixel 7 and Pixel 7 Pro full specs leak ahead of Google launch event


Top Car News Car News