facebook's ai research could spur smarter ar glasses and robots

© Provided by CNET Facebook has been working on augmented reality glasses. Getty Images

Facebook envisions a future in which you'll learn to play the drums or whip up a new recipe while wearing augmented reality glasses or other devices powered by artificial intelligence. To make that future a reality, the social network needs its AI systems to see through your eyes. 

"This is the world where we'd have wearable devices that could benefit you and me in our daily life through providing information at the right moment or helping us fetch memories," said Kristen Grauman, a lead research scientist at Facebook. The technology could eventually be used to analyze our activities, she said, to help us find misplaced items, like our keys.

That future is still a ways off, as evidenced by Facebook's Ray-Ban branded smart glasses, which debuted in September without AR effects. Part of the challenge is training AI systems to better understand photos and videos people capture from their perspective so that the AI can help people remember important information. 

facebook's ai research could spur smarter ar glasses and robots

© Provided by CNET Facebook says analyzing video shot from the first-person perspective is challenging for computers. Facebook

Facebook said it teamed up with 13 universities and labs that recruited 750 people to capture more than 2,200 hours of first-person video over two years. The participants, who lived in the UK, Italy, India, Japan, Saudi Arabia, Singapore, the US, Rwanda and Colombia, shot videos of themselves engaging in everyday activities such as playing sports, shopping, gazing at their pets or gardening. They used a variety of wearable devices, including GoPro cameras, Vuzix Blade smart glasses and ZShades video recording sunglasses.

Starting next month, Facebook researchers will be able to request access to this trove of data, which the social network said is the world's largest collection of first-person unscripted videos. The new project, called Ego4D, provides a glimpse into how a tech company could improve technologies like AR, virtual reality and robotics so they play a bigger role in our daily lives.

The company's work comes during a tumultuous period for Facebook. The social network has faced scrutiny from lawmakers, advocacy groups and the public after The Wall Street Journal published a series of stories about how the company's internal research showed it knew about the platform's harms even as it downplayed them publicly. Frances Haugen, a former Facebook product manager turned whistleblower, testified before Congress last week about the contents of thousands of pages of confidential documents she took before leaving the company in May. She's scheduled to testify in the UK and meet with Facebook's semi-independent oversight board in the near future.

facebook's ai research could spur smarter ar glasses and robots

© Getty Images

Facebook has been working on augmented reality glasses.

Even before Haugen's revelations, Facebook's smart glasses sparked concerns from critics who worry the device could be used to secretly record people. During its research into first-person video, the social network said it addressed privacy concerns. Camera wearers could view and delete their videos, and the company blurred the faces of bystanders and license plates that were captured. 

Fueling more AI research

facebook's ai research could spur smarter ar glasses and robots

© Provided by CNET Doing laundry and cooking looks different in video from various countries. Facebook

As part of the new project, Facebook said, it created five benchmark challenges for researchers. The benchmarks include episodic memory, so you know what happened when; forecasting, so computers know what you're likely to do next; and hand and object manipulation, to understand what a person is doing in a video. The last two benchmarks are understanding who said what, and when, in a video, and who the partners are in the interaction.

"This sets up a bar just to get it started," Grauman said. "This usually is quite powerful because now you'll have a systematic way to evaluate data."

Helping AI understand first-person video can be challenging because computers typically learn from images that are shot from the third-person perspective of a spectator. Challenges such as motion blur and footage from different angles come into play when you record yourself kicking a soccer ball or riding a roller coaster.

Facebook said it's looking at expanding the project to other countries. The company said diversifying the video footage is important because if AR glasses are helping a person cook curry or do laundry, the AI assistant needs to understand that those activities can look different in various regions of the world. 

Facebook said the video dataset includes a diverse range of activities shot in 73 locations across nine countries. The participants included people of different ages, genders and professions.

The COVID-19 pandemic also created limitations for the research. For example, more footage in the data set is of stay-at-home activities such as cooking or crafting rather than public events. 

Some of the universities that partnered with Facebook include the University of Bristol in the UK, Georgia Tech in the US, the University of Tokyo in Japan and Universidad de los Andes in Colombia. 

Internet Explorer Channel Network


LATEST NEWS

NEWS RELATED

It's time to get your flu shot. What to know and how it could be different this year

© Provided by CNET It’s flu season. And flu vaccine season. CNET It was barely 10 a.m. and a long line of cars snaked through the parking lot at a mass vaccination center for a major hospital steps away from Apple’s Cupertino, Calif. headquarters. Only this time, it wasn’t a…

Read more: It's time to get your flu shot. What to know and how it could be different this year

WILLIE WALSH: We're all sick to death of Zoom

© Provided by Daily Mail MailOnline logo The travel restrictions caused by the pandemic have had a catastrophic impact on our ability to connect with families, friends, or for business.  Globally, air travel collapsed more than 90 per cent at the height of the crisis, and the UK was one…

Read more: WILLIE WALSH: We're all sick to death of Zoom

China's tech regulator says scrutiny of internet industry to deepen as targeted six-month campaign continues

China’s industry regulator plans to deepen scrutiny of the internet industry as it moves into the second half of a six-month campaign launched in July, the agency’s minister told Chinese media, prolonging a crackdown on the technology sector that has lasted for nearly a year. The Ministry of Industry and…

Read more: China's tech regulator says scrutiny of internet industry to deepen as targeted six-month campaign continues

Amazon Great Indian Festival: Top offers on Alexa, Fire TV, and Kindle during the sale

Amazon is hosting its Great Indian Festival sale offering some irresistible deals and discounts across categories and products. The sale that kicked off on October 3 was available for Prime members a day before. Amazon’s Fire Stick allows you to stream shows or movies on Prime Video, Disney Hotstar, YouTube,…

Read more: Amazon Great Indian Festival: Top offers on Alexa, Fire TV, and Kindle during the sale

Nothing by OnePlus Co-Founder Carl Pei May Launch its First Phone in Early 2022

© Provided by News18 Nothing by OnePlus Co-Founder Carl Pei May Launch its First Phone in Early 2022 Nothing by OnePlus co-founder Carl Pei is reportedly planning to launch its first smartphone next year. The latest leak comes just days after Nothing announced a partnership with the US chipmaker Qualcomm.…

Read more: Nothing by OnePlus Co-Founder Carl Pei May Launch its First Phone in Early 2022

Heathrow's price hikes 'could put £100 on price of family break'

© Provided by Daily Mail MailOnline logo The cost of a family summer holiday could soar by up to £100 next year due to Heathrow airport’s ‘outrageous’ price hikes, former British Airways boss Willie Walsh warned last night. Heathrow, which is owned by a consortium of billionaire investors, wants to…

Read more: Heathrow's price hikes 'could put £100 on price of family break'

Superman's new motto swaps 'American way' for 'a better tomorrow'

© Provided by CNET DC Superman DC We all know what Superman stands for. Now DC has updated the iconic superhero’s motto: Superman stands for “truth, justice and a better tomorrow.” Up until now Superman’s mission statement has been about “truth, justice, and the American Way,” a slogan that originated…

Read more: Superman's new motto swaps 'American way' for 'a better tomorrow'

Czech car sector to make 250,000 fewer vehicles this year due to chip shortage

Cars are parked in the courtyard of Skoda Auto's factory in Mlada Boleslav, Czech Republic, as the company restarts production after shutting downdue to the coronavirus pandemic (COVID-19) . Picture taken in Mlada Boleslav, Czech Republic, April 27, 2020. REUTERS/David W Cerny/File Photo PRAGUE, Oct 17 (Reuters) – Czech car…

Read more: Czech car sector to make 250,000 fewer vehicles this year due to chip shortage

Upgrade to Windows 11? Here's How You Can Check Whether Your PC is Compatible

Flipkart Big Diwali Sale Now Live: Best Smartphones You Can Buy Under Rs 15,000

iPhone-style notch expected on Apple's next-gen MacBook Pro, says new leak

Deals of the day in Amazon sale: Apple Watch, Redmi smartphone, Lenovo laptop and more

Flipkart Big Diwali Sale Now Live: Cool Offers on iPhone 12, Pixel 4a and More

Amazon Great Indian Festival: Top offers computer accessories you can grab

Warm regards from Ironbridge Gorge ... ‘the birthplace of climate change'

Tech launches of the week: Bose SoundLink Flex, OnePlus 9RT, and more

OTHER NEWS