EyeMU enables users to interact with their screens without lifting a finger

As more people watch movies, edit video, read the news and keep up with social media on their smartphones, these devices have grown to accommodate the bigger screens and higher processing power needed for more demanding activities.
The problem with unwieldy phones is they frequently require a second hand or voice commands to operate – which can be cumbersome and inconvenient.
In response, researchers in the Future Interfaces Group at Carnegie Mellon University’s Human-Computer Interaction Institute (HCII) are developing a tool called EyeMU , which allows users to execute operations on a smartphone by combining gaze control and simple hand gestures.
“We asked the question, ’Is there a more natural mechanism to use to interact with the phone?’ And the precursor for a lot of what we do is to look at something,” said Karan Ahuja, a doctoral candidate in human-computer interaction.
Gaze analysis and prediction aren’t new, but achieving an acceptable level of functionality on a smartphone would be a noteworthy advance.
“The eyes have what you would call the Midas touch problem,’ said Chris Harrison , an associate professor in the HCII and director of the Future Interfaces Group. “You can’t have a situation in which something happens on the phone everywhere you look. Too many applications would open.”
SCS researchers are developing a tool, called EyeMU, that allows users to execute operations on a smartphone by combining gaze control with simple hand gestures.
CMU researchers show how gaze estimation using a phone’s user-facing camera can be paired with motion gestures to enable a rapid interaction technique on handheld phones.
Software that tracks the eyes with precision can solve this problem. Andy Kong, a senior majoring in computer science, had been interested in eye-tracking technologies since he first came to CMU. He found commercial versions pricey, so he wrote a program that used a laptop’s built-in camera to track the user’s eyes, which in turn moved the cursor around the screen – an important early step toward EyeMU.
“Current phones only respond when we ask them for things, whether by speech, taps or button clicks,” Kong said. “If the phone is widely used now, imagine how much more useful it would be if we could predict what the user wanted by analyzing gaze or other biometrics.”
It wasn’t easy to streamline the package so it could work at speed on a smartphone.
“That’s a resource constraint. You must make sure your algorithms are fast enough,” Ahuja said. “If it takes too long, your eye will move along.”
Kong, the paper’s lead author, presented the team’s findings with Ahuja, Harrison and Assistant Professor of HCII Mayank Goel at last year’s International Conference on Multimodal Interaction. Having a peer-reviewed paper accepted to a major conference was a huge achievement for Kong, an undergraduate researcher.
Kong and Ahuja advanced that early prototype by using Google’s Face Mesh tool to study the gaze patterns of users looking at different areas of the screen and render the mapping data. Next, the team developed a gaze predictor that uses the smartphone’s front-facing camera to lock in what the viewer is looking at and register it as the target.
The team made the tool more productive by combining the gaze predictor with the smartphone’s built-in motion sensors to enable commands. For example, a user could look at a notification long enough to secure it as a target and flick the phone to the left to dismiss it or to the right to respond to the notification. Similarly, a user might pull the phone closer to enlarge an image or move the phone away to disengage the gaze control, all while holding a large latte in the other hand.
“The big tech companies like Google and Apple have gotten pretty close with gaze prediction, but just staring at something alone doesn’t get you there,” Harrison said. “The real innovation in this project is the addition of a second modality, such as flicking the phone left or right, combined with gaze prediction. That’s what makes it powerful. It seems so obvious in retrospect, but it’s a clever idea that makes EyeMU much more intuitive.”

SMASH Lab Uses Wearables To Train New Privacy-Preserving Sensors

ReSkin Could Help Researchers Discover a Sense of Touch

“Smiling Eyes” Likely Don’t Signify True Happiness After All

Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213
412’268 -2900

TECH NEWS RELATED

Tecno Spark 8P Budget Smartphone With 50MP Triple Camera Setup Confirmed to Launch in India

Tecno is expected to launch the Spark 8P this month in India.

View more: Tecno Spark 8P Budget Smartphone With 50MP Triple Camera Setup Confirmed to Launch in India

Diverse Origins of Earth’s Minerals: Helping Find New Minerals, Habitable Planets, and Extraterrestrial Life

A newly completed 15-year study details the origins and diversity of every known mineral on Earth. An innovative new perspective on mineralogy, findings rank amongst the field’s most important in the last century. Scientists decipher and catalog the diverse origins of Earth’s minerals; will inform models of life’s history, ...

View more: Diverse Origins of Earth’s Minerals: Helping Find New Minerals, Habitable Planets, and Extraterrestrial Life

How to find marker genes in cell clusters

Graphical abstract. Credit: Journal of Molecular Biology (2022). DOI: 10.1016/j.jmb.2022.167525 Which genes are specific for a certain cell type, i.e., “mark” their identity? With the increasing size of datasets nowadays, answering this question is often challenging. Often, marker genes are simply genes that have been found in specific cell ...

View more: How to find marker genes in cell clusters

Here are the most effective things you can do to fight climate change

The first three charts show, in descending order, how effective different behaviours are at cutting emissions. Credit: Max Callaghan, Author provided Limiting global warming to 1.5℃ above pre-industrial levels requires reaching net zero emissions by the middle of this century. This means that, in less than three decades, we ...

View more: Here are the most effective things you can do to fight climate change

How reindeer eyes transform in winter to give them twilight vision

Credit: Shutterstock Reindeer are loved the world over for their dark, expressive eyes, majestic antlers and magical association with Santa Claus. The moment you learn the cold, hard truth of how Christmas presents arrive under the tree is a harrowing one that blights many a childhood. But reindeer are ...

View more: How reindeer eyes transform in winter to give them twilight vision

Decrease in CO2 emissions during pandemic shutdown shows it is possible to reach Paris Agreement goals

Global daily emissions changes in 2019 and 2020. Credit: Nature Geoscience (2022). DOI: 10.1038/s41561-022-00965-8 An international team of researchers has found that the sudden drop in CO2 emissions during the early days of the COVID-19 pandemic demonstrates that it is possible to reduce emissions enough to meet the 1.5 ...

View more: Decrease in CO2 emissions during pandemic shutdown shows it is possible to reach Paris Agreement goals

Characterizing the materials for next-generation quantum computers with nonlinear optical spectroscopy

Imaginary part of the 2D spectrum of the Kitaev ring in (a) the topologically trivial phase with μ=0.005Λ, w=Δ=0.495Λ, and (b) the nontrivial phase with μ=0.495Λ and w=Δ=0.005Λ for N=60. Credit: Physical Review Letters (2022). DOI: 10.1103/PhysRevLett.129.017401 Researchers at the Department of Physics and the Cluster of Excellence “CUI: Advanced Imaging ...

View more: Characterizing the materials for next-generation quantum computers with nonlinear optical spectroscopy

Waiting for the sun to set to find a rare bird

A 4-day old whip-poor-will chick. Like their parents, developing chicks have cryptic coloration that allows them to go unseen on the ground.  Credit: Sarah Stewart When most people are just getting home from their workdays, I’m about to start mine. I am a researcher studying the breeding behavior of ...

View more: Waiting for the sun to set to find a rare bird

Research reveals the complexity of loneliness during lockdown

Scientists unravel working principles of a promising material for fast-charging batteries

Lung cancer: When radiation does not work

Dorm decorations and essentials: Homekit lights, smart speakers, and more

Researchers find evidence for periodic disk instabilities in the massive nucleus of galaxy NGC 4258

Michio Kaku: “We need a second planet”

Speed-dating experiment reveals what makes a great first date

Fauci reports COVID rebound, says his is “much worse” than initial illness

Is Something Else Hidden at the Center of the Milky Way?

Black Friday Amazon 2022: Here Are the Best Amazon Echo Deals for Smart Speakers!

UCL releases animal research statistics alongside fellow top institutions

Hong Kong developer Billion fails to sell a single flat at The Horizon project in Tai Po

OTHER TECH NEWS

Top Car News Car News