Building a brighter future with robotics

From exploring shipwrecks to air-dropping supplies in a pandemic, U of M faculty and students are designing robots for the 21st century.

Scroll to explore

Sitting on the floor, a toddler listens to her mechanical companion, who also sits.

“Clap your hands. Can you clap your hands?” her companion says.

The girl claps enthusiastically. She then stands up and dances vigorously to pop music with her companion. When all is over, she reaches down, pats the motionless companion on the head, and says to the young woman who’s been watching: “I like your robot.”

The scene was a baseline study of how young, healthy children interact with robots like this toddler's companion. The young woman, Marie Manner, was a graduate student with Maria Gini, a professor in the University of Minnesota Department of Computer Science and Engineering. Manner was investigating whether humanoid robots could help push back the age when autism can first be detected so that treatment may begin earlier. The idea is that children with and without autism in their future may interact with a robot—a standard, bias-free presence—in subtly different ways.

Maria Gini with two robots in her lab

New students of Gini’s are often surprised to discover that she employs a pair of small, fully automated robots in her classroom as exemplars of state-of-the-art robotic devices

toddler inspects robot with look of awe and curiosity

—and as a sure-fire way of engaging their imaginations.

A specialist in artificial intelligence (AI), Gini is a mainstay of MnRI, the U of M’s Minnesota Robotics Institute, a unit of the College of Science and Engineering (CSE). She and other MnRI faculty are designing and building robots to perform in ways that mimic the human ability to collect information, process it, and act based on it—in other words, to learn from experience.

In the long term, in trying to make [robotics] work in the real world, we develop some understanding of ourselves.—Volkan Isler, professor

Humans do this without giving it a thought. Two people asked to move the same heavy object from point A to point B will naturally try to lift or push it together. But robots must be taught to make connections like this, which means their human designers must know not only about circuits and digital messaging, but about how their own brains work.

Challenges like this excite MnRI researchers, from undergrads up through seasoned professors of computer science and engineering like Gini, Volkan Isler, and MnRI Director Nikolaos Papanikolopoulos. In fact, Papanikolopoulos says the dedication of students and former students makes his job a joy.

“Seeing them lead the pack in industry, seeing them create hundreds of jobs in Minnesota—I never imagined, as a young student in Greece, I’d be part of such a thing,” he says.

Mastery at a young age

From its birth in 2019, MnRI has been luring top students from the U of M and around the world and bringing them together with faculty in Shepherd Laboratory on the Twin Cities campus. Among its distinctions, MnRI offers a rare, three-semester M.S. in Robotics program.

The U of M is one of the top universities for people who want to pursue a degree in robotics. I would say the laboratory resource is the most valuable, as you can dive deeper into an interesting topic with all these experts.—Jun-Jee Chao, M.S. student

“A master’s degree in robotics allows you to explore many possibilities, as some might be interested in programming while others are more interested in hardware design,” says M.S. student Jun-Jee Chao. “The Robotics Institute provides lots of resources for you to discover your interest."

Adds fellow M.S. student Kai Wang: “I found my interest in computer vision and robotics in my junior year [at the U of M]. This degree offered an opportunity to take more professional courses and to do hands-on research in robotics.

“The U has a really strong robotics department and a powerful Gemini-Huntley Robotics Research Laboratory. The most valuable part [for me] is definitely the research experience in the Robotic Sensor Networks Laboratory—it gives me a real picture of today's field robots.”

3d illustration of robot that could care for COVID-positive patients and limit exposure to healthcare professionals

An undergraduate initiative to fight COVID-19

With MnRI assistance, a team of undergrads designed a prototype robot to minister to COVID-19 patients and reduce potential transmission to health care professionals. Team members worked tirelessly to source, manufacture, and assemble parts for the robot. The robot has been mechanically constructed and undergone preliminary testing, which involved "the thermal camera (taking temperatures) and overall usability of the remote control system, where we had a couple people learn to drive it and suggest improvements," says team member Julia Schatz.

Send in the Scouts

Some of the earliest robots built at the U of M came out of Papanikolopoulos' and Gini's labs. Called Scouts, these autonomous robots resembled soda cans with wheels at either end and could both roll and jump. They were designed to enter and relay information from dangerous situations, such as what soldiers and police may encounter, even in total darkness. They have been deployed in dozens of countries, and today their descendants are learning to scale previously insurmountable obstacles. Graduate student Dario Canelon-Suarez is researching the next generation of these robots (see "This is not science fiction" video, above).

Also, Ruben D’Sa, a former graduate student in Papanikolopoulos’ lab, designed an unmanned aerial vehicle (UAV) that can take off vertically as a typical multirotor drone and then, in midair, unfold flaps and transform into a fixed-wing aircraft. This dual nature combines the efficiency and range of a fixed-wing aircraft with the maneuverability and hovering capabilities of a multirotor platform, which can be critical in pickup and delivery missions.

Flying robot switches shape in mid-air.

Self-driving “cowbot” mows a pasture.

Robots in the heartland

Isler has long worked on sensing systems and developed a system to track invasive fish. Now, he’s designing robots that can manipulate their environments. One, the “cowbot,” is trained to navigate around pastures after cows have grazed them and mow leftover weeds—like a rural Roomba. Why use a robot? Because pastures make for a jarringly rough ride.

Leading the project are two members of Isler’s Robotic Sensor Networks Lab: postdoc Parikshit Maini and Ph.D student Minghan Wei. The team modified a lawn mower and is collaborating with the U of M's West Central Research and Outreach Center to make the machine solar-powered and self-sufficient.

“We just finished one big field test. We’re getting good performance,” says Isler. “It now follows a given trajectory. The next stage is, we’re going to make it detect weeds and avoid obstacles.”

Flying robot monitors apples in an orchard.

Isler’s group has also designed a flying robot that can monitor orchards, and has a project on robotic fruit picking.

“We can count apples and measure their size across an entire orchard," Isler says. "There’s now a U of M startup [Farm Vision Technologies] commercializing this technology.”

Isler and David Mulla, director of the U of M Precision Agriculture Center in the College of Food, Agricultural and Natural Resource Sciences, have a patent on a system to combine the talents of ground and aerial robots to monitor farm fields and apply water or nutrients only and exactly where needed. This practice will improve yields while eliminating excessive water use and runoff of nutrients into waterways.

Protecting lakes, oceans, and streams

In Junaed Sattar’s lab, swimming robots learn to outperform humans. Someday, one could, for example, walk to a lake, dive and take samples of mud or organisms, then surface and walk back to the lab, he says.

An assistant professor of computer science and engineering, Sattar works with autonomous underwater vehicles (AUVs) equipped with sensors to help them make intelligent decisions. They have profound potential in hazardous situations, such as searching shipwrecks or clearing lakes of invasive species. His team can, for example, train robots to identify and locate invasive weeds like Eurasian watermilfoil, which changes water chemistry and affects wildlife important to the Minnesota economy.

It takes the DNR (state Department of Natural Resources) three weeks to clear a lake, then multiply that by 10,000 lakes. We’d like robots to do this dangerous task by themselves.—Junaed Sattar, assistant professor

His AUV sensors can identify objects like rocks, fish, plants, and shipwrecks. The AUVs could learn to retrieve objects from wrecks, and even have a special algorithm for robots to see better in spite of artifacts such as bubbles, the bane of many an AUV.

A team of robots could, he says, scour a lake bottom, take images and sensor data, then provide that to experts. Or monitor the health of coral reefs.

As Sattar’s team works, the shadow of Malaysia Airlines Flight 370, lost in the Indian Ocean in March 2014, is never far away.

“If they find the wreckage, people will want black boxes,” Sattar says. “That’s one of our biggest motivations.”

The underwater domain poses unique challenges. For example, neither GPS, Wi-Fi, phones, nor any other device that uses electromagnetic waves will work underwater. Sattar’s team has only cameras, and acoustic (sonar) pings to work with.

In many human-robotic interactions, humans learn to speak "Robot." My goal is to make robots learn "Human." This won’t be solved by computer scientists or electrical engineers alone. We need people from psychology, math, life sciences, cognitive sciences, linguistics, and so on so we can look at the elephant from all angles.—Junaed Sattar, assistant professor

Sattar has personally witnessed “a lot of kids who want to do underwater human-robot interaction.” Today, however, his is one of only a few groups worldwide that work on them, he says.

His team, including students—grad, undergrad, and even high school—built the LoCO AUV in-house for only $4,000. Underwater robots usually cost six figures, he says, but “we made LoCO available open source.”

LoCO has performed well in pool tests and field trials in both the Caribbean Sea off Barbados and Minnesota’s Lake Minnetonka.

Safe robots

As drones handle more pickups and deliveries, especially in high-traffic areas, the specter of collisions and “mission failure” grows. But drones aren’t cheap, and some payloads are priceless. To build reliable drones, researchers like Derya Aksaray—who with her students implements algorithms on real robots—first provide “proof of concept.”

illustration flying drones carrying packages with cloudy sky in the background

“We developed a priority-based planning algorithm to ensure collision avoidance among drones,” says Aksaray, an assistant professor of aerospace engineering and mechanics.

“For example, if two drones have pickups at the same location, the drone closer to its mission completion gets higher priority. Robots would start communicating when they get close and can see each other via cameras.

“We can make robots avoid collisions and complete their missions on time or within a tolerable delay. We’re collaborating with Honeywell on safe autonomy and getting industrial feedback.”

Also for robots flying solo, say, doing a survey of a farm field, Aksaray uses reinforcements—rewards—to get them to focus on areas that require more attention.

“Suppose a drone explores, trying to find issues [like poor crop growth],” she explains. “At first it tries a certain trajectory and spends one minute in each location. Back at base station, humans could look at the coordinates of the various locations explored and reward interesting ones [that need attention] with points.”

Next, says Aksaray, the drone would start again, this time apportioning its time according to how many points each location earned.

In these and related projects, Aksaray has one goal: “I’m interested in developing provably correct algorithms that don’t just work, but can be counted on to work all the time.”

The challenge of ordinary conversation

Can robots learn human-level skills in understanding and producing speech? Maria Gini has set her sights squarely on answering that central question.

She has produced a prototype “chatbot” for radio stations. It will answer common listener questions, like “What were those last two songs?” Eventually, the chatbots will have voices and personalities to fit each station’s style.

Another project addresses the problem of getting robots to work together by, for example, pushing the aforementioned heavy object.

“One question is, Do they need language or some kind of signaling—perhaps through gestures—or do they learn in a random way?” Gini says. “That project is in the early stages.”

And then there’s the challenge of creating robots that can realistically converse with people, especially those who need help. This multilayered work brings in colleagues from the Colleges of Design (notably Professor Lucy Dunne, a specialist in wearable technology), Liberal Arts (Psychology), and Pharmacy, as well as CSE.

“We want to see if there’s a correlation between what people say and what amount of stress they’re experiencing,” Gini explains. “Can we get, for instance, a more sophisticated watch that can maybe say, ‘Whoa, looks as though you’re stressed’?"

She notes how compression vests are used to calm autistic children and envisions one that can tell from physiological data that something’s wrong and then say, “I’ll give you a hug” or simply warm the body. Gini is also near the end of a two-year project to design a robot that can, for example, remind people of tasks or get them to talk about their lives and store that information.

It goes way beyond what Alexa can do. —Maria Gini, professor

As Gini envisions it, “I’m trying to have a real conversation. The program will figure out what I’m saying. Am I asking a question or making a statement? What am I talking about?”

She’s convinced that organization is key. “People learn how to build sentences from examples,” Gini says. “We have memory structures. Will AI be able to construct them?”

This is an interesting time for AI, says Gini, thanks to today’s immense computing power and the questions it raises.

“With more computational power, will computers be able to learn everything?” she muses. “Or is there something unique about the human brain?”

Meet the people behind the story

Nikolaos Papanikolopoulos, PhD

Nikolaos Papanikolopoulos, PhD

McKnight Presidential Endowed Professor
Minnesota Robotics Institute Director
College of Science and Engineering

Maria Gini, PhD

Maria Gini, PhD

Professor, College of Science and Engineering

Volkan Isler, PhD

Volkan Isler, PhD

Professor, College of Science and Engineering

Derya Aksaray, PhD

Derya Aksaray, PhD

Assistant Professor, College of Science and Engineering

Junaed Sattar, PhD

Junaed Sattar, PhD

Assistant Professor, College of Science and Engineering

Dario Canelon Suarez, PhD

Dario Canelon Suarez, PhD

Graduate Research Assistant, Center for Distributed Robotics

Marie Manner, PhD

Marie Manner, PhD

Former Graduate Student, College of Science and Engineering