Deep Learning for Object Recognition on a Mobile Robot

Name: Bill Smart
Affiliation: Oregon State University
Phone: (541) 737-0670
Knowledge Required: machine learning; deep networks; python programming; Robot Operating System (ROS) is a plus but not strictly required.
Motivation: We want to use deep learning on our robots, but we're having a hard time collecting enough training data by hand. This will help. We'd also like to publish the code that comes out of this project in the ROS ecosystem, so that other people can use it.
Description: Deep learning is a very effective technique for object recognition from still images of the world, outperforming most other techniques. However, it relies on large, labeled training sets on which to learn. This makes it problematic for use with robots, since gathering data on a real robot platform is often very time-consuming. The goal of this project is to implement a framework that harnesses the power of deep learning on a real mobile robot system, allowing it to autonomously collect data and train object detectors with a minimum of human supervision.
Objectives: Implement a framework that will: (1) allow a human to specify an object of interest in a camera image; (2) localize this object in 3D space, using the sensors on one of our robot platforms; (3) take a large training set of images on which to train a deep network, from different viewpoints, using the known 3D location of the object to label each image; (4) train a deep network to recognize the object; (5) find new instances of the object, ask a human for confirmation, and then train the network to also recognize these; and (6) demonstrate that this whole thing works on a real mobile robot platform. Stretch goals are: (1) incrementally build the training set, selecting new viewpoints based on the performance of the system on a test set (active learning); and (2) investigate the utility of object recognizers trained on a specific instance of an object (eg, my cup) vs. those trained on classes of objects (eg, all cups).
Deliverables: Code that implements the project objectives, published in the ROS ecosystem, under an open-source license.
Other comments: A lot of the robot-specific parts of this project are already available in ROS, so don't be put off if you have never made a robot move before. Much of this will be wrapping up existing frameworks (ROS, deep learning), and building a system out of them. It's not as much work as you might think after reading the objectives.

   D. Kevin McGrath
   Last modified: Tue Mar 6 10:29:08 2018