This project targets the problems of interaction with robots in human’s proximity, within the direct line of sight.

We study the ways to capture and use human pointing gestures with the help of wearable sensors. This approach allows to avoid the typical problems of computer vision systems, such as occlusions, low illumination, and the need for extensive computational resources. The main drawback of this approach is the lack of localization between the user and the robot—the knowledge that is usually available for “free” with vision systems. To solve it, we require the user to point at and follow the autonomously moving robot for a few seconds: during that stage we collect synchronized pairs of pointing rays and robot positions. We then align the pointing rays with robot’s trajectory to estimate the coordinate frame transformation between the agents.

An overview of our approach is shown in the following video:

For more details, please refer to the publications section at the bottom of the page.

Live Demo @HUBweek (Boston, MA)

We demonstrated our system live on October 8-9, 2018 at the Aerial Futures: The Drone Frontiers event during the HUBweek in Boston, MA, USA.

This implementation uses the MetaMotionR+ bracelet (similar to a smartwatch) and the Crazyflie 2.0 drone. Note that we do not use any external localization system, the drone is controlled directly in its (visual) odometry frame.

Acknowledgment

This work was partially supported by the Swiss National Science Foundation (SNSF) through the National Centre of Competence in Research (NCCR) Robotics.

Publications

  1. B. Gromov, G. Abbate, L. Gambardella, and A. Giusti, “Proximity Human-Robot Interaction Using Pointing Gestures and a Wrist-mounted IMU,” in 2019 IEEE International Conference on Robotics and Automation (ICRA), 2019, to appear.

    @inproceedings{gromov2019proximity,
      author = {Gromov, Boris and Abbate, Gabriele and Gambardella, Luca and Giusti, Alessandro},
      title = {Proximity Human-Robot Interaction Using Pointing Gestures and a Wrist-mounted {IMU}},
      booktitle = {2019 IEEE International Conference on Robotics and Automation (ICRA)},
      year = {2019},
      month = may,
      note = {to appear},
      video = {https://youtu.be/hyh_5A4RXZY},
    }
    
  2. B. Gromov, J. Guzzi, G. Abbate, L. Gambardella, and A. Giusti, “Video: Pointing Gestures for Proximity Interaction,” in HRI ’19: 2019 ACM/IEEE International Conference on Human-Robot Interaction, March 11–14, 2019, Daegu, Rep. of Korea, 2019.

    @inproceedings{gromov2019video,
      author = {Gromov, Boris and Guzzi, J{\'e}r{\^o}me and Abbate, Gabriele and Gambardella, Luca and Giusti, Alessandro},
      title = {Video: Pointing Gestures for Proximity Interaction},
      booktitle = {HRI~'19: 2019 ACM/IEEE International Conference on Human-Robot Interaction, March 11--14, 2019, Daegu, Rep. of Korea},
      conference = {2019 ACM/IEEE International Conference on Human-Robot Interaction},
      location = {Daegu, Rep. of Korea},
      year = {2019},
      month = mar,
      video = {https://youtu.be/yafy-HZMk_U},
    }
    
  3. B. Gromov, L. Gambardella, and A. Giusti, “Robot Identification and Localization with Pointing Gestures,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, pp. 3921–3928.

    @inproceedings{gromov2018robot,
      author = {Gromov, Boris and Gambardella, Luca and Giusti, Alessandro},
      title = {Robot Identification and Localization with Pointing Gestures},
      booktitle = {2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
      year = {2018},
      pages = {3921-3928},
      keywords = {Robot sensing systems;Robot kinematics;Solid modeling;Manipulators;Drones;Three-dimensional displays},
      doi = {10.1109/IROS.2018.8594174},
      issn = {2153-0866},
      month = oct,
      video = {https://youtu.be/VaQ3aZBf_uE},
    }
    
  4. B. Gromov, L. Gambardella, and A. Giusti, “Video: Landing a Drone with Pointing Gestures,” in HRI ’18 Companion: 2018 ACM/IEEE International Conference on Human-Robot Interaction Companion, March 5–8, 2018, Chicago, IL, USA, 2018.

    @inproceedings{gromov2018video,
      author = {Gromov, Boris and Gambardella, Luca and Giusti, Alessandro},
      title = {Video: Landing a Drone with Pointing Gestures},
      booktitle = {HRI~'18 Companion: 2018 ACM/IEEE International Conference on Human-Robot Interaction Companion, March 5--8, 2018, Chicago, IL, USA},
      conference = {2018 ACM/IEEE International Conference on Human-Robot Interaction Companion},
      doi = {10.1145/3173386.3177530},
      isbn = {978-1-4503-5615-2/18/03},
      location = {Chicago, IL, USA},
      year = {2018},
      month = mar,
      acmid = {3177530},
      publisher = {ACM},
      video = {https://youtu.be/jpG8Jsmth2Y},
    }
    
  5. D. Broggini, B. Gromov, L. M. Gambardella, and A. Giusti, “Learning to detect pointing gestures from wearable IMUs,” in Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence, February 2-7, 2018, New Orleans, Louisiana, USA, 2018.

    @inproceedings{broggini2018learning,
      author = {Broggini, Denis and Gromov, Boris and Gambardella, Luca M. and Giusti, Alessandro},
      title = {Learning to detect pointing gestures from wearable {IMUs}},
      booktitle = {Proceedings of Thirty-Second {AAAI} Conference on Artificial Intelligence, February 2-7, 2018, New Orleans, Louisiana, {USA}},
      year = {2018},
      month = feb,
      publisher = {{AAAI} Press},
      url = {https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16259/16463},
    }
    
  6. B. Gromov, L. M. Gambardella, and G. A. Di Caro, “Wearable multi-modal interface for human multi-robot interaction,” in Safety, Security, and Rescue Robotics (SSRR), 2016 IEEE International Symposium on, 2016, pp. 240–245.

    @inproceedings{gromov2016wearable,
      title = {Wearable multi-modal interface for human multi-robot interaction},
      author = {Gromov, Boris and Gambardella, Luca M and Di Caro, Gianni A},
      booktitle = {Safety, Security, and Rescue Robotics (SSRR), 2016 IEEE International Symposium on},
      pages = {240--245},
      year = {2016},
      organization = {IEEE},
      doi = {10.1109/SSRR.2016.7784305},
      video = {https://youtu.be/FWMCxARQYhY}
    }