Applications

In all systems that contain interactions between humans and machines, the element that defines the quality of that interaction is the human/machine interface, also known as the user interface. A graphical user interface (GUI) can make the human interaction more engaging and intuitive, constrain user inputs to valid ranges and units, supply tabular and plot-based output where needed, and provide users with data from sensors on the robots that are needed for control.

JPL Robotics has created a number of very successful user interfaces that have been tuned to the robotic systems being controlled. These robots have ranged from multi-arm manipulators for microgravity deployment or terrestrial surgical applications, to legged prototype-rovers for rough terrains, to multi-wheeled rovers including flight systems such as the Mars Pathfinder Sojourner Rover, the Mars Exploration Rovers (MER), Spirit and Opportunity, the Phoenix Lander, the Mars Science Laboratory (MSL) rover, Curiosity, and the current Mars 2020 (M2020) rover and Helicopter, Perseverance and Ingenuity. Current robot-human interface software developed at JPL for the Mars missions are:

RSVP (Robot Sequencing and Visualization Program)

RSVP has been used for all the surface operation in Mars missions since 1997. It is used to create all the command sequences for robotic operation sent to the Mars rovers and landers of all the Mars missions, including MER, MSL, Phoenix, MSL, InSight and M2020.

The RSVP suite of robot-human interfaces consists of the following tools:

• RoSE (Rover Sequence Editor), which provides efficient GUI support to all command in the mission-command dictionary and also interfaces with the simulation tools for resource computations and sequence validation.
• HyperDrive provides multiple 3D graphics interfaces to the stereo images and 3D terrain-model data. It is used to plan all rover motions, including driving over the surface of Mars and complex robotic arm motions to place instruments on sites of scientific interests.
• ArmSketch is a visual sequencing tool, which generates arm sequences by optimizing path and collision avoidance algorithm.
• MobSketch is a visual sequencing tool which generates drive sequences by expanding pre-defined macros.
• ROCS (Robotic Operations and Computational Services) is a set of stateless, RESTful API that provides useful computations for rover simulation, planning, and operations. The ROCS suite includes services which calculate inverse kinematics (arm and RSM), time conversions, shadowing, frame transformations, terrain settling, path planning, and terrain classifier.

SSim (Surface Simulation)

SSim is flight-software-in-the-loop simulation software that predicts what a given set of rover commands will do before uplinking the commands to the rover on Mars. Its key focus is supporting the simulation of rover mobility and arm activities. It is required to do this quickly (thousands of times real time) so that rover planners can quickly iterate the development of their commands with the simulation. SSim takes as input the rover command sequences, terrain models, the rover nonvolatile memory state, and other parameters. It outputs the rover motion history, command durations, and flight software log messages. SSim was first developed for use on MSL as part of the RSVP tool suite.

In M2020, SSim is contained within RSVP like on MSL, but it is also a web service used by validation and verification of all the sequences. Simulating the uplinking command sequences SSim generates the predicted states and flight software output EVent Report (EVR) for command validation, onboard resources prediction and flight-rule checking.

Honeycomb 3D visualization platform

Honeycomb is a general, web-based 3D visualization platform designed with the goal of easily integrating with missions to help in understanding the state and intent of robotic systems at various stages of development from hardware development to operations. It can be deployed as both a browser tool or a desktop tool to accommodate the needs of a userbase and project.

On Mars 2020 Honeycomb is used in a variety of tools including visualizing autonomy products from the enhanced driving navigation system. Mars 2020 improves on the MSL autonomous driving capabilities by enabling the rover to plan ahead while driving in parallel making it much more complicated. The visualization tool displays flight products used during autonomous driving including cost map, heightmap, potential drive arcs, perceived obstacles, and more to help operators and developers understand and debug the driving process. Users can toggle individual displays, scrub through the timeline, and interact with the 3D view to dig into the algorithm and ultimately get into the “mind” of the rover to understand the autonomous decisions made while driving.

Honeycomb has been used for mission mobility development and V&V as a desktop tool and is used by operators as a web tool during Mars operations to comprehend and diagnose system behavior. The platform is planned to be expanded for applications on other missions.

MobSketch, Mars 2020 Mobility Planning Tool

MobSketch is a tool designed to integrate with existing rover operations tooling to enable quick and easy drive path planning. Operators can load terrain products, drag and place drive waypoints, and adjust drive parameters to define arcs, backwards drives, turns, and precision drives. Individual parameters can be manually adjusted through a user interface panel and terrain can be loaded for context while planning. The drive path is automatically expanded into flight commands and can be simulated and previewed to validate the plan.