Skip to main content

Remote Teleop

Provides a widget for remotely controlling/joysticking a robot including a low-latency video stream of one or more cameras. Establishes a direct connection to the robot using WebRTC, always preferring local connections over proxied ones.

The capability supports publishing messages on ROS topics of type geometry_msgs/Twist or sensor_msgs/Joy (resp. geometry_msgs/msg/Twist and sensor_msgs/msg/Joy in ROS 2). The video stream is provided by the WebRTC Video capability, more details on the supported video sources can be found there. Once installed, the device page on the Transitive Robotics Portal can be used to select available parameters and obtain an HTML snippet to embed the parameterized widget into other web applications.

The widget can be customized using CSS. In the default configuration the joystick is below the video, which makes it convenient to use as a soft-joystick when walking with the robot, but using CSS it can also be placed as an overlay on the video, e.g., in the lower left or right corner.

The capability also supports game controllers, e.g., XBox controllers, plugged into the same machine where the web page with the front-end widget is open. This is particularly powerful when using Joy topics, as it transmits all the axes and buttons the controller has to offer. Combined with joy_teleop running on the robot, this enables the user to control multiple degrees of freedom of the robot, e.g., arms, forks of a fork-lift, and direction of a snow-blower. The buttons can be configured to turn things on and off on the robot such as mowers, snow-blowers, or lights. See below for an example config for joy_teleop.

Disclaimer

Please be aware of the dangers of remotely controlling a robot. By using this capability you assume complete responsibility for the safe operation of the robot you are controlling using this capability and accept and agree to our Terms of Service.

Features

  • All features of WebRTC Video capability, incl.:
    • stream up to 10 cameras simultaneously
    • efficient h264 video compression
    • no need for a VPN
  • Supports both Twist and Joy topics
  • Supports both ROS 1 and ROS 2
  • On-screen soft-joystick
  • 🎮 Gamepad support
  • Click-events from clicking on video

Safety Features

  • uses the same connection for video and control, preventing control without working video
  • dead-person switch: only sends movement commands when user actively drags the joystick. Stops when:
    • network drops
    • client-side disruptions occur (e.g., a popup appears)
  • stops sending commands when connection lag increases beyond 500 ms.
  • explicit activation:
    • user needs to explicitly enable controls (on-screen joystick + gamepad) before use
    • can be disabled by setting alwayson="true" in the HTML attributes.

Dependencies

Please see the Dependencies for WebRTC Video, the capability this one is based on.

Testing

One very easy way to test this capability is using turtlesim (ros2 run turtlesim turtlesim_node) together with joy_teleop using the following config.yml, which seems to work well for Xbox controllers. This uses Button LB as dead-person switch, and the A button clears:

joy_teleop:
ros__parameters:
turtle1:
type: topic
interface_type: geometry_msgs/msg/Twist
topic_name: turtle1/cmd_vel
deadman_buttons: [4]
axis_mappings:
linear-x:
axis: 1
scale: -1
offset: 0
angular-z:
axis: 0
scale: -1
offset: 0

clear:
type: service
interface_type: std_srvs/srv/Empty
service_name: clear
buttons: [0]

Gamepads

The capability supports Xbox and PS4 controllers connected to the computer where the front-end component is rendered in a browser. When a Joy topic is used, the Xbox inputs are mapped to match the format of input generated by PS4 controllers (four axes + 17 buttons). When a Twist topic is used, only the left stick is used since Twist only has two axes: angular velocity and linear velocity.

On-screen Controls

Depending on the topic type used, the front-end will render different on-screen controls (soft-joystick). When a Twist topic is used, there is only one joystick. When a Joy topic is used, there are two joysticks plus a number of buttons. In addition, when the on-screen controls are used, the front left top button is pressed to allow its use as a dead-man switch (see the testing example above).

In both cases, when a gamepad is connected, the on-screen controls are disabled but will reflect the state of the gamepad controls.

Click Events

When the user clicks on the video image, a ROS message will be published to /transitive/remote_teleop/click in any initialized ROS versions (1 and/or 2). The topic has type geometry_msgs/Point, with:

  • x being the relative x-offset on the video (left = 0, right = 1.0), and
  • y being the relative y-offset on the video (top = 0, bottom = 1.0).

These ROS messages can be used to implement various interaction modalities on the robot, e.g., projecting the clicked point into map-coordinates and commanding the robot to navigate there.


Changelog

v0.6

  • based on webrtc-video 0.17: improved congestion control

v0.5

  • added video-click events
  • on-screen controls now show state of gamepad
  • when using a Joy topic, the on-screen controls now include two joysticks and eight buttons, arranged like a PS4 controller
  • added feature to require user to enable controls explicitly before being able to use them (this also applies to any attached gamepad)
  • v0.5.5:
    • Fixes an issue regarding the installation of ROS 2 bindings
Details
Version 0.6.0, published 3/10/2024, 8:53:24 PM
$20/robot/month Includes 10 GB of relay traffic. $0.3/GB after that. Only applies when video needs to be relayed through the cloud because no peer-to-peer connection could be established.