Remote Teleop

Reliable remote teleoperation (teleop) for your robots. This capability provides a front-end UI component that allows you to remote control/joystick a robot including a low-latency video stream of up to ten cameras. Establishes a direct connection to the robot using WebRTC, always preferring local connections over proxied ones.

The capability supports publishing messages on ROS topics of type geometry_msgs/Twist or sensor_msgs/Joy (resp. geometry_msgs/msg/Twist and sensor_msgs/msg/Joy in ROS 2). The video stream is provided by the WebRTC Video capability, more details on the supported video sources can be found there. Once installed, the device page on the Transitive Robotics Portal can be used to select available parameters and obtain an HTML snippet to embed the parameterized widget into other web applications.

The widget can be customized using CSS. In the default configuration the joystick is below the video, which makes it convenient to use as a soft-joystick when walking with the robot, but using CSS it can also be placed as an overlay on the video, e.g., in the lower left or right corner.

The capability also supports game controllers, e.g., XBox controllers, plugged into the same machine where the web page with the front-end widget is open. This is particularly powerful when using Joy topics, as it transmits all the axes and buttons the controller has to offer. Combined with joy_teleop running on the robot, this enables the user to control multiple degrees of freedom of the robot, e.g., arms, forks of a fork-lift, and direction of a snow-blower. The buttons can be configured to turn things on and off on the robot such as mowers, snow-blowers, or lights. See below for an example config for joy_teleop.

Please be aware of the dangers of remotely controlling a robot. By using this capability you assume complete responsibility for the safe operation of the robot you are controlling using this capability and accept and agree to our Terms of Service.

• All features of WebRTC Video capability, incl.:
  • stream up to 10 cameras simultaneously
  • efficient h264 video compression
  • support ROS and ROS2 image topics
  • no need for a VPN
• Supports both Twist and Joy topics
• Supports both ROS 1 and ROS 2
• On-screen soft-joystick
• 🎮 Gamepad support
• Click-events from clicking on video

• uses the same connection for video and control, preventing control without working video
• dead-person switch: only sends movement commands when user actively drags the joystick. Stops when:
  • network drops
  • client-side disruptions occur (e.g., a popup appears)
• stops sending commands when connection lag increases beyond 500 ms.
• explicit activation:
  • user needs to explicitly enable controls (on-screen joystick + gamepad) before use
  • can be disabled by setting alwayson="true" in the HTML attributes.

Please see the Dependencies for WebRTC Video, the capability this one is based on.

One very easy way to test this capability is using turtlesim (ros2 run turtlesim turtlesim_node) together with joy_teleop using the following config.yml, which seems to work well for Xbox controllers. This uses Button LB as dead-person switch, and the A button clears:

joy_teleop:
  ros__parameters:
    turtle1:
      type: topic
      interface_type: geometry_msgs/msg/Twist
      topic_name: turtle1/cmd_vel
      deadman_buttons: [4]
      axis_mappings:
        linear-x:
          axis: 1
          scale: -1
          offset: 0
        angular-z:
          axis: 0
          scale: -1
          offset: 0

    clear:
      type: service
      interface_type: std_srvs/srv/Empty
      service_name: clear
      buttons: [0]

The capability supports Xbox and PS4 controllers connected to the computer where the front-end component is rendered in a browser. When a Joy topic is used, the Xbox inputs are mapped to match the format of input generated by PS4 controllers (four axes + 17 buttons). When a Twist topic is used, only the left stick is used since Twist only has two axes: angular velocity and linear velocity.

Depending on the topic type used, the front-end will render different on-screen controls (soft-joystick). When a Twist topic is used, there is only one joystick. When a Joy topic is used, there are two joysticks plus a number of buttons. In addition, when the on-screen controls are used, the front left top button is pressed to allow its use as a dead-man switch (see the testing example above).

In both cases, when a gamepad is connected, the on-screen controls are disabled but will reflect the state of the gamepad controls.

When the user clicks on the video image, a ROS message will be published to /transitive/remote_teleop/click in any initialized ROS versions (1 and/or 2). The topic has type geometry_msgs/Point, with:

• x being the relative x-offset on the video (left = 0, right = 1.0), and
• y being the relative y-offset on the video (top = 0, bottom = 1.0).

These ROS messages can be used to implement various interaction modalities on the robot, e.g., projecting the clicked point into map-coordinates and commanding the robot to navigate there.

• Added bandwidth optimization.
  • This saves bandwidth when the scene in the video is mostly static by using variable bitrate encoding. When using the "high" setting for this, it can reduce bandwidth to as low as 5KB/s.
• Reduced latency to around 100ms.
• Increased robustness:
  • Better error handling and reporting when sessions fail.
  • Eliminated more cases where a request could get "stuck", blocking future sessions using the same parameters.
  • Reduced delay in starting the stream in the face of packet loss when using NVIDIA hardware encoders.

• v0.9.1:

  • Add quantizer setting (bandwidth optimization) to config provided in Embed.
  • Correctly set bitrate, peak-bitrate, and/or maxbitrate on NVIDIA jetson devices.

• v0.9.2:

  • Ease-of-use: update video sources on page load and right away when a session ends, to update busy status of v4l2 devices.
  • New feature: auto-bitrate mode. This allows you to leave the bitrate unspecified, letting the capability decide automatically what an appropriate bitrate is given the video sources you selected. This, together with the new bandwidth optimization feature, makes it extremely easy to get good quality streams without any tuning.

• v0.9.3:

  • Multi-touch support: you can now use both on-screen axes at the same time, e.g., for landscape layouts on an iPad with one axes in each bottom corner of the screen.

• v0.9.4:

  • Safety: Detect stale video sources (not just lag) and disable controls.
  • Ease-of-use: Automatically recover from busy USB cameras and timeouts receiving ROS messages.
  • Ease-of-use: Always send control commands while a button is pressed (not just on press/release).
  • Namespace ROS 2 nodes in /transitive.

• v0.9.5:

  • Fixes a race condition with ROS topics on NVIDIA hardware encoders (VIC only, dGPU was not affected).

• v0.9.8:

  • Fixes a bug in one of the safety mechanisms that affected iOS devices (enabled the safety lock out all the time due to "old frames").

• v0.9.10:

  • Fixes a bug when doing local recording of ROS topic sources
  • Sets the default MTU to 1,200, adds new parameter rtpMtu to overwrite this

• Improves CPU efficiency of h264 encoding by 3x when using software encoder.
• Adds ability to customize test-image resolution and framerate.

• Based on webrtc-video 0.17: improved congestion control.

• Added video-click events.
• On-screen controls now show state of gamepad.
• When using a Joy topic, the on-screen controls now include two joysticks and eight buttons, arranged like a PS4 controller.
• Added feature to require user to enable controls explicitly before being able to use them (this also applies to any attached gamepad).

• v0.5.5:

  • Fixes an issue regarding the installation of ROS 2 bindings.