Provides a widget for remotely controlling/joysticking a robot including a low-latency video stream of one or more cameras. Establishes a direct connection to the robot using WebRTC, always preferring local connections over proxied ones.
The capability supports publishing messages on ROS topics of type
sensor_msgs/msg/Joy in ROS 2). The video stream is provided by the WebRTC Video capability, more details on the supported video sources can be found there. Once installed, the device page on the Transitive Robotics Portal can be used to select available parameters and obtain an HTML snippet to embed the parameterized widget into other web applications.
The widget can be customized using CSS. In the default configuration the joystick is below the video, which makes it convenient to use as a soft-joystick when walking with the robot, but using CSS it can also be placed as an overlay on the video, e.g., in the lower left or right corner.
The capability also supports game controllers, e.g., XBox controllers, plugged into the same machine where the web page with the front-end widget is open. This is particularly powerful when using
Joy topics, as it transmits all the axes and buttons the controller has to offer. Combined with
joy_teleop running on the robot, this enables the user to control multiple degrees of freedom of the robot, e.g., arms, forks of a fork-lift, and direction of a snow-blower. The buttons can be configured to turn things on and off on the robot such as mowers, snow-blowers, or lights. See below for an example config for
Please be aware of the dangers of remotely controlling a robot. By using this capability you assume complete responsibility for the safe operation of the robot you are controlling using this capability and accept and agree to our Terms of Service.
- All features of WebRTC Video capability, incl.:
- stream up to 10 cameras simultaneously
- efficient h264 video compression
- no need for a VPN
- Supports both
- Supports both ROS 1 and ROS 2
- On-screen soft-joystick
- 🎮 Gamepad support
- Click-events from clicking on video
- uses the same connection for video and control, preventing control without working video
- dead-person switch: only sends movement commands when user actively drags the joystick. Stops when:
- network drops
- client-side disruptions occur (e.g., a popup appears)
- stops sending commands when connection lag increases beyond 500 ms.
- explicit activation:
- user needs to explicitly enable controls (on-screen joystick + gamepad) before use
- can be disabled by setting
alwayson="true"in the HTML attributes.
Please see the Dependencies for WebRTC Video, the capability this one is based on.
One very easy way to test this capability is using
ros2 run turtlesim turtlesim_node) together with
joy_teleop using the following
config.yml, which seems to work well for Xbox controllers. This uses Button
LB as dead-person switch, and the
A button clears:
The capability supports Xbox and PS4 controllers connected to the computer where the front-end component is rendered in a browser. When a Joy topic is used, the Xbox inputs are mapped to match the format of input generated by PS4 controllers (four axes + 17 buttons). When a Twist topic is used, only the left stick is used since Twist only has two axes: angular velocity and linear velocity.
Depending on the topic type used, the front-end will render different on-screen controls (soft-joystick). When a Twist topic is used, there is only one joystick. When a Joy topic is used, there are two joysticks plus a number of buttons. In addition, when the on-screen controls are used, the front left top button is pressed to allow its use as a dead-man switch (see the testing example above).
In both cases, when a gamepad is connected, the on-screen controls are disabled but will reflect the state of the gamepad controls.
When the user clicks on the video image, a ROS message will be published to
/transitive/remote_teleop/click in any initialized ROS versions (1 and/or 2). The topic has type
xbeing the relative x-offset on the video (left = 0, right = 1.0), and
ybeing the relative y-offset on the video (top = 0, bottom = 1.0).
These ROS messages can be used to implement various interaction modalities on the robot, e.g., projecting the clicked point into map-coordinates and commanding the robot to navigate there.
- added video-click events
- on-screen controls now show state of gamepad
- when using a Joy topic, the on-screen controls now include two joysticks and eight buttons, arranged like a PS4 controller
- added feature to require user to enable controls explicitly before being able to use them (this also applies to any attached gamepad)
- Fixes an issue regarding the installation of ROS 2 bindings