Skip to main content

Arm Control

Remote Teleop provides several means for controlling robotic arms, whether for ongoing teleoperation or on-demand intervention and tele-assist.

Mobile robots can be easily and intuitively controlled using just two axes (forward/backward, left/right), and since the operating domain of mobile robots is not bound, it makes sense for operators to control these robots using speed control, i.e., by expressing continuous relative change along these axes, e.g., "go forward". This is not the case for arms. Arms typically have 6 or more degrees of freedom and in very limited 3D envelope. There are no good controller modalities for simultaneous speed control in that many dimensions, but fortunately, because of the limited range of operation, we can resort to position control instead, specifically relative position control.

VR + controllers

Using VR controllers is ideal for continuous bi-manual teleoperation.

VR headsets detect and report the absolute pose (position + orientation) of VR controllers in the 3D space in front of the VR headset. In addition, the buttons and gamepad controls on the controllers are reported.

Remote Teleop uses WebXR to render the configured video streams in the VR viewer and to retrieve and transmit the controllers poses and their gamepad controls. When you open the Remote Teleop capability in the browser or web-view of a VR headset it will automatically detect the support for immersive augmented reality and show a "Start XR" button that is enabled as soon as the WebRTC connection is established and the video starts streaming. Clicking the button will start an immersive AR session showing the video and monitoring the controls.

Usage: front-end

Pose

Press and hold the grasp button to transmit controller pose-changes to your robot. The pose transmitted to the robot is relative to the pose of the controller when the grasp button was pressed. This is how relative position control works and it's conceptually the same as what you are used to from using a mouse to control a cursor on your screen.

Gamepad + Trigger

The gamepad controls and the trigger button in front are sent separately whenever any button or control is pressed/exercised.

Usage: robot

On the robot, Remote Teleop publishes the received poses and gamepad controls in various topics on all enabled ROS distributions (1 and/or 2) and zeroMQ, if enabled (see the zeroMQ example on the Remote Teleop page).

  • /transitive/remote_teleop/controller/right: continuously publishes the pose of the right-hand controller while the grasp button is pressed. It publishes one zero message (all coordinates in position and orientation = 0) when the grasp button is released. This is an important signal for your arms IK controller: it signals that the next non-zero message resets the reference pose relative to which future poses received on this topic are to be interpreted. This is like lifting your mouse off the table and setting it back down somewhere else.

  • /transitive/remote_teleop/controller/left: the same as the above but from the left-hand controller.

  • /joy_vr/right: the gamepad controls of the right controller, including the trigger button, represented as an array of real-values numbers: the first two values are always zero, the third and forth represent the gamepad joystick axes, the fifth is the trigger button, which is a good candidate for controlling the gripper on your robotic arm.

  • /joy_vr/left: the same as the previous but from the left controller.

How you use these messages to control your robot is up to you, but we recommend using the received pose messages to generate Cartesian goal poses as input for your IK solver, resetting the reference pose when the pose message is all zeros, and using the absolute state of the trigger button to control the absolute state of your gripper, i.e., when the trigger button is released, bring the gripper into a fully open position, and close it when the trigger button is squeezed.

Mouse + Keyboard

Remote Teleop implements a novel, proprietary control modality we have developed as an alternative to VR.

Using a mouse and keyboard is ideal for occasional control such as for on-demand interventions ("tele-assist"), such as when your robot is acting autonomously most of the time but needs a little bit of help from time to time. Mouse + keyboard do not provide the same high-dimensional input as VR controllers, but starting a control session is much faster since the operator doesn't need to put on the headset and enter an XR session. It also drastically increases the set of people who can provide such guidance to the robot, since no specialized hardware or software is required. Furthermore, it causes operators much less fatigue than the continuous use of VR.

Usage: front-end

To control your robot using mouse and keyboard, enable "mouse gestures" in the control and/or set it in the embedding React/HTML code. When enabled, mouse gestures on the video streams themselves produce control input that is sent to the robot, labeled with the video stream the gesture originated from.

The gestures follow the controls of online maps and CAD software: dragging (press mouse button and move) will "drag the point in the video to track your mouse". This is a non-trivial, proprietary modality specific to Transitive's Remote Teleop, and an intuitive way for operators without any prior training to control "the scene" -- and hence the robot. These controls are published as wrenches on the robot.

Specifically:

  • drag with left mouse button: rotate up/down (y) and left/right (z) so that the point under the mouse cursor follows the cursor;
  • drag with middle mouse button: translate up/down and left/right;
  • drag with right mouse button: open/close gripper;
  • mouse wheel: "zoom" in and out, i.e., translate forward/backwards;
  • shift + mouse wheel: end-effector roll (rotate around x).

Usage: robot

On the robot, these controls are published as geometry_msgs/Wrench messages, or equivalent JSON messages in zeroMQ:

# This represents force in free space, separated into its linear and angular parts.

Vector3  force
	float64 x
	float64 y
	float64 z
Vector3  torque
	float64 x
	float64 y
	float64 z

Topics:

  • /transitive/remote_teleop/gesture: publishes the wrenches required to achieve the results implied by the operators gestures, as described in the front-end section above.
  • /transitive/remote_teleop/gesture_stamped: the same as the previous but including a header that indicates the frame_id the control is referencing, namely transitive_remote_teleop_camera_N where N is the zero-based index of the camera stream the control originated from (for instance, 0 = left wrist camera, 1 = right wrist camera), as well as the robot-timestamp of the last video frame acknowledged by the front-end (to implement safety features as discussed on the Remote Teleop main page).
  • /transitive/remote_teleop/gesture_secondary: publishes the gripper control (dragging with right mouse button) in force.x
  • /transitive/remote_teleop/gesture_secondary_stamped: similar to /gesture_stamped but for the gripper control.

Gamepad

Lastly, for fine-grained speed control or to supplement mouse and keyboard controls, you can use the same /joy controls published by attached game controllers and the on-screen gamepad for controlling the motions of your arm. The combination of gamepad + mouse also makes sense for controlling mobile manipulators, i.e., to control mobile base and attached arms simultaneously.