HECTUR

1. Introduction

(a) End Goal

The goal of HECTUR (Head-Eye Control and Tracking for UR7e) is to develop an accessible, non-invasive human-robot interface that allows users to control a robotic manipulator using head gestures, eye blinks, and facial movements. The system enables users–particularly individuals with limited limb mobility–to teleoperate a UR7e robotic arm using only a standard webcam and computer vision, without relying on expensive prosthetics or neural-based interfaces. The project aims to translate intuitive facial and head movements into safe, real-time, velocity-based robotic control, suitable for performing simple, day-to-day manipulation tasks.

(b) Interesting Problems

We were interested in this project because it tackles a real-world accessibility problem using affordable equipment by combining computer vision and robotic control. We had to figure out how to extract a 3D head pose and facial state from a 2D live video captured by a simple monocular RGB camera. Then, we had to ensure that noisy, unintentional, instinctive actions were filtered and smoothed. Lastly, with limited measurable DOF, we had to design an intuitive yet capable control system with quality-of-life features for common everyday tasks.

(c) Real-World Applications

2. Design

(a) Design Criteria

Our project must be able to sense, plan, and actuate. We wanted our robot to be able to detect head and facial gestures reliably and use that as human input to safely teleoperate the robot arm. For planning, we designate a checkpoint that the robot can automatically return to, both on command. We also want to be able to recenter (adjust the zero state) of the user's head on command and emergency stop the arm on command.

(b) Design Choice

The system uses MediaPipe Face Mesh to track 468 facial landmarks and compute head yaw, pitch, and roll; eye aspect ratio for blink detection; and mouth openness as a discrete command (either checkpoint saving/returning or gripper toggle). These signals are processed through a ROS2-based pipeline consisting of a sensing node (head_pose_blink_node), a mapping node (head_teleop_mapper_node), and a robot control node (facemesh_ur7e_control_node). Processing includes applying a significant deadzone around the neutral head and facial state, then smoothing out the signal and filtering any estimations.

Control Mappings

State #1:

State #2:

Eye-Blink Commands (3 seconds):

To return to a waypoint, knowing the goal end-effector pose and the current joint state, we compute the inverse kinematics and use that to form a trajectory via MoveIt.

(c) Design Choices & Trade-offs

• Velocity-based control instead of position control for smoother, safer, and more precise motion. We did not want the robot to move at a high speed spontaneously in the case that we fail to catch a noisy measurement.
• State-based control scheme: Since we can only detect limited facial states, we designated one of the discrete inputs to swap between 2 states and overload the mappings to allow for more DOF on the robot arm. The trade-off is that it will increase the cognitive load on the user's side.
• Discrete binary signals for blink and mouth movements since they are harder to control precisely. The trade-off is that we can't directly use them to control the robotic arm movements, so we lose potential degrees of freedom, but we can use them for other necessary tasks/features.

(d) Impact on Real Engineering Criteria

These choices make the system viable for real-world assistive use. However, because we compromise on speed and DOF for safety, precision, and other features, further sensing improvements are needed to improve efficiency and robustness.

3. Implementation

(a) Hardware

• UR7e Robotic Arm
• Logitech C922 Webcam
• Standard workstation running ROS2

(b) Parts Used

No additional manufacturing was conducted, strictly a Logitech Webcam and a UR7e Robotic Arm.

(c) Software Architecture

Launch System

• Declares runtime arguments (camera selection)
• Starts the MoveIt motion planning stack for the UR7
• Initializes the custom teleoperation and control node

MoveIt is launched before the teleoperation node to prevent deadlock and ensure motion planning requests don't fail because of an uninitialized MoveIt.

MoveIt Configuration

Node Descriptions

planning/ik

Planning is used for returning to a saved waypoint, a goal-based command for which we need to compute the IK and construct a trajectory.

IK Computation:

• Constructs a PoseStamped target in the base_link frame
• Anchors the solution using the current joint state
• Requests a collision-aware IK solution via /compute_ik
• Validates the result against joint limits

Motion Planning:

• Converts the IK result into joint constraints
• Requests a trajectory from /plan_kinematic_path
• Uses RRTConnect for planning
• Returns a time-parameterized joint trajectory

IK Planner Visualization

System Flow Diagram

System architecture showing the complete pipeline from webcam input to robot control

(d) Complete System Workflow

1. Webcam captures user's face
2. MediaPipe extracts facial landmarks
3. Head pose, blinks, and mouth state are computed and filtered
4. Gestures are mapped according to the active control state
5. Commands are scaled and adjusted, computing IK if necessary
6. Joint velocities are sent to the UR7e
7. Safety commands are prioritized
8. Robot executes motion or returns to savepoint/tuck position

4. Results

(a) Performance & Tasks

The system successfully:

• Controlled multiple UR7e joints using head gestures
• Switched control modes via blink detection
• Safely stopped and reset motion
• Toggled the gripper
• Returned to a saved pose or tuck position via mouth commands
• Helped the user perform simple common tasks like picking up objects and putting them in designated buckets

(b) Video Demonstrations

Final Presentation video for EECS 106a

Final Presentation video for EECS 106a

📊 View Presentation Slides

Planning Demonstrations

Showcasing planning with side-by-side video of head commands via FaceMesh and with robotic arm execution

5. Conclusion

(a) Design Criteria Assessment

Our final solution met the final criterion pretty well:

• Planning: We implemented a change post-presentation to our Planning requirement. By toggling your mouth opening, you are able to save a pose for the UR7e robotic arm, and when you toggle again, MoveIt is deployed to take in the current position of the robot and the saved waypoint, and undergoes an IK planner node to restore the arm to that saved pose.
• Sensing: Real-time tracking of 468 different points on a face mesh from a 2D frame converted into 3D commands. The MediaPipe FaceMesh program also has to encounter recentering for dynamic environments or camera setups via blink commands.
• Actuation: The actuation node sends ROS 2 messages in the Joint-Velocities space for the robotic arm to execute commands in a safe, smooth manner.
• Hardware: We utilized the UR7e robotic arm and a Logitech C922 Webcam.

(b) Post-Presentation Changes

Following the initial presentation, we implemented a significant enhancement to our planning requirement using MoveIt. We developed an IK planner that enables users to toggle a save mode, allowing them to save a checkpoint pose of the robotic arm. When the save mode is toggled again, the system uses MoveIt to compute the inverse kinematics solution and plan a trajectory to return the arm to the previously saved pose. This feature provides users with the ability to quickly return to important positions during manipulation tasks, reducing redundant movements and improving overall efficiency.

(c) Difficulties Encountered

Ideating a planning requirement due to the ambiguity of what was acceptable before the presentation did cause some confusion on our part for developing our planning node. In addition, we noticed slight errors in our extreme turns and nods for movement commands for the FaceMesh, as when you move to a point where your eyes are not tracked, it can cause an unintentional reset state.

(d) Flaws, Hacks, and Future Improvements

As mentioned in the previous question, we noticed shortcomings in the MediaPipe FaceMesh for extreme commands. What we would like to improve:

• Fine-tune the gains so that turns can correlate to fast velocities without causing excessively unsafe motor controls if outside our set safety bounds
• Look into finetuning the Eye Aspect Ratio (EAR) metric for maybe hindering such edge cases
• Look into different open-source CV models that can offer a more accurate 3D reconstruction and will not be as sensitive
• Include a headset with the webcam integrated (or incorporating a VR headset) for a more modular setup for real-life applications
• Fine-tune the facial regions of FaceMesh to account for more DOF without switching states (eyebrow movements, jaw twitching, etc.)

6. Team

(a) Team Members

(b) Major Contributions

7. Additional Materials

• Code, URDFs, and launch files: Available in the repository
• CAD models: No additional hardware was manufactured
• Data sheets: UR7e and Logitech C922 specifications available from manufacturers
• Videos and images: Demonstration videos and screenshots included in Results section
• GitHub Repository: View on GitHub

Workspace Cleanup

After completing our project, we returned all equipment to UC Berkeley's ESG and cleaned up our workstation. We did not use any additional parts or make any changes to the workstation beyond our project requirements. Below is proof of our workspace cleanup:

Proof of workspace cleanup after returning equipment to UC Berkeley's ESG. No additional parts were used and no changes were made to the workstation.

Peer Evaluations

Proof of completing peer evaluations for the project:

Proof of completing peer evaluations for the EECS 106A final project