Haptikos hands plugin#486
Conversation
Signed-off-by: Giorgos Babanis <giorgosbaba@hotmail.com>
Signed-off-by: Giorgos Babanis <giorgosbaba@hotmail.com>
…eleop into haptikos-plugin
Signed-off-by: Giorgos Babanis <giorgosbaba@hotmail.com>
Signed-off-by: Giorgos Babanis <giorgosbaba@hotmail.com>
📝 WalkthroughWalkthroughThis PR introduces a complete Haptikos Hands Plugin for Isaac Teleop that integrates hand tracking data from Haptikos exoskeletons into OpenXR. The implementation includes a HaptikosHandsPlugin class that runs a 90Hz worker thread reading controller and hand pose snapshots from Haptikos, transforming them via coordinate remapping, and injecting synthesized hand joint locations into OpenXR. Build configuration conditionally enables the plugin when HaptikosAPI is available, installs required shared libraries, and includes comprehensive documentation for setup and usage. Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/plugins/haptikos/haptikos_hands_plugin.cpp`:
- Around line 82-95: The catch blocks in HaptikosHandsPlugin's update thread
currently call std::exit(1), which force-terminates the process; instead set
m_running = false, ensure m_left_injector.reset() and m_right_injector.reset()
are called, and return from the thread so the worker loop exits normally and
destructor/RAII cleanup runs; update the catch(...) and catch(const
std::exception& e) handlers in the update function to log the error, set
m_running = false, reset injectors, and return rather than calling std::exit.
- Around line 103-121: The m_client.GetData(...) calls inside worker_thread
(e.g., the call used to populate right_data and the symmetric left call) can
throw and are not currently caught; wrap each GetData invocation (or the loop
body around calculate_hand_pose and injector logic in worker_thread) in a
try-catch that catches std::exception (and a catch-all) and logs the error, then
continues the loop so the thread doesn't terminate; ensure you reference
m_client.GetData, right_data/left_data and the injector logic
(m_right_injector/m_left_injector) when adding the try-catch and avoid letting
exceptions escape worker_thread.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1a19f2f3-a796-4656-a1bb-d3eab88203fa
📒 Files selected for processing (8)
CMakeLists.txtsrc/plugins/haptikos/.gitignoresrc/plugins/haptikos/CMakeLists.txtsrc/plugins/haptikos/README.mdsrc/plugins/haptikos/haptikos_hands_plugin.cppsrc/plugins/haptikos/inc/haptikos/haptikos_hands_plugin.hppsrc/plugins/haptikos/main.cppsrc/plugins/haptikos/plugin.yaml
| catch (const std::exception& e) | ||
| { | ||
| std::cerr << "HaptikosHandsPlugin update error: " << e.what() << std::endl; | ||
| m_left_injector.reset(); | ||
| m_right_injector.reset(); | ||
| std::exit(1); | ||
| } | ||
| catch (...) | ||
| { | ||
| std::cerr << "HaptikosHandsPlugin update error: unknown exception" << std::endl; | ||
| m_left_injector.reset(); | ||
| m_right_injector.reset(); | ||
| std::exit(1); | ||
| } |
There was a problem hiding this comment.
std::exit(1) from worker thread bypasses all cleanup.
Calling std::exit(1) here terminates the process immediately without running the HaptikosHandsPlugin destructor or main's exception handlers. The reset() calls before exit don't help—std::exit() won't return to let them complete properly.
Set m_running = false and return from the thread instead. Let the main loop detect the shutdown and handle cleanup via RAII.
Proposed fix
catch (const std::exception& e)
{
std::cerr << "HaptikosHandsPlugin update error: " << e.what() << std::endl;
m_left_injector.reset();
m_right_injector.reset();
- std::exit(1);
+ m_running = false;
+ return;
}
catch (...)
{
std::cerr << "HaptikosHandsPlugin update error: unknown exception" << std::endl;
m_left_injector.reset();
m_right_injector.reset();
- std::exit(1);
+ m_running = false;
+ return;
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/plugins/haptikos/haptikos_hands_plugin.cpp` around lines 82 - 95, The
catch blocks in HaptikosHandsPlugin's update thread currently call std::exit(1),
which force-terminates the process; instead set m_running = false, ensure
m_left_injector.reset() and m_right_injector.reset() are called, and return from
the thread so the worker loop exits normally and destructor/RAII cleanup runs;
update the catch(...) and catch(const std::exception& e) handlers in the update
function to log the error, set m_running = false, reset injectors, and return
rather than calling std::exit.
| if (right_tracked.data) | ||
| { | ||
| Haptikos::HandData right_data = m_client.GetData(true, Haptikos::GlobalToWrist, true, true, false); | ||
| bool valid_wrist = false; | ||
| XrPosef rigth_controller = oxr_utils::get_aim_pose(*right_tracked.data, valid_wrist); | ||
|
|
||
| if (right_data.IsValid() == 1 && valid_wrist) | ||
| { | ||
| calculate_hand_pose(right_joints, right_data, rigth_controller); | ||
| if (!m_right_injector) | ||
| { | ||
| const auto handles = m_session->get_handles(); | ||
| m_right_injector = std::make_unique<plugin_utils::HandInjector>( | ||
| handles.instance, handles.session, XR_HAND_RIGHT_EXT, handles.space); | ||
| } | ||
|
|
||
| m_right_injector->push(right_joints, time); | ||
| rigth_published = true; | ||
| } |
There was a problem hiding this comment.
m_client.GetData() can throw but is not caught.
The GetData() calls at lines 105 and 134 are outside the try-catch block. If the Haptikos API throws an exception here, it will propagate out of worker_thread(), causing std::terminate() since threads don't propagate exceptions to the caller.
Either wrap the entire loop body in the try-catch, or specifically handle potential exceptions from the Haptikos client.
Proposed fix: extend try-catch scope
while (m_running)
{
auto frame_start = std::chrono::steady_clock::now();
+ try
+ {
core::ControllerSnapshotTrackedT left_tracked;
core::ControllerSnapshotTrackedT right_tracked;
core::HandPoseTrackedT left_hand;
core::HandPoseTrackedT rigth_hand;
- try
- {
// Update DeviceIOSession (handles time and tracker updates)
m_deviceio_session->update();
// Read tracker data in the same exception boundary as update.
left_tracked = m_controller_tracker->get_left_controller(*m_deviceio_session);
right_tracked = m_controller_tracker->get_right_controller(*m_deviceio_session);
- }
- catch (const std::exception& e)
- // ... rest of try-catch stays the same but now covers GetData() calls too
+
+ // ... rest of loop body including GetData() calls ...
+
+ std::this_thread::sleep_until(frame_start + target_frame_duration);
+ }
+ catch (const std::exception& e)
+ {
+ std::cerr << "HaptikosHandsPlugin error: " << e.what() << std::endl;
+ m_left_injector.reset();
+ m_right_injector.reset();
+ m_running = false;
+ return;
+ }
+ }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/plugins/haptikos/haptikos_hands_plugin.cpp` around lines 103 - 121, The
m_client.GetData(...) calls inside worker_thread (e.g., the call used to
populate right_data and the symmetric left call) can throw and are not currently
caught; wrap each GetData invocation (or the loop body around
calculate_hand_pose and injector logic in worker_thread) in a try-catch that
catches std::exception (and a catch-all) and logs the error, then continues the
loop so the thread doesn't terminate; ensure you reference m_client.GetData,
right_data/left_data and the injector logic (m_right_injector/m_left_injector)
when adding the try-catch and avoid letting exceptions escape worker_thread.
1 participant
A plugin that utilizes the Haptikos Exoskeletons through our API to inject hand data into an OpenXR session replacing the headset's optical hand tracking. It is modeled after controller synthetic hands plugin.
It was tested and developed on an Ubuntu 24.04 computer with an Ada RTX 6000 48 GB, using the Meta Quest 3 Headset. It does require a controller to calculate the position of the wrist in 3D space, so it should work with any Headset that has a controller and is compatible with the Isaac Teleop framework. It has a dependency on our API, but if it is not provided the plugin is just skipped.
To validate that the plugin provides the correct data we teleoperated the Fourier GR1T2 robot inside IsaacLab using the pick and place task.
Summary by CodeRabbit
New Features
Documentation
Chores