Real-time cable keypoint detection, 3D lifting, and Kalman filtering at ~100 Hz — CPU-only
Dual Franka Panda arms holding a bright-red DLO cable in MuJoCo. Left: interactive 3D viewer. Right: 6-panel perception pipeline running at every timestep.
HSV segmentation isolates the cable from background; skeleton is extracted every frame
10 keypoints sampled along skeleton and lifted to 3D via depth buffer
Kalman filter maintains temporal consistency; spline interpolation handles occlusion
Phase 1 delivers a real-time DLO perception and tracking pipeline running entirely on CPU in MuJoCo simulation — the foundation for Phase 2 shape control and CBF safety.
Full perception loop (segment → skeleton → 3D lift → Kalman update) at over 10× the target rate on CPU.
Distance-to-Centreline metric confirms the estimated cable shape matches ground truth with high fidelity.
Cubic spline interpolation recovers tracking through 30% keypoint occlusion without loss of cable shape.
A four-stage classical CV pipeline — no deep learning required. Each stage is CPU-native and runs inside the MuJoCo physics loop.
OpenCV HSV thresholding isolates the bright-red DLO cable from the scene
Thinning algorithm extracts the cable centreline; 10 keypoints sampled uniformly along it
7×7 min-depth patch from MuJoCo depth buffer back-projects each 2D keypoint to world frame
6-state constant-velocity Kalman filter per keypoint; greedy NN association across frames
get_dlo_state(rgb, depth) → (10, 3)The pipeline is packaged as a single reusable function. Phase 2 shape control and CBF safety will consume this output directly — no changes to the perception code required.
All metrics measured CPU-only on a single core. Ground truth from mj_data.site_xpos.
| Tracker | RMSE (mm) | DTC (%) | Speed (Hz) | Occlusion RMSE |
|---|---|---|---|---|
| Kalman ✅ | 213 | 1.9% | 1 674 | 199 mm |
| Particle Filter | 209 | 2.5% | 306 | 194 mm |
| EMA | 210 | 2.5% | 5 004 | 203 mm |
Note: High RMSE (mm) is a camera resolution artefact — 2.5 m overhead camera gives ~7.5 mm/pixel. DTC (shape accuracy relative to cable length) is the meaningful metric.
CPU-native physics + off-screen RGB-D rendering via Python API. No GPU required.
Two 7-DoF arms with 20-segment DER ball-joint DLO cable bundled in models/
HSV segmentation, morphological ops, multi-panel visualization at runtime
Zhang-Suen skeletonization extracts cable centreline from binary mask
Kalman filter, particle filter, cubic spline occlusion interpolation
get_dlo_state() → (10,3) output feeds directly into Phase 2 shape control and CBF-QP
Real-time pipeline: HSV segmentation → skeletonization → 3D lifting → Kalman filter. ~100 Hz, CPU-only, 1.9% DTC. Live MuJoCo + OpenCV demo.
Jacobian-based incremental shape control consuming get_dlo_state().
Online Jacobian estimation from keypoint deltas.
DIT* or JIT adapted for 14-DoF + DLO configuration space with catenary-aware cost function.
CBF-QP safety filter: h_arm-arm ≥ 0, h_arm-env ≥ 0,
h_cable-env ≥ 0. Mode-switching for clip insertions.
Benchmark hierarchical CBF vs standard MPC vs vanilla RRT on fast motions and dynamic obstacles.
Data analysis, final manuscript, and defense presentation formatting.