Skip to content

ci: expand wheel regression to all 9 type combinations - #86

Closed
woosik2510 wants to merge 10 commits into
masterfrom
ci/expand-wheel-eval
Closed

ci: expand wheel regression to all 9 type combinations#86
woosik2510 wants to merge 10 commits into
masterfrom
ci/expand-wheel-eval

Conversation

@woosik2510

@woosik2510 woosik2510 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Replace stereo_wheel with 9 combos (2D/3D x Ang/Lin/Cen x planar/non-planar for 3D). Vicon companion sensor. WHEEL_TYPE injected via docker -e + sed before sim loop. Distinct noise values (noise_w=0.05, noise_v=0.03, noise_p=0.001) expose Q matrix fix in NEES. Chains off PR #85.

@github-actions

github-actions Bot commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Unit Test Results

Test Result Duration
test_UpdaterVicon ✅ Passed 0.07s
test_UpdaterWheel2D ✅ Passed 0.07s
test_UpdaterWheel3D ✅ Passed 0.07s
test_UpdaterWheelDt ✅ Passed 0.07s
test_UpdaterWheelInt ✅ Passed 0.07s

5/5 passed

@woosik2510 woosik2510 closed this Jul 31, 2026
@github-actions

github-actions Bot commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Simulation regression report - master vs PR

8 seeds/config, sim UD_Warehouse, informational only (does not block merge).
Delta is PR relative to master (mean). Lower is better for RMSE / Time; NEES closer to ~1-3 is well-calibrated.

full
Metric master PR Delta (mean)
RMSE ori (deg) 0.164 +/- 0.038 0.165 +/- 0.034 +0.6%
RMSE pos (m) 0.053 +/- 0.009 0.055 +/- 0.009 +3.8%
NEES ori 3.400 +/- 1.300 3.700 +/- 1.100 +8.8%
NEES pos 2.700 +/- 0.700 2.900 +/- 0.700 +7.4%
Time (s) 25.100 +/- 0.500 25.000 +/- 0.300 -0.4%
gps
Metric master PR Delta (mean)
RMSE ori (deg) 8.650 +/- 21.885 8.650 +/- 21.885 +0.0%
RMSE pos (m) 16.959 +/- 46.541 16.959 +/- 46.541 +0.0%
NEES ori 533.300 +/- 1486.500 533.300 +/- 1486.500 +0.0%
NEES pos 1244076.600 +/- 3517754.000 1244076.600 +/- 3517754.000 +0.0%
Time (s) 0.300 +/- 0.000 0.300 +/- 0.000 +0.0%
vicon
Metric master PR Delta (mean)
RMSE ori (deg) 0.148 +/- 0.005 0.148 +/- 0.005 +0.0%
RMSE pos (m) 0.008 +/- 0.000 0.008 +/- 0.000 +0.0%
NEES ori 4.200 +/- 0.200 4.200 +/- 0.200 +0.0%
NEES pos 4.200 +/- 0.200 4.200 +/- 0.200 +0.0%
Time (s) 0.700 +/- 0.000 0.700 +/- 0.000 +0.0%
vicon_lidar
Metric master PR Delta (mean)
RMSE ori (deg) 0.132 +/- 0.005 0.132 +/- 0.005 +0.0%
RMSE pos (m) 0.008 +/- 0.000 0.008 +/- 0.000 +0.0%
NEES ori 3.900 +/- 0.300 3.900 +/- 0.300 +0.0%
NEES pos 4.100 +/- 0.200 4.100 +/- 0.200 +0.0%
Time (s) 10.500 +/- 0.100 10.500 +/- 0.100 +0.0%
vio_stereo
Metric master PR Delta (mean)
RMSE ori (deg) 0.174 +/- 0.036 0.174 +/- 0.036 +0.0%
RMSE pos (m) 0.111 +/- 0.032 0.111 +/- 0.032 +0.0%
NEES ori 2.000 +/- 0.900 2.000 +/- 0.900 +0.0%
NEES pos 12.600 +/- 16.500 12.600 +/- 16.500 +0.0%
Time (s) 19.400 +/- 0.300 19.400 +/- 0.300 +0.0%
wheel_2d_ang
Metric master PR Delta (mean)
RMSE ori (deg) 0.127 +/- 0.005 0.129 +/- 0.005 +1.6%
RMSE pos (m) 0.006 +/- 0.000 0.007 +/- 0.000 +16.7%
NEES ori 3.800 +/- 0.300 3.900 +/- 0.200 +2.6%
NEES pos 3.800 +/- 0.200 4.100 +/- 0.300 +7.9%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_2d_cen
Metric master PR Delta (mean)
RMSE ori (deg) 0.129 +/- 0.004 0.134 +/- 0.005 +3.9%
RMSE pos (m) 0.007 +/- 0.000 0.007 +/- 0.000 +0.0%
NEES ori 3.900 +/- 0.200 4.500 +/- 0.500 +15.4%
NEES pos 4.000 +/- 0.200 4.500 +/- 0.300 +12.5%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_2d_lin
Metric master PR Delta (mean)
RMSE ori (deg) 0.129 +/- 0.005 0.133 +/- 0.005 +3.1%
RMSE pos (m) 0.007 +/- 0.000 0.007 +/- 0.000 +0.0%
NEES ori 3.900 +/- 0.300 4.300 +/- 0.400 +10.3%
NEES pos 4.000 +/- 0.300 4.300 +/- 0.300 +7.5%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_3d_ang
Metric master PR Delta (mean)
RMSE ori (deg) 74.833 +/- 0.011 74.832 +/- 0.007 -0.0%
RMSE pos (m) 14.368 +/- 0.000 14.368 +/- 0.001 +0.0%
NEES ori 1491339.600 +/- 7677.700 2082290.500 +/- 20630.100 +39.6%
NEES pos 16069389.900 +/- 125530.900 15516828.000 +/- 133009.400 -3.4%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_3d_ang_planar
Metric master PR Delta (mean)
RMSE ori (deg) 0.132 +/- 0.005 0.113 +/- 0.004 -14.4%
RMSE pos (m) 0.007 +/- 0.000 0.006 +/- 0.000 -14.3%
NEES ori 3.700 +/- 0.300 3.700 +/- 0.200 +0.0%
NEES pos 3.700 +/- 0.300 3.700 +/- 0.200 +0.0%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_3d_cen
Metric master PR Delta (mean)
RMSE ori (deg) 74.833 +/- 0.011 74.832 +/- 0.008 -0.0%
RMSE pos (m) 14.368 +/- 0.000 14.368 +/- 0.000 +0.0%
NEES ori 1491281.300 +/- 6250.100 2038981.600 +/- 30269.100 +36.7%
NEES pos 16074561.700 +/- 137844.900 15475966.300 +/- 89842.100 -3.7%
Time (s) 0.800 +/- 0.000 0.700 +/- 0.000 -12.5%
wheel_3d_cen_planar
Metric master PR Delta (mean)
RMSE ori (deg) 0.132 +/- 0.005 0.115 +/- 0.004 -12.9%
RMSE pos (m) 0.007 +/- 0.000 0.006 +/- 0.000 -14.3%
NEES ori 3.700 +/- 0.300 3.900 +/- 0.200 +5.4%
NEES pos 3.800 +/- 0.300 3.900 +/- 0.200 +2.6%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_3d_lin
Metric master PR Delta (mean)
RMSE ori (deg) 74.833 +/- 0.011 74.833 +/- 0.007 +0.0%
RMSE pos (m) 14.368 +/- 0.000 14.368 +/- 0.000 +0.0%
NEES ori 1497606.100 +/- 5101.600 2067953.400 +/- 19626.000 +38.1%
NEES pos 19459484.500 +/- 142877.700 15600094.700 +/- 104251.700 -19.8%
Time (s) 0.800 +/- 0.000 0.800 +/- 0.000 +0.0%
wheel_3d_lin_planar
Metric master PR Delta (mean)
RMSE ori (deg) 0.132 +/- 0.005 0.114 +/- 0.004 -13.6%
RMSE pos (m) 0.007 +/- 0.000 0.006 +/- 0.000 -14.3%
NEES ori 3.700 +/- 0.300 3.800 +/- 0.200 +2.7%
NEES pos 3.900 +/- 0.300 3.800 +/- 0.200 -2.6%
Time (s) 0.700 +/- 0.000 0.700 +/- 0.000 +0.0%

@github-actions

github-actions Bot commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Calibration Convergence Report

10 seeds · sim UD_Warehouse · informational only (does not block merge).

Scenario Parameters ±3σ
Camera Time Offset 1/1 params ✅
Camera Extrinsic (IMU↔Cam) 6/6 params ✅
Camera Intrinsics 8/8 params ✅
GPS Time Offset 1/1 params ✅
GPS Extrinsic (IMU↔GPS) 3/3 params ✅
LiDAR Time Offset 1/1 params ✅
LiDAR Extrinsic (IMU↔LiDAR) 6/6 params ✅
Vicon Time Offset 1/1 params ✅
Vicon Extrinsic (IMU↔Vicon) 6/6 params ✅
Wheel Time Offset 0/1 params ❌
Wheel Extrinsic (IMU↔Wheel) 5/6 params ❌
Wheel Intrinsics (Wheel2DAng) 3/3 params ✅
Wheel Intrinsics (Wheel3DAng) 3/3 params ✅
Per-parameter details

✅ Camera Time Offset

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| time offset (s) | 0.000179 | 0.00015 | ✅ |

✅ Camera Extrinsic (IMU↔Cam)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| φ_x (rad) | 0.00015 | 0.000271 | ✅ |
| φ_y (rad) | 6.79e-05 | 0.000287 | ✅ |
| φ_z (rad) | 0.000162 | 0.000268 | ✅ |
| p_x (m) | 0.00314 | 0.00382 | ✅ |
| p_y (m) | 0.00489 | 0.0049 | ✅ |
| p_z (m) | 0.0039 | 0.00348 | ✅ |

✅ Camera Intrinsics

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| fu (px) | 0.109 | 0.0788 | ✅ |
| fv (px) | 0.115 | 0.0998 | ✅ |
| cu (px) | 0.085 | 0.0998 | ✅ |
| cv (px) | 0.0855 | 0.102 | ✅ |
| k₁ | 0.000165 | 0.000196 | ✅ |
| k₂ | 0.000104 | 0.000131 | ✅ |
| p₁ | 2.43e-05 | 3.69e-05 | ✅ |
| p₂ | 3.16e-05 | 2.35e-05 | ✅ |

✅ GPS Time Offset

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| time offset (s) | 0.0031 | 0.00235 | ✅ |

✅ GPS Extrinsic (IMU↔GPS)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| p_x (m) | 0.0132 | 0.0229 | ✅ |
| p_y (m) | 0.0216 | 0.0233 | ✅ |
| p_z (m) | 0.0238 | 0.0237 | ✅ |

✅ LiDAR Time Offset

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| time offset (s) | 0.00083 | 0.000187 | ✅ |

✅ LiDAR Extrinsic (IMU↔LiDAR)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| φ_x (rad) | 0.000539 | 0.000328 | ✅ |
| φ_y (rad) | 0.000284 | 0.000315 | ✅ |
| φ_z (rad) | 0.000256 | 0.000374 | ✅ |
| p_x (m) | 0.00329 | 0.00556 | ✅ |
| p_y (m) | 0.00567 | 0.00585 | ✅ |
| p_z (m) | 0.00663 | 0.00836 | ✅ |

✅ Vicon Time Offset

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| time offset (s) | 0.000105 | 0.000148 | ✅ |

✅ Vicon Extrinsic (IMU↔Vicon)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| φ_x (rad) | 0.000155 | 0.000546 | ✅ |
| φ_y (rad) | 0.000295 | 0.000535 | ✅ |
| φ_z (rad) | 0.000384 | 0.000576 | ✅ |
| p_x (m) | 0.00313 | 0.00908 | ✅ |
| p_y (m) | 0.00442 | 0.00919 | ✅ |
| p_z (m) | 0.00327 | 0.00981 | ✅ |

❌ Wheel Time Offset

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| time offset (s) | 0.00551 | 0.000178 | ❌ |

❌ Wheel Extrinsic (IMU↔Wheel)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| φ_x (rad) | 0.0124 | 0.000641 | ❌ |
| φ_y (rad) | 0.000318 | 0.000212 | ✅ |
| φ_z (rad) | 0.00015 | 0.00025 | ✅ |
| p_x (m) | 0.00101 | 0.00135 | ✅ |
| p_y (m) | 0.00452 | 0.00875 | ✅ |
| p_z (m) | 0.0305 | 0.0228 | ✅ |

❌ Wheel Intrinsics (Wheel2DAng)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| left radius (m) | 7.65e-05 | 0.000118 | ✅ |
| right radius (m) | 8.9e-05 | 0.000118 | ✅ |
| baseline (m) | 0.0117 | 0.00413 | ✅ |

✅ Wheel Intrinsics (Wheel3DAng)

| Parameter | final |error| | final σ | ±3σ |
|---|---|---|---|
| left radius (m) | 5.34e-05 | 0.000332 | ✅ |
| right radius (m) | 7.07e-05 | 0.000333 | ✅ |
| baseline (m) | 0.00751 | 0.009 | ✅ |

cam_dt

cam_ext

cam_int

gps_dt

gps_ext

lidar_dt

lidar_ext

vicon_dt

vicon_ext

wheel_dt

wheel_ext

wheel_int_2d_ang

wheel_int_3d_ang

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WoosikLee2510 WoosikLee2510 Awaiting requested review from WoosikLee2510 WoosikLee2510 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@woosik2510