Bundle Adjustment

The goal is to optimise camera calibration and 3D world point positions from 2D observations. We refine the noisy cameras and noisy world points from data/noisy-calibrations-and-world-points.json so that the projected 3D points match the supplied image observations.

Submitted Output

answer.json is generated by the manual C++ full bundle adjustment implementation in apps/bundle_adjustment_optimisation.cpp.

It uses:

data/noisy-calibrations-and-world-points.json

The output has the same schema as the input file and contains optimised camera extrinsics, optimised focal lengths, and optimised 3D world points.

Approach

• OpenCV-style pinhole camera model with no distortion coeff.
• Fixed principal point (cx, cy) and image size.
• Aspect ratio fixed to 1, so each camera has one optimised focal length f.
• Camera parameters are represented as Rodrigues rotation vector, translation, and focal length: 7 parameters per camera.
• The full bundle adjustment parameter vector has size 7N + 3M.
• Residuals are stacked reprojection errors over all camera observations.
• The main optimiser is a hand-written Levenberg-Marquardt implementation.
• Jacobians are numerical finite differences.
• The full BA path supports dense and sparse Jacobian construction. The default path is sparse.
• The sparse Jacobian uses the BA block structure: each observation only depends on one camera block and one point block.
• The LM step solves the damped normal equations:

(J^T J + lambda diag(J^T J)) delta = -J^T r

Implementations

• projection_check: validates the projection model using projection-check.json.
• parameterisation_check: validates Rodrigues conversion.
• calib_only_optimisation: solves the alternative calibration-only route using accurate world points.
• bundle_adjustment_optimisation: main submitted full bundle adjustment (optimising cam params and world points); writes answer.json.
• ba_with_triangulation: extension that ignores supplied world points and triangulates initial points using the noisy calibrations.
• ba_with_sfm: extension that ignores supplied calibrations and world points, then initialises from observations.
• ceres_bundle_adjustment: comparison implementation only; the submitted core optimiser is the manual LM implementation.

Build

The project was built with:

mkdir build
cd build
cmake ..
make

Run

From the build/ directory:

./projection_check
./parameterisation_check
./calib_only_optimisation
./bundle_adjustment_optimisation
./ba_with_triangulation
./ba_with_sfm

The main submitted answer is regenerated with:

./bundle_adjustment_optimisation

This writes:

../answer.json

Visualisation

A simple visualisation script can be run from the repository root with:

python3 visualisation/visualise_scene.py

This reads answer.json and writes:

visualisation/scene_topdown.png
visualisation/scene.ply

scene_topdown.png is a 2D X/Z plot of the optimised 3D points and camera centres.

scene.ply is a simple ASCII PLY scene containing the optimised 3D points and camera centres. World points are light grey, and camera centres are red crosses.

Camera centres are computed from the world-to-camera extrinsics using:

C = -R^T t

The file can be opened in any PLY viewer.

Results

Check	Initial	Final
Projection check (max error)	—	6.43e-13 px
Rodrigues round-trip error	—	1.07e-14
Calibration-only RMS	141.07 px	~1e-11 px
Manual full BA RMS	147.78 px	1.10e-11 px
Ceres BA RMS	147.78 px	2.44e-09 px
Triangulation-initialised BA RMS	147.09 px	1.1e-11 px
SfM-initialised BA RMS	27.97 px	5.8e-03 px

Libraries Used

Library	Version	Purpose
Eigen	5.0.1	Matrix/vector algebra and sparse linear algebra.
nlohmann/json	3.12.0	JSON parsing and writing.
Ceres Solver	2.2.0	Comparison only against the manual bundle adjustment implementation.
OpenCV (calib3d/core)	4.12.0	Used for the SfM initialisation extension only.

No hard version constraints are set in CMakeLists.txt.

References

• Levenberg-Marquardt implementation notes: https://people.duke.edu/~hpgavin/m-files/lm.pdf
• SciPy Cookbook bundle adjustment sparsity example: https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html
• OpenCV Rodrigues implementation reference: opencv/modules/calib3d/src/calibration_base.cpp
• OpenCV triangulation reference: https://github.com/opencv/opencv_contrib/blob/4.x/modules/sfm/src/triangulation.cpp
• Ceres non-linear least squares tutorial: http://ceres-solver.org/nnls_tutorial.html
• OpenCV calibrated two-view and PnP flow, referenced while implementing the SfM initialisation path: findEssentialMat, recoverPose, solvePnPRansac, and solvePnP in OpenCV calib3d.

AI Assistance

AI was used for brainstorming, development support, and refinement, specifically for choosing data structures, refactoring helper functions, and getting the sparse Jacobian indexing right.