Perspective-n-point problem

Recover camera pose from matched points#

Each correspondence pairs a point whose position is known in an object or world frame with the pixel where that point appears. Given the camera intrinsics, a PnP solver finds the rotation and translation that make the projected 3D points agree with their 2D observations. The result is a rigid-body transformation between the camera and reference frame.

The EPnP paper states the calibrated-camera problem explicitly and presents a non-iterative method for (n) 3D-to-2D correspondences. Its method handles planar and non-planar configurations for (n \geq 4), with an optional nonlinear refinement after the closed-form estimate.

A common pose-estimation subproblem#

Robots use PnP after detecting fiducial corners, object keypoints, or mapped landmarks. The estimated camera pose can localise a mobile robot, align a manipulator with an object, or initialise tracking. If the 3D points are expressed in a robot frame, the result can also contribute to hand-eye calibration.

PnP is not camera calibration. Calibration estimates lens and projection parameters, while ordinary PnP assumes those intrinsics are already known. PnP is also not a complete correspondence system: another stage must decide which image feature matches which 3D point.

Geometry and correspondence quality control the answer#

Noisy pixel locations, inaccurate 3D landmarks, and biased intrinsics all affect pose. Nearly collinear points and weak depth variation make some pose directions poorly constrained. Minimal configurations can also admit multiple mathematical solutions.

Real scenes add outliers from missed or incorrect matches. An outlier-resistant estimator may sample subsets and then refine on inliers, but its result still depends on the inlier geometry and threshold. Reprojection error alone does not guarantee that the recovered metric pose is accurate enough for contact or grasping.