Lens/camera for OpenCV style camera parameterisation

Hi,

I’m aiming to get Panda3D to duplicated my OpenCV/computer vision style camera parameterisation (the result of calibrating a real-world camera) so that Panda can be used for rendering things onto images captured by the camera. Basically what is done with augmented reality, but more for the purposes of debugging and visualising the results of various computer vision algorithms.

In computer vision, we commonly represent the projection of a point from a world coordinate system to the image by a transformation such as:

pi = P.L.pw

where:
pw: point in world space
L : transformation that takes point from world space to camera space
P : 3x4 projection matrix [K|0], where K is a 3x3 “camera calibration” matrix and the right hand column of P is all 0s.

K normally looks like this:

f   s    cx
0  a.f  cy
0   0    1

and has f (focal length), s (skew, often 0), (cx,cy) principle point (point in the image where the optical axis intersects the image plane), and a (aspect ratio, often assumed to be 1.0 )

Generally, the image coordinate system is top-left origin, x positive to the right, and y positive down. The camera coordinate system is the same as the image, with z positive in the direction the camera looks (“into the screen”), making a right handed coordinate system. The world coordinate system is again right handed, and rarely standardised but often the z=0 plane would be the ground.

Allowing for homogeneous division, P projects directly from 3D to the 2D image.

Given that I can get suitable calibration for my cameras using OpenCV (calibration provides P and L), I want to now be able to make a camera in Panda that will duplicate my real-world cameras.

I can do this in OpenGL by specifying the projection matrix as:

2*K(0,0)/w,  2*K(0,1)/w,  ( w - 2*K(0,2))/w       ,                          0,
         0,  2*K(1,1)/h,    ( h - 2*K(1,2))/h     ,                          0,
         0,           0, (-far - near)/(far-near) ,   -2*far*near/(far - near),
         0,           0,                        -1,                         0 ;

(where w and h are the image width and height, far and near are opengl clipping planes, K is the K matrix I described above)

And ensuring that there is a 180 rotation around the x-axis before projection (openGL having a camera coordinate system that is right-handed, y-up, z-negative i.e. 180 rotation from my calibrated camera system).

Can anyone advise me what I need to do to enable me to do this with Panda? (I’m relatively new to Panda, having used it a few years ago. I’ve been just using OpenGL directly for a while, but would now like to avoid being so low level for new visualisations and tools).

I suspect that what I need to know is what the projection matrix should be trying to achieve. Does it project directly to the image, or, like OpenGL, does it project to some “normalised device coordinate” system first?

The projection matrix Panda generates transforms to a normalised space, indeed, not image coordinates.

However, Panda should be able to calculate the correct projection matrix for you if you pass in the appropriate parameters to your PerspectiveLens. Judging by how you calculated it in OpenGL, this should achieve the same result:

lens.setFocalLength(f)
lens.setFilmSize(w, h)
lens.setFilmOffset(w*0.5 - cx, h*0.5 - cy)
lens.setNearFar(near, far)
lens.setAspectRatio(a) # Optional, otherwise computed from film size
print(lens.getProjectionMat())

Panda defaults to Z-up right-handed coordinate system for the world. You may specify otherwise by calling setCoordinateSystem on the lens used to render that world or by setting the “coordinate-system” configuration variable. However, this is usually not necessary.

You can use setViewVector to apply a rotation to the lens. Normally, the lens points in the Y direction, with +Z being up, but you can use this method to change that.

Finally, if all else fails, there is also the opportunity for you to specify the matrix yourself. You can do this by creating a MatrixLens instance and assigning that to the camera. You can specify the matrix on that using setUserMat().

Thanks, set_view_vector was pretty much the last thing that I needed.

After a bit of digging and finally finding the right details, I had it mostly worked out with MatrixLens with a few gotchas along the way.

First off, MatrixLens is a good approach, and in default state I can actually use the projection matrix that I pass to OpenGL (film size is set suitable for the normalised coordinates to be the same as OpenGL’s normalised device coordinates).

First gotcha was that Panda seems to use row vectors and pre-multiply - pretty much the first place I’ve seen something not use column vectors and post-multiply (v.M vs. M.v), so that threw me for a while, and ultimately meant I needed the transpose of the projection matrix I mention in my first post.

Second gotcha was that my near and far clipping planes were too far apart, meaning I got bad depth buffer resolution (which made me go in circles trying to recalculate the right factors in the matrix until I realised what the actual problem was). Still have an open question over how to set/find out the bit-depth of the depth-buffer Panda uses.

The last bit of the puzzle after that was the set_view_vector as an easy way to allow me to look along +z without worrying about throwing in extra rotations etc beneath the camera node.

So looking good now that Panda will enable me to escape writing low-level OpenGL for my new debugging renderer…

print(base.win.get_fb_properties().get_depth_bits())

If you need more depth bits, you can put “depth-bits 32” in your Config.prc file (or specify it at window creation time via a FrameBufferProperties object).