Apply texture from OpenCV image

Hi,

I am working on a demo of augmented realty using the ball-in-maze example. I grab images from the camera and I want to apply them in the background. This is what I am doing:

 
  // Copy the image and flip it
  cv::Mat m_cvIflip;
  pthread_mutex_lock(&m_mutex_img);
  m_cvIflip = m_cvI.clone();
  pthread_mutex_unlock(&m_mutex_img);
  cv::flip(m_cvIflip,m_cvIflip,0);

 // Create the texture
  cv::Size size = m_cvIflip.size();
  PT(Texture) tex = new Texture();
  tex->set_compression(Texture::CM_off);
  // tex->compress_ram_image(Texture::CM_off);
  tex->set_quality_level(Texture::QL_best);
  tex->setup_2d_texture(size.width, size.height, Texture::T_unsigned_byte, Texture::F_rgb8);

  PTA_uchar pt;
  int total = size.width * size.height * m_cvIflip.channels();
  pt.resize(total);

  std::vector<uchar> data(m_cvIflip.ptr(), m_cvIflip.ptr() + total);
  std::string s(data.begin(), data.end());

  pt.set_data(s);

  tex->set_ram_image(pt, Texture::CM_off);
  m_picPlane.set_texture(tex);// m_picPlane is a simple plane with proportional dimensions to the size of the picture.

This way seems not very efficient, without applying the texture the demo run at 1k fps, otherwise at 180 fps. At this lower frame rate (~180) the collision of the ball with the maze is not working very well, sometimes the ball goes through the walls. How can I improve the fps?

The display is actually updated at the same framerate? because of course I don’t need to update the image at 1k Hz (I grab the images at 15fps in fact I could keep a counter of the images and just apply the texture is the image is new).

Should I scale before the images in some way using OpenCV?

By default, Panda3D will automatically rescale any texture image down to the nearest smaller power of two when you read it from disk, so you usually don't have to think about this--but your application will load faster if you scale your textures properly in the first place.

Thanks!

(I am just applying the same orientation w.r.t the camera of the picture on the left to the virtual maze, I have still to figure out the right way to do it but this is another problem)

Sorry for the second post, I wanted just to add that applying the texture just when a new image is arrive improved the fps to ~800. So the question is: is there a better way to add a background from a streaming of images?

I’m not sure I entirely understand the question (there seems to be several unrelated things asked here), but I will give you some performance hints:

1. On some systems (eg. NVIDIA and perhaps also Intel GPUs), it is faster to upload F_rgba8 textures, even if the fourth channel is not used.

2. Your method of assigning the data is very inefficient, making several unnecessary copies. You could try this instead:

{
  PTA_uchar pt = tex->modify_ram_image();
  memcpy(pt.p(), m_cvIflip.ptr(), total);
}

3. Consider modifying the existing texture data using the above code when a new image comes in, rather than creating a new texture for every frame.

4. In any case, add this to Config.prc:

textures-power-2 none

This will prevent Panda from rescaling the texture.

As for improving the collisions, 180 fps should be more than fast enough. I’m not sure why your ball would be going through the walls of the maze unless it was going very fast or if your collision sphere was too small.

This is exactly what I was looking for, thank you very much. I used your suggestions and it is working fine.

About the collision I am not sure too. I have just noticed that at higher fps the collision are working properly.