Import System for C++ Modules

I’ve just checked in support for a shiny new ‘panda3d’ module that better organizes the C++ classes of Panda3D. This feature has been requested for a while now as many were annoyed with the long, unorganized and imports through pandac.PandaModules. Basically, it now allows you to import Panda3D classes similar to this:

from panda3d.egg import EggData
from panda3d.ode import OdeJoint
from panda3d.core import *

The current system

There are a number of Panda3D dynamic libraries that contain a Python module, for example libp3direct, libpandaexpress, libpanda, libpandaegg and libpandaode. You can directly import them from Python by importing them by their library name, and thus access the wrapped classes and functions that this library exposes to CPython.
However, not all of the functions for those classes are implemented in C++. There are a bunch of functions that are implemented in Python that provide extra functionality to the C++ classes. Those methods are just convenience methods to make your life easier, or exist to make an interface more Pythonic. Some of these methods have already been deprecated in favor of implementing it on the C++ side (or in the wrapper generator tool, interrogate.)
These extension functions are defined in pandac/libnameModules.py, where ‘libname’ is the name of the library in question. Once you import pandac/PandaModules.py, the Python code is imported from all of Panda3D’s libraries, and the extension functions are added to the classes.
The extension functions are not the only reason why the ‘pandac’ tree exists. I’ve just said that you can directly import a dynamic library from Python, but that is not entirely true – it is not possible anymore, as of Python 2.5, on Mac and Windows. (On Windows, you can still import it by renaming the dll into pyd, but that trick doesn’t work for Mac OSX dylibs.)
Therefore, we need to manually import the library by locating it first and then directly loading it via the load_dynamic function in Python’s imp module. The code to do this has been added to ‘pandac’, as that’s the place where the Panda3D libraries are usually imported.

The new system

In the new system, instead of importing the class in question from pandac.PandaModules, you will import the class from a submodule of the ‘panda3d’ module.
We did not group the classes by their source directory, as was proposed. Organizing the ‘panda3d’ module in such a way that the sub-packages represent source directories would make it needlessly complicated.
Instead, we chose to group classes by the C++ dynamic library. The name of the submodule is defined by the name of the library without the “libpanda” or “libp3” prefix. For instance, libpandaegg maps to panda3d.egg, and libp3direct maps to panda3d.direct.
There is one exception for the libpanda and libpandaexpress libraries, they are merged into one module, panda3d.core. We chose to do this because libpanda and libpandaexpress contain the Python wrappers for the core of Panda3D – you will need these libraries to do anything useful with Panda3D. Furthermore, the distinction between libpanda and libpandaexpress is arbitrary and not meaningful to the developer.
When more libraries are added later, such as libpandaphysx, and libpandaai, we could simply add those to the list.
As for the direct tree, we have left it in it’s place. It has been suggested that it be merged into the panda3d module, but we decided not to do it, as it would become needlessly complicated and confusing. Furthermore, I think there is a benefit in keeping Panda3D’s Python modules and C++ modules separated.
We do acknowledge the unfortunate naming of the direct tree, and we might change that for a future release, but that would be a major change.
The new import system is all bundled into one file, panda3d.py. This monstrosity of Python black magic does everything needed – it locates the Panda3D libraries, adds their location to the system’s library path, registers phony modules, and dynamically loads the required libraries when somebody starts using a class from it.
As for the Python extension functions, those are not supported in the new system. The reason for this is that we are trying to phase it out, and try to move most functions to the C++ side of Panda3D.
Another cool feature is that the ‘panda3d’ module implements a lazy-loading system for it’s libraries. That means that the libraries will only get loaded when you actually use one of it’s classes. This has the advantage that libraries you don’t use will not get loaded, which can result in a slight speed and memory gain.

Rationale

Now, why did we do it? Besides the fact that it has been requested various times, there are a few benefits to it. First of all, it is better organized, allows people to type it faster and more easily, and it leads to cleaner code. Furthermore, you don’t import the libraries you don’t need, leading to slightly faster import times, and slightly less memory usage. But even more importantly, because we need it for the plugin/runtime system.
Uh, what, why? I’ll explain. When you distribute a game that can be used with the Panda3D runtime (be it the browser plugin, or be it the standalone runtime), you pack it into a .p3d file. That .p3d file will indicate which version of Panda3D it is built for. When someone runs a .p3d file through the runtime, the runtime downloads a small ultra-optimized build of Panda3D to run the provided .p3d package.
This Panda3D package that is downloaded by the runtime must be as small as possible. Because the entire Panda3D build will contain big components that not every game will use, we have split those into separate packages. For instance, a “panda3d” package containing the core, an “egg” package containing libpandaegg (usually, a packed game will only contain .bam files, so this is usually not needed), libpandaode, a “models” package containing the default models, one package per audio library, etc. The .p3d file can indicate which packages it depends on, so that the download size is kept to a minimum.
That gives a problem with the old pandac.PandaModules-style import system, though. First of all, pandac.PandaModules imports every single library, while it is not guaranteed that all libraries are on the system (as the .p3d file may not need some of them). We had to put hacky ImportError exception handlers in PandaModules.py to work around that.
Furthermore, with the old system, we have no control anymore over the imported classes. Why we would need that? Well, the game has the ability to download and install more components of Panda3D while it is running. With the new system, when for example the “egg” component is installed, we can easily add a hook into the ‘panda3d.egg’ module to be automatically updated with the new installed libpandaegg library. This allows game developers to keep the pre-download time to a minimum by installing packages on demand (for example, if only a part of the game uses ODE physics, the developer can choose to have the ‘ode’ package installed whenever the end-user chooses to run that part of the game.)

How it affects you

If you’re still awake after reading all that, you might be wondering what will happen to all the existing code. Will this break anything? The answer is, no. Right now, it’s just a small Python file that I added, which allows you to import Panda3D classes using a different convention. This is still experimental, so the ‘pandac’ tree is still around, and not even deprecated. When, perhaps after the 1.7 release series, the new system has been thoroughly tested and proved to work better, we might switch the Python code and sample programs to the new system, and recommend people to use it.
But if you don’t feel like switching, don’t worry. As we care about backward compatibility, and because most of the code is heavily dependent upon the old structure, the ‘pandac’ structure will probably be around for a loooong time.

Examples

Last but not least, some example imports showing you a bit how it works:

from panda3d.egg import EggData
 
from panda3d import ode
joint = ode.OdeJoint
 
from panda3d.core import *
tex = Texture()
tex.read(Filename("img.png"))
 
import panda3d.direct
ival = panda3d.direct.CInterval()