Panda3D runtime distributable

I had an idea recently. I think it would be a great idea for panda3d.org to provide a “Panda Runtime” distributable. I imagine something like a panda3d.exe, that will run the Python code and associated modules that are packaged up in whatever .mf file you give it.

Here’s the idea: Joe builds his really cool Panda app. He puts all of the stuff–python files, bam files, prc files, textures, whatever–into joesapp.mf, and emails it to his friend Fred. Fred, who doesn’t have the Panda development environment installed, installs panda3d.exe and then double-clicks on joesapp.mf, which is opened and executed by panda3d.exe. Boom, app-in-a-box.

Of course, Joe could have just used packpanda to build joesapp.exe. But there are a few advantages to having a separate downloadable runtime instead of bundling it all up together.

(1) Perhaps most importantly, we can pre-compile panda3d.exe using OPTIMIZE 4, which is Panda’s compilation mode for production environments of fully-tested code. We can’t provide the development version of Panda compiled in this mode, because a simple parameter mistake will make the whole thing crash. But we can provide the runtime-only version compiled this, which will make the whole thing run approximately 10% faster.

(2) It’s also automatically platform-independent. If Fred is running Mac but Joe only has Windows, it doesn’t matter–Fred will download the runtime appropriate to his computer, and Joe doesn’t need to do anything special to provide a Mac build.

(3) Joe’s distributable is relatively small, just his application data. If Fred downloads several different Panda applications from various friends, he only has to download the Panda runtime once.

(4) Fred can configure his Panda runtime the way he likes it. For instance, if he prefers to run in fullscreen 1024x768 mode, and he knows his DirectX driver is better than his OpenGL driver, he can set these things once, and they will apply to all of the .mf files he runs.

(5) It paves the way for an eventual Phase II, which is to make this Panda executable available as a browser plug-in! With a browser plug-in, Joe would only have to post joesapp.mf along with some suitable HTML wrapper code to embed his application in a browser window. When Fred visited Joe’s page, it could automatically fetch and install the Panda runtime, very much like Flash works today.

Of course, there are a few disadvantages too. I can think of:

(a) Panda hasn’t been very good historically at maintaining backward compatibility, so each application would be version-specific. We’d have to provide a panda3d.exe for each version of Panda that we release, or at least for the major version changes (e.g. panda3d-1.5.exe, panda3d-1.6.exe, and so on). Fred might need to have several different runtime versions all installed at once, in order to run his various friends’ mf files.

(b) It would be difficult to include additional third-party software in your mf file, like pygame or whatever. It would all have to be pure Python+Panda. I guess we might be able to work around this to a certain extent.

© We’d have to find a better solution for the sound library problem. FMod is non-commercial, OpenAL is full of problems. Bleah.

It would be really quite easy to do this, technically. We’d just need to write a tiny bit of glue code that would implement a Python importer over .mf files, mount the .mf file given on the command line, and start running it (for instance, by importing some standard file name, like “PandaMain.py”). Then we could use a tool like packpanda to build an executable of out of this glue code, and we’re done.

The real burden would be on the release maintenance. This would double the distribution burden, because now we’d have to have a Panda runtime as well as a Panda development release, for each version. Still, I think it might be worth it.

What do you think?

David

Definitely sounds like an awesome idea! I have a few questions however:

• Would this work the same with linux? Like debs and rpms? Maybe this runtime can be a default library like libpanda3d.rpm, and that the binaries we now have will be like panda3d-devel.rpm packages? That would make it easier for linux people to package their games, I guess.
• If we strip the minor number from the release numbers, like panda3d-1.6.exe, how would we go about important low-level bugfixes? Would we release a new runtime under the same name?
  About the audio thing, we’ll need to rewrite the OpenAL stuff anyways, it’s terribly buggy and slow.

I was thinking about that too, I don’t know in which release you actually plan to integrate this idea? Also, I have no idea when CMU starts finding a replacement for Josh.

Not only true third-party software, but also all extensions or add-ons to Panda3D (following the “skel” pattern).

A workaround could be a plugin-system, another one an addon system. After installing panda.exe a user installs panda-addon-x.exe, which copies it’s files (.dll, .so, whatever) into a specific directoy of the panda.exe installation, where it can be found. It would be nice to have a predefined way of building such add-on installer, to make them uniform in behaviour and usage, like the python distutils.

enn0x

Right, it would work the same with Linux. Making a panda3d.rpm and a panda3d-devel.rpm would be a sensible division, consistent with other software.

I don’t mean to strip the minor number from the release altogether. It should still be part of the download and so on. But it shouldn’t be part of the installed filename, so that downloading 1.6.3 will replace 1.6.2, for instance, but have no effect on 1.5.4. Also, it (probably) shouldn’t be part of the version-requirement system, whatever that will be. If a particular package says it requires version 1.6, it should be allowed to run with 1.6.2, or 1.6.3, or whatever minor version of 1.6 the user happens to have around.

I’d say, whenever it’s ready. No reason to rush it out too soon.

I wonder about that, too.

Right, well, really it’s just a problem with distributing additional compiled C++ code in general. We could allow this, by allowing pyd’s (or dll’s) in the mf file and unpacking them and loading them, though this impedes portability.

The add-on proposal is a workable solution that might help the portability issue, provided the add-on developer is willing and able to compile his add-on for every available platform. (Of course, if the developer could do this, he could also bundle up all of the dll’s, so’s, and dylib’s in his mf file.)

It all sounds terribly complicated, though, and my first suggestion is to keep thinking about solutions for this problem in the back of our minds, but not to let the existence of the problem prevent us from doing the easy thing for pure Python applications.

David

This is exactly how my 2aw works currently. 2aw code and data is just one mf file. I have a simple loader which just starts the code in the mf file and even unpacks with some c++ extension. All data and code is in the mf file so its easy to patch. I actually have a blue print for this already and have though about the problem:

blueprints.launchpad.net/panda3 … ntime-only

And as you notice is an old one.

Also about distributing C++ code, there is platform independed but architecture depended way to pass the code around as compiled objects which are linked to the proper os libraries during linking stage… i mainly seen this with x86 assembly projects though not the c++ but it might be possible if the packages use only the panda3d functions.

whant to update this thread with what David said

Just to give my view on third party extensions… I think it should be installed separately from the .mf file, requiring the user to explicitly download and install a separate executable. Reason being I see the main value of this being the ability to ultimately get web browser support - and then security becomes an issue. There is no way that you can allow arbitrary code in a web browser loaded app, as any malicious website could destroy you once the plugin is installed, so I think the ‘standard’ for these .mf files should be consistent in not supporting such extensions. Yeah, its not ideal, but if this web capability comes into being then security has to come first.

Has there been any progress on this idea yet?

I’m working on getting Panda to compile under the openSuse build service, and I already succeeded for rpm-based distros. This made it very easy for me to split it up into multiple packages (since I don’t have to do any compiling myself now.) So, I’ve already split it up into a normal panda3d package, a -devel and -debug and source package.
The question is: what if someone has multiple versions of the runtime, how would we handle that case on Linux?
Also, we’d need to find a nice way to do this on Windows.

PS. Wouldn’t it be a good idea to have the extension for a game-packaged-into-multifile be something different than .mf? That would separate the concept of “packaged game” and “just an archive with some model files”.

Hmmmm, another thought. This really shouldn’t happen on a Saturday morning. When using a .mf file as a web plugin it will need to have its file writing capability limited obviously, otherwise it could go and replace shared libraries etc, which would be bad. You might want to pre-empt that by defining some function that returns a directory that its allowed to write to, for storing local data - saved games etc. Would have to be keyed off the path of the .mf file, be it local path or url. Also include a function that returns how much space its got - will probably want to cap that for web apps, otherwise you could get 100meg files from a browser game, which I would personally frown on. Its a simple little task, but if web games are your ultimate goal then standardising this stuff early will help.

These are all good points. I do like the idea of a special extension for application-based .mf files.

Note that security with a web browser plugin is a going to be a major issue, and one not yet solved. The problem is that Python itself is real hard to secure, by its nature–simply prohibiting arbitrary C++ code from running doesn’t help at all to secure the plugin, because arbitrary Python code is just as bad. People have made attempts to secure Python from time to time, and they all either (a) don’t actually work, or (b) cripple the Python runtime so much that it can’t do anything.

I might be a little harsh in my estimate for (b). In our case, we don’t actually need to protect Python code from compromising other running Python code, unlike the Zope project. All we really need to do is prevent arbitrary disk writes, which might be doable by simply recompiling Python with a crippled file object that cannot write anything. And then we can require users to go through the Panda mechanism (e.g. direct.stdpy.file) in order to write to disk, and that code we can lock down.

I’m now thinking the runtime itself should manage multiple versions. Kind of like the way DirectX works, with all of its backward compatibilty internally built-in. The runtime would load up the mf file and check what version it requires, and if it requires an older version of the current runtime, then it would automatically switch to that older version before continuing. This would all have to be packaged up within the same application/plugin/whatever, so it would have to be a pretty smart system, ideally even capable of automatically downloading and installing the appropriate versions when needed.

David

Considering that editing and then maintaining a python fork is a bad idea running the entire process in some kind of sandbox might be the best way - you wouldn’t need to intercept all system calls, just once involving access to other processes, filesystem, gui that isn’t yours and possibly networking. Sound and graphics could left unmolested beyond initial window creation, so there wouldn’t be much/any slow down. Problem is I know of no such software - its all heavy full virtualisation; and writing such a beast, multi-platform, would be a serious project. But thought I’ld mention this solution in case something pops up - never know what OS project might suddenly appear!

(And the extension of a game .mf file should surly be ‘.p3d’ right?)

Eww. Yeah, making a custom build of Python is a pain in the behind, but it’s just one or two trivial changes, and it is still nowhere near as horrible as a full, cross-platform, virtualized process. Especially a virtualized process that can successfully access the 3-D graphics hardware.

David

I see some work on “pfreeze” in the current commits and its ability to output to MF files?

Can we get some more info on what your vision about this is?

It would be cool to be able to package your entire app into a MF file and have some panda3d runtime be more like jvm which just runs them client side. Then move onto running them inside browsers, iphones, or any other device thing.

That is precisely the vision.

David

I see this is the doc for pyfreeze:

I am wondering what should the policy be on using python based tools for panda3d. It looks like most tools are c++ based except this one. I have several python tool i want to contribute to panda3d obj2eg.py, eggoctree.py, bunch of egg cleaner utils. But they are all written in python unlike standard panda3d tools. I wonder how i get get them in and have them work. It would be cool obj2egg.py and 3do2egg.py could be made part of the loader too. Its much easer to write importers/exporters in python then C++ for me. The c++/python tool boundary been the major road block for me making my python tools general purpose.

It’s true we don’t have a standard policy for installing Python scripts yet, though we do have a growing library of these things. One problem is that on Windows, it’s not obvious how to install them usefully to people who are not running a Cygwin or other unix-like shell.

One answer is just to ignore that particular problem, and install these scripts in a standard place (e.g. /usr/local/panda/bin) where they will be convenient to Unix users, and let Windows users either do without, or install a Unix-like shell. Since these tend to be command-line tools anyway, that’s not altogether unreasonable.

Another answer is to install .bat files for Windows users that invoke the Python interpreter correctly. genPyCode is installed in this way, for instance. This gets to be a nuisance when there are more than a handful of scripts, though.

(Note that pfreeze.py is just a work in progress and is not likely to persist in its current form for long.)

David

In my opinion, talking about the linux side that I live, P3D got without doubt to follow the repository way of distribution because is the way to go on the main desktop distros and because the user is accustomed to install packages in that way, therefore P3D got to push to enter into the main deb and rpm repositories one day or another (pro_rsoft is already heading in that direction I guess). So, as long as P3d library is in that stream, the game dev gotta just think to make his game and build his package with the tools provided by his distro defining P3d just as a dependency, as well as eventual several others used into his game, like pygame and so on.
Can’t see a similar workflow for Windows and Mac though…

Or, we could do it how makepanda ships the python scripts such as genpycode and packpanda. In directbase, theres “ppython.cxx” which is a C script that invokes python with the full path to the correct script (specified in #define’s.)
So, the path to the script relative from the panda root is hardcoded in the resulting executable.

Might be easier to just freeze them in the compile or install step, though.

While I see the benefits, this also has downsides. It’s a lot more pain to maintain than you think, and just releasing a new version is not as easy as just uploading a new package.
I did speak to one of the maintainers of the openSUSE repositories and we might just be able to get Panda in there - that’s easy to maintain since it’s so tightly connected to the openSUSE build service. If it finishes building something that happens to be in the openSUSE repository, it will automatically upload itself to there.

who are you referring to, the library mantainer, the game dev or both, and what are the difficulties (pain) you’re talking about?

For the game dev it would only be easier. For the package maintainer it would be hard - and I don’t know if they even allow non-backward-compatible updates to be made to software packages.
Also, since not all Panda versions are backward compatible, there are downsides to the just-setting-panda3d-as-dependency approach.

Do note that packpanda is designed to handle this - it packs a copy of Panda into the game itself, resulting in a .deb or .rpm. I believe it supports deb and rpm since version 1.6.0.

But anyways, I think we’re drifting away from the original purpose of this thread.