Successfully Deploy-ng

I’m currently attempting to build a distributable demo of my game, and am using Deploy-ng to that end. I’ve gotten to the point of having Deploy-ng successfully run all the way through, producing a zip-file containing the game. This thread, then, is for the remaining issues:

First of all, the resultant game crashes immediately on startup. Running it from the command-line (after renaming the executable to no longer have spaces–I’m not sure of why it wouldn’t accept it before), I get the following output:

Known pipe types:
  glxGraphicsPipe
(all display modules loaded.)
:audio(error):   load_dso(libp3openal_audio.so) failed, will use NullAudioManager
:audio(error):     No error.
Traceback (most recent call last):
  File "GameCore.py", line 969, in <module>
  File "GameCore.py", line 82, in __init__
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/build/__whl_cache__/panda3d-1.10.0-cp27-cp27mu-linux_x86_64.whl/direct/showbase/ShowBase.py", line 1685, in enableParticles
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named ParticleManagerGlobal

So, it’s finding the expected graphics output, which is good. Unfortunately, it doesn’t seem to be finding the audio module, nor “ParticleManagerGlobal”. My “setup.py” file does include “p3dopenal_audio” in the “plugins” section.

Here is the full “setup.py”:

from setuptools import setup

setup(
    name = "A Door to the Mists",
    options = {
        "build_apps" : {
            "include_patterns" : [
                "*.png",
                "*.egg",
                "*.ogg",
                "*.cur",
                "*.ico",
                "*.txt",
                "*.dat",
                "*.glsl"
                "*.prc"
            ],
            "gui_apps": {
                "A Door to the Mists" : "GameCore.py"
            },
            "plugins": [
                "p3dopenal_audio",
                "pandagl",
                "pandabullet"
            ],
            "platforms": [
                "linux_x86_64",
                #'win32',
                #'win_amd64'
            ],
        }
    }
)

Next, is cross-platform building available with deploy-ng, and if so, how might it be done? (Simply commenting-out the linux entry and un-commenting the “win_amd64” entry above results in the system attempting to get a copy of Panda3D from the server, it seems. At a guess, do I cross-compile the Panda “wheel” somehow, allowing it to use that?

Finally, the system doesn’t seem to like including “.prc” files–and I see that it automatically includes two of its own. Up until now I’ve been loading my own “.prc” file via “loadPrcFile”–I gather that there’s a better way, which I imagine that setuptools draws on. How do I specify that a given “*.prc” file is the one to use for my project, both when run via the IDE or in a distributable build.

[edit]

Ah, I stand corrected on “.prc” files–that was the result of a mistake on my part in my “setup.py” file. Sorry about that! ^^;

Still, I have the impression that there is a “more correct” way of handling “.prc” files, so the overall question stands, if it may.

[edit 2]

A minor edit: I figured out how to correctly run an executable from the command-line when it has spaces. I do think that I’ve done this before, so I’m putting my difficulty with this down to tiredness. >_<

[edit 3]

I’ve cleared the audio error–another mistake on my part, I’m afraid: If you look above, my “setup.py” asks for the plugin “p3dopenal_audio”–but the correct plugin lacks the “d”, being just “p3openal_audio”. Sorry about that! ^^;

(The “ParticleManagerGlobal” error remains, however.)

You need to explicitly add an import to direct.particles.ParticleManagerGlobal somewhere in order for the particle system to be included. This is to avoid the particle system from being shipped by default.

You need to add “p3openal_audio” to the plugins list, not “p3dopenal_audio”. Also, check that your build of Panda actually includes libp3openal_audio.so, and you didn’t accidentally build without OpenAL support.

Cross-platform building is supported by adding the appropriate platforms. If you are using a modified build of Panda3D, you will need to make wheels for all the platforms and put those in a directory. Otherwise, you can simply use the provided ones.

We’re still working on a better solution for handling .prc files, but for now, it should be fine just including .prc files and explicitly loading them with load_prc_file.

Thank you for the answers! :slight_smile:

Ah, I see. I’m importing “ParticleEffect” in a few places, I believe, which I take it isn’t enough. Do I take it that it’s safe to just import it into the main game script? (I’ll do so in the meanwhile, I think.)

Yup, I discovered the mis-naming between my post and yours (I added mention of it via an edit, but presumably too late). Sorry about that! ^^;

Fair enough. I’ll experiment with building cross-compiled wheels, then, I intend!

[edit]
It seems that cross-compilation from Linux to Windows doesn’t work, which is a pity. :confused: I suppose that I’m looking into building under Windows, then.

(I prefer to use my own build because I know what’s been built into it, and can thus be somewhat confident about what licences I might be subject to.

Well, that and the fact that I have an additional Bullet-method exposed, but I imagine that I could request that it be exposed in the official implementation.)
[/edit]

Fair enough; I’ll leave it as-is for now, then. :slight_smile:

I suppose we should more clearly publish a list of packages and their licenses. However, deploy-ng is smart enough not to pull in packages that aren’t being used, so you shouldn’t be subject to eg. the FMOD license if you aren’t pulling in the p3fmod_audio plug-in.

I’m happy to consider your patch to expose another Bullet method. Perhaps other users can benefit from it as well!

Hmm… When I build Panda, I build with a long list of “–no-<insert_package_here>” parameters, in order to reduce the number of potentially-problematic licences. For example, I believe that I exclude fmod, WX, Eigen, the various Maya- and Max- plugins, and so on.

So, between the existence of those various parameters, and what you say, do I take it then that when a build is made, whatever third-party elements are included in the build are built as separate modules, not folded into the engine itself, and thus that they’re only used in a given project if that project requests them? Or are the parameters perhaps vestigial?

I think that those parameters may be one of the sources of my uncertainty: the fact that I can exclude things suggests that they’re otherwise included.

As to the patch, it’s a very small thing: I just expose a single method. I’m currently doing it in BulletWorld -.h and -.i, I believe; that may not be the best place to have it, so be prepared for the possibility of moving it to elsewhere.

I’m… not sure of how to make a proper patch. This is what I came up with via Git:

diff --git a/panda/src/bullet/bulletWorld.I b/panda/src/bullet/bulletWorld.I
index 61893a1..3054aec 100644
--- a/panda/src/bullet/bulletWorld.I
+++ b/panda/src/bullet/bulletWorld.I
@@ -95,3 +95,13 @@ get_dispatcher() const {
   return _dispatcher;
 }
 
+/**
+ *
+ */
+INLINE void BulletWorld::
+set_force_update_all_aabbs(bool force_update_all_aabbs) {
+
+  _world->setForceUpdateAllAabbs(force_update_all_aabbs);
+}
+
+
diff --git a/panda/src/bullet/bulletWorld.h b/panda/src/bullet/bulletWorld.h
index 021b6ee..b39e147 100644
--- a/panda/src/bullet/bulletWorld.h
+++ b/panda/src/bullet/bulletWorld.h
@@ -167,6 +167,9 @@ PUBLISHED:
   MAKE_SEQ_PROPERTY(constraints, get_num_constraints, get_constraint);
   MAKE_SEQ_PROPERTY(manifolds, get_num_manifolds, get_manifold);
 
+  // Set whether the world forces the updates of all AABBs, even of static objects
+  INLINE void set_force_update_all_aabbs(bool force_update_all_aabbs);
+
 PUBLISHED: // Deprecated methods, will be removed soon
   void attach_ghost(BulletGhostNode *node);
   void remove_ghost(BulletGhostNode *node);

In short, I’m exposing the “setForceUpdateAllAabbs” method of the Bullet world-class.

Okay, a new problem:

I have a number of sub-directories that hold various specific script files–things like level-scripts, cutscene scripts, language files (which I’m storing as simple Python scripts), etc. Unfortunately, at least some of these don’t seem to be included in the build; on attempting to run a build, I get the following error:

Traceback (most recent call last):
  File "GameCore.py", line 971, in <module>
  File "GameCore.py", line 151, in __init__
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/Adventuring/Common.py", line 1396, in setLanguage
    Common.importText(entry)
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/Adventuring/Common.py", line 1408, in importText
    result = __import__("%s.%s.%s" % (Common.LANGUAGE_DIRECTORY, Common.language, entry), fromlist = module)
ImportError: No module named Languages.eng.Misc

As you may guess, my project directory has a sub-directory structure and relevant Python files to compose the module indicated above.

I’ve tried adding these scripts via “include_patterns” in my “setup.py”, of the following form:

"Adventuring/Languages/*.py"

But this doesn’t seem to work: the files seem to still be omitted. :confused:

It may be that these modules are being missed as a result of not having their importation hard-coded: they tend to be imported dynamically, at run-time, as I recall. (Indeed, an example of this can be seen in the error above.)

~

By the way, I had an odd experience: In attempting to test a change to my “setup.py”, I ran Deploy-ng without deleting my “build” directory. This appeared to work–until it reached the point at which it said “adding files” (or something to that effect). At this point it remained, and made no further progress–and my computer slowed down, then seemed to effectively freeze. I tried this twice, and ended up forcing my computer to shut down on both occasions. After the second, I deleted the “build” directory, ran Deploy-ng–and it worked as expected. However, it’s hard to be confident of what changes I might have made between the first failed run and the run before, so I’m not sure that this is a useful report. :confused:

On a more positive note, I am glad to see that Deploy-ng doesn’t re-download its wheels for each run, as the old build-system used to, I recall!

[edit]

Okay, I’ve found a few more details regarding the “missing modules”:

Attempting to add the relevant Python or “.pyc” files via “include_patterns” fails because Deploy-ng automatically adds both file-extensions to the “ignore” list, which overrides entries in 'include_patterns".

I do see now that there is a list (or possibly dictionary; I’m not sure) that can be included in the “setup.py” file in order to add such “hidden” imports, called “include_modules”. This may well work for such things as level-scripts (once I’ve figured out its expected structure). However, I’d rather not have my language files included via “freezing”: While I don’t know whether this game will ever be translated, I want the functionality to be available. To that end I would prefer that language-files remain separate from the game, potentially allowing new translations to be “dropped in”.

All right, I may have found a workaround for the “language-file” issue–or at least the start of one. While Deploy-ng doesn’t transfer the language-“scripts”, simply copying them over seems to work. Previous experiments had seemed to indicate that it didn’t, but I’ve now moved them in the game’s directory-structure, which may have helped: They had been below a sub-directory that included game-scripts that were part of the freezing process. It seems plausible (to my admittedly limited knowledge in this) that there might be a collision between two “versions” of that sub-directory: one within the “frozen” program, and one in the actual directory.

Could support perhaps could be added to Deploy-ng for this sort of thing? It doesn’t seem “clean”, but perhaps a new list in “setup.py”, similar to the “include_patterns” list, that allows the developer to nominate files that should be copied as normal files, but that overrides the “ignore” or “exclude” lists?

I have added supporting non-frozen Python scripts to the todo list for deploy-ng.

Excellent, and thank you very much! :slight_smile:

Except for Eigen, all of the packages you named are built as separate modules and not included unless you ask for them. Eigen is compiled into the core (it gives a significant performance improvement to linear math operations) and licensed under the permissive MPL2 license.

FMOD is only included if you add p3fmod_audio to the plugins list.

Wx is actually not used in the SDK at all, only for the runtime, which is no longer relevant with deploy-ng. I also believe we don’t actually build the Linux wheels with max/maya plug-ins, but even if we did, it wouldn’t matter since deploy-ng would not include them.

The other options to makepanda that are built into the core and not excludable are: zlib, png, jpeg, tiff, openexr, squish, freetype, harfbuzz, openssl, fftw, nvidiacg. We don’t compile with fftw support because it is licensed under the restrictive GPL license, and the rest are permissively licensed, with the exception of nvidiacg, which has a proprietary (but not restrictive) license. I believe you are using Cg in your game, though, correct? (We do plan on moving the Cg support to an easily excludable plug-in.)

Even if the other packages are excludable at runtime, that doesn’t mean the options are vestigial. Some people who build Panda themselves simply don’t want to wait for makepanda to build parts of Panda they don’t use. :slight_smile:

Thanks, your suggested change is in the repository and has made its way into the deploy-ng branch.

Hmm… I’ll want to look over the MPL2 licence. Based on a partial look at the FAQ for it, it seems fine–but I want to check, and I also don’t want a situation in which I’m required to bloat my game’s licence with a long chunk of legal text. (A small chunk is fine.)

Huh–that’s interesting to know! :slight_smile:

Of those, the only one that I seem to be explicitly excluding is “fftw”. According to the “third-party dependencies” page, it’s either under GPL2 (which I seem to recall is a potential problem, isn’t it?), or a proprietary licence that requires payment.

I’m using GLSL, actually. (I started off with Cg, but switched over to GLSL.)

Unless the change is likely to happen soon, that may be one that I should look into, then. :confused: (If it’s not restrictive, then that’s good–but I want to check that it’s not something that’s going to complicate matters for me in some way, such as by requiring royalties, or the above-mentioned licence-bloat.)

(I wonder how I missed excluding it. Weird. o_0)

(Licences are. A. Pain. >_<)

That’s fair enough, I suppose!

Excellent! Thank you very much, and I’ll hopefully run a “git pull” soon. :slight_smile:

[edit] Of course, in light of the discussion above, I should perhaps just download the latest build. Building from source has become a bit of a habit, I suppose! ^^;

All right, I’ve managed to create a working build with deploy-ng! :smiley:

(As I posted in the GitHub issue list, I believe that I did encounter an issue with the texture-references stored in bam files; my workaround should be noted there.)

So far, all seems to work. :slight_smile: I did get the following output in the terminal, however:

:loader(warning): Unable to load libp3ptloader.so: No error.
:loader(warning): Unable to load libpandaegg.so: No error.

Should I be concerned about either? I imagine that “libpandaegg” is only called for if there are egg files to be loaded, and since I’m using processed bam files, the lack of it should be fine.

This has now been added, but it was only partially tested (Python file at least gets copied to the build) since my build got messed up and it is getting late here.

The change makes it so now only frozen Python modules are added to the ignore list instead of all Python files. You can use exclude_modules to not freeze certain modules, and you can use include_pattern to specify a pattern that matches the Python files you wish to copy. Something like:

'include_patterns': ['plugin.py'],
'exclude_modules': {'*': ['plugin']}

As for the missing plugins you’re seeing, I think they can be ignored if you are loading BAM files. However, we should clean those up to avoid confusing users. I believe this is already on our todo list.

fftw has a problematic license, as you said, but again, it’s not a problem because none of the official builds include it.

To comply with the MPL, you have to include a notice (it does not have to be a particularly obvious place) indicating that includes the Eigen library, that it is licensed under the terms of the Mozilla Public License, and indicate where the Eigen source code can be obtained from. (I am not a lawyer, needless to say.)

NVIDIA Cg has a completely royalty-free license.

You can remove the messages about libpandaegg and libp3ptloader being missing by removing references to pandaegg and p3ptloader from the .prc files in the resulting build. We hope to automate this.

Thank you very much! I appreciate the effort. :slight_smile:

I intend to give it a try today; if I encounter any issues, I’ll presumably report them–but hopefully there’ll be none. :slight_smile:

Erm… but you said above that it was included, I thought. o_0

From your earlier post:

That’s reassuring, at least. Thank you. :slight_smile:

(I’ll still look both over just in case, I intend–but the reassurance nevertheless helps, I believe.)

Fair enough! I’m glad that they’re harmless–thank you for clearing that up! :slight_smile:

(Since they’re harmless, I see little reason to remove them at this point, I think.)

Hmm… A new question, if I may: do programs built via deploy-ng produce log-files, and if so, where are those files stored under Ubuntu? (Ubuntu 16.04, I believe, in case there are differences between versions.)

I seem to be encountering a problem, but I’m not sure of what’s causing it, and I’m hoping that log-file output will help.

[edit] Actually, I could use the location of the log file in Windows 8.1, for that matter–I’m seeing a crash when I run a deployed program there. :confused:

(I’ve checked the old location under Ubuntu at least, I believe–"~.panda3d"–but I don’t see a new log file there.)

There’s currently no automatic log file writing, I’m afraid. This is something we could consider as a TODO. I’ll bring it up with Moguri. In the meantime, you can use the notify-output setting to redirect the output from Panda3D, and assign sys.stdout / sys.stderr to an opened file in your Python code to redirect the output from Python.

However, on Ubuntu, you should be able to run your program from the command-line and you should get the regular command-line output.

On Windows, in order to get command-line output, you will need to build it as a console app, not as a GUI app. We will definitely need to think about how we want to handle logging here.

Keep reading, in the next line I said:

Fair enough, and thanks. :slight_smile:

That is indeed what I’ve been doing, and I still didn’t see the output that I was looking for. However, sometimes silence is itself the answer: your indication that output should be appearing there implies that I’m seeing neither error messages nor the print-outs that I expected. Thus the code in question wasn’t being reached at all.

In short, it turns out that before loading the script-files for cutscenes, I was first checking that the files existed. While dealing with actual Python files, this makes sense. However, when the scripts are “frozen”, no file is left behind, and thus the existence-check fails.

I could use the newly-added functionality that allows me to copy Python files over, but in this case I don’t think that it’s called for: I’ve simply removed the existence check, which seems to have fixed the problem. (The code that actually sets the script-object has a “try-except” block around the importation of the script, allowing me to silently ignore any that aren’t actually present.)

Fair enough, and thanks–I’ve made a new Windows build and edited the generated “Config.prc” file, as well as (hopefully) redirecting sys.stdout and sys.stderr. We’ll see how it works once I’ve booted into Windows and tried it out! :slight_smile:

Ah, you’re quite right–I presumably misread that. Sorry about that! ^^;

Right, log-file reporting under Windows via redirection of sys.error and sys.stdout worked just as described, I’m glad to say! :slight_smile:

And indeed, I believe that I’ve found the source of my crash: it seems that “setForceUpdateAllAabbs” (from the patch that I submitted earlier in the thread), while present in the Ubuntu wheel, isn’t present in the Windows one. Has the latter perhaps not been built recently?

[edit 2] Actually, looking at the wheel mentioned in the error, and comparing it with the files listed on the download page, it seems that deploy-ng may have downloaded an old wheel for some reason. o_0 [/edit 2]

[edit]

Here’s the full error:

Traceback (most recent call last):
  File "GameCore.py", line 365, in performExpensiveOperationInternal
  File "GameCore.py", line 693, in enterLevelInternal
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/Adventuring/Level.py", line 496, in __init__
AttributeError: 'panda3d.bullet.BulletWorld' object has no attribute 'setForceUpdateAllAabbs'
Traceback (most recent call last):
  File "GameCore.py", line 979, in <module>
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/build/__whl_cache__/panda3d-1.10.0.dev1405+deployng.183-cp27-cp27m-win_amd64.whl/direct/showbase/ShowBase.py", line 3044, in run
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/build/__whl_cache__/panda3d-1.10.0.dev1405+deployng.183-cp27-cp27m-win_amd64.whl/direct/task/Task.py", line 531, in run
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/build/__whl_cache__/panda3d-1.10.0.dev1405+deployng.183-cp27-cp27m-win_amd64.whl/direct/task/Task.py", line 485, in step
  File "GameCore.py", line 365, in performExpensiveOperationInternal
  File "GameCore.py", line 693, in enterLevelInternal
  File "/home/thaumaturge/Documents/My Game Projects/Adventuring/Adventuring/Level.py", line 496, in __init__
AttributeError: 'panda3d.bullet.BulletWorld' object has no attribute 'setForceUpdateAllAabbs'

The Windows and macOS builds were out of date. Up-to-date ones are now available.