Compiling 1.9.4, rtdist: __path__ empty

Return to Compiling or Editing the Panda Source

Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Thu Aug 31, 2017 12:45 pm

I'm attempting to create a custom build of 1.9.4, including an attempted back-porting of 1.10's vertex-colour handling. Thus far, the SDK and runtime seem to build successfully (although I've neither tested nor installed either yet, I believe). The "rtdist", however, is giving me trouble.

Specifically, the build is exiting with an "IndexError", noting that a list index is out of range, in FreezeTool.py, line 633. Here's the full error:
Code: Select all
Traceback (most recent call last):
  File "direct/src/p3d/ppackage.py", line 254, in <module>
    packages = packager.readPackageDef(packageDef, packageNames = packageNames)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/p3d/Packager.py", line 2811, in readPackageDef
    solo = solo)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/p3d/Packager.py", line 2915, in beginPackage
    package = self.Package(packageName, self)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/p3d/Packager.py", line 381, in __init__
    self.freezer = FreezeTool.Freezer(platform = self.packager.platform)
  File "/home/thaumaturge/panda3d-release-1.9.x/built_thaumaturge/direct/showutil/FreezeTool.py", line 633, in __init__
    modulefinder.AddPackagePath(moduleName, path[0])
IndexError: list index out of range


I added some simple "print" statements above the offending line, and it looks like the problem is happening when it attempts to run the above for "pkg_resources._vendor.six.moves.urllib". The module itself is loaded, but "__path__" seems to contain an empty list--hence the IndexError.

What's going wrong here? It's entirely possible that I'm missing some dependency--I've never quite gotten the hang of figuring out which complaints in the build output are relevant, I'm afraid--but which one? :/

My thanks for any help given!
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby rdb » Mon Sep 04, 2017 5:44 am

Is this Python 2 or Python 3? I believe the 1.9 rtdist won't compile with Python 3.

Any idea why your code is importing this problematic module? Perhaps you can explicitly add it as an excludeModule to proceed?
rdb
 
Posts: 11173
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Mon Sep 04, 2017 12:01 pm

rdb wrote:Is this Python 2 or Python 3? I believe the 1.9 rtdist won't compile with Python 3.


It looks as though I'm building with Python 2.7.12

rdb wrote:Any idea why your code is importing this problematic module? Perhaps you can explicitly add it as an excludeModule to proceed?


I have no idea why it's there, I'm afraid--it's not something that I've made a point of adding, as far as I'm aware.

As to removing it, what is it's purpose in Panda3D--what might I be losing if I remove it?

Oh, one thing that I perhaps forgot to mention: I think that I recall reading somewhere that the "__path__" variable may be empty in the case of built-in methods--as I think that this was indicated to be when I had Python "print" the module. I'm not sure of whether this applies to Python 2.7 or 3, however.
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Fri Sep 22, 2017 9:43 am

*bump* Any further thoughts, here? I'd really like to get a working rtdist built, but I'm hesitant to attempt to exclude the above-mentioned module without knowing whether or not it's important... ^^;
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Tue Oct 10, 2017 1:33 pm

*one more bump*

Is "pkg_resources._vendor.six.moves.urllib" important to Panda3D? What functionality might I lose if I exclude it? It looks like something related to networking--given that my game is offline-only, is that something safe to break?

RDB, you mentioned adding it as an "excludeModule"--looking around, do I guess correctly that this is done by adding a line calling that function to "panda3d.pdef", and presumably the one in "direct/src/p3d"? If not, how is it done, please? (I don't see a mention of it in the manual, although I may have missed it.)
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby rdb » Tue Oct 10, 2017 3:54 pm

I believe it's not part of Panda, so it must be pulled in by a module imported in your code. By placing an excludeModule command in the pdef file that builds your .p3d file. If you use the (less configurable) packp3d command, then there may be an equivalent option, or maybe not, I'm not sure.

Do first check whether it might actually be part of the built rtdist by grepping for the module name in the .import.xml files in the stage directory. If so, you need to check which package it's part of and place an excludeModule in the appropriate .pdef file in direct/src/p3d.

I highly doubt that it's important. It looks like it is part of a vendored module inside a vendored module, and it may not even be used by any code. I suppose you'll find out when you try it.
rdb
 
Posts: 11173
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Tue Oct 10, 2017 7:46 pm

rdb wrote:I believe it's not part of Panda, so it must be pulled in by a module imported in your code.


But I'm just building the Panda3D source code, with the back-ported vertex-colour changes from 1.10, I'm pretty sure. o_0

(I'm building with makepanda.py, in case that's relevant.)

rdb wrote:Do first check whether it might actually be part of the built rtdist by grepping for the module name in the .import.xml files in the stage directory. If so, you need to check which package it's part of and place an excludeModule in the appropriate .pdef file in direct/src/p3d.


Hmm... Where is that directory? Having just built the SDK and runtime, I don't see it...

Come to that, looking again at my "runtime" and "rtdist" makepanda commands, what directory should be specified for their "--host" parameters? I currently have them as "<source-root>/built_thaumaturge/stage", but I see that the runtime, when built, produces the directory "built_thaumaturge_rt" (although said directory doesn't have a subdirectory named "stage"). Should the "_rt" be appended to the runtime's host-parameter, or to the "rtdist"'s, perhaps? (I don't see anything in the documentation that covers this.)

rdb wrote:I highly doubt that it's important. It looks like it is part of a vendored module inside a vendored module, and it may not even be used by any code. I suppose you'll find out when you try it.


Hah, fair enough!
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby rdb » Wed Oct 11, 2017 3:23 am

Oops, sorry, I thought you were encountering it when building your application code. Indeed, try and find out which package it's in, then put the excludeModule into the definitions in the panda3d.pdef or thirdparty.pdef file. Or just put it in all of them if you don't know. You should be able to see in the build log which package it was building last, though, right?

The --host should be a URL pointing to the directory containing the contents.xml, which will be "built_thaumaturge/stage" by default. Note that this should be a file:/// URL.

The _rt is for appended for the runtime build. You don't strictly need to build your own runtime, other than to get a version of the runtime that defaults to the host you specify, but the host location of the "panda3d" package is determined from the package requirement specification in the .p3d file.

On a sidenote, the new deploy-ng system that is intended to replace the rtdist path is becoming increasingly stable, so at some point it may become less effort to get it to work the way you want than the old rtdist system.
rdb
 
Posts: 11173
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Wed Oct 11, 2017 7:21 pm

rdb wrote:Oops, sorry, I thought you were encountering it when building your application code.


Fair enough! I thought that that might have been the case, I believe. ^_^

rdb wrote:Indeed, try and find out which package it's in, then put the excludeModule into the definitions in the panda3d.pdef or thirdparty.pdef file. Or just put it in all of them if you don't know. You should be able to see in the build log which package it was building last, though, right?


Ah, you're right. I just ran the "rtdist" build again, and now that it's failed, I see that the message just before the traceback reads as follows:

Code: Select all
Generating twisted/thaumaturge_1.9/linux_amd64/twisted.thaumaturge_1.9.linux_amd64.mf


Looking that up, I see that it's a networking framework--for which the inclusion of something called "urllib" makes sense, I daresay.

I recall that Panda3D uses a client-host approach to distributing games--even self-contained installers retrieve elements from one or more sources while building. As a result, this might not be safe to omit.

On the other hand, the issue seems to only show up in the "rtdist" build--the "runtime" and SDK builds seem unaffected. As such, perhaps it's only used for game-level networking, in which case perhaps it is safe to omit, given that I'm making an offline-only single-player game?

rdb wrote:The --host should be a URL pointing to the directory containing the contents.xml, which will be "built_thaumaturge/stage" by default. Note that this should be a file:/// URL.


I take it that the "stage" directory is created as part of the build process, then? And should the same directory (as opposed to the "_rt" version) be used when building the runtime?

rdb wrote:The _rt is for appended for the runtime build. You don't strictly need to build your own runtime, other than to get a version of the runtime that defaults to the host you specify, but the host location of the "panda3d" package is determined from the package requirement specification in the .p3d file.


Hum... So what determines the host referenced in the .p3d file? I don't see an option for that in packp3d...

rdb wrote:On a sidenote, the new deploy-ng system that is intended to replace the rtdist path is becoming increasingly stable, so at some point it may become less effort to get it to work the way you want than the old rtdist system.


That does sound good! I'm a little shy of risking a new and untried deployment system as things stand (especially since I do want this game to be commercial, eventually), but if deploy-ng proves sufficiently stable and effective I may well end up switching over to it.
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby rdb » Thu Oct 12, 2017 5:12 am

Yes, twisted isn't used by anything in Panda. You can comment out the definition in thirdparty.pdef. In fact you can remove anything in thirdparty.pdef that you don't need.

The "stage" directory contains what you'd upload to a package host such as https://runtime.panda3d.org although if you only ever use it locally that isn't very relevant.

Thaumaturge wrote:I take it that the "stage" directory is created as part of the build process, then? And should the same directory (as opposed to the "_rt" version) be used when building the runtime?

Assuming I understand your question correctly, yes, if you want your runtime to pull its "images" and "coreapi" packages from the rtdist you have built, you need to specify the built_thaumaturge/stage directory via a file:// url as host.
There is no stage directory in the built_thaumaturge_rt directory.

Thaumaturge wrote:Hum... So what determines the host referenced in the .p3d file? I don't see an option for that in packp3d...

packp3d is built as part of an rtdist, which was built with a --host flag. So packp3d itself has a dependency on a 'panda3d' package from some host or another, which I believe is used as default for other package references.

(I could be wrong and it might actually be using the host that the runtime was built with, I'm not entirely sure.)

Either way, when specifying a dependency, you can explicitly specify a host URL and version with something like -r package,cmu_1.9,https://runtime.panda3d.org or, if you are using the more flexible .pdef specification to create your p3d files, via extra arguments to the require() command.
rdb
 
Posts: 11173
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Thu Oct 12, 2017 3:10 pm

rdb wrote:Yes, twisted isn't used by anything in Panda. You can comment out the definition in thirdparty.pdef. In fact you can remove anything in thirdparty.pdef that you don't need.


Ah excellent--that's good to know! ^_^

I'm currently rebuilding, having edited "thirdparty.pdef". It's taking some time, so I'll report back once it's done, I intend!

rdb wrote:The "stage" directory contains what you'd upload to a package host such as https://runtime.panda3d.org although if you only ever use it locally that isn't very relevant.


Ah, fair enough.

rdb wrote:Assuming I understand your question correctly, yes, if you want your runtime to pull its "images" and "coreapi" packages from the rtdist you have built, you need to specify the built_thaumaturge/stage directory via a file:// url as host.
There is no stage directory in the built_thaumaturge_rt directory.


Fair enough, and noted!

rdb wrote:packp3d is built as part of an rtdist, which was built with a --host flag. So packp3d itself has a dependency on a 'panda3d' package from some host or another, which I believe is used as default for other package references.
...


Ah, fair enough. That seems straightforward, then!

Thank you for all of your help! ^_^
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Thu Oct 12, 2017 7:07 pm

Update: The builds (both SDK and rtdist) seem to have completed successfully! I haven't yet installed or tested either, but this particular issue seems solved. Thank you again for your help! ^_^
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Mon Oct 16, 2017 1:39 pm

Okay, one more question, if I may: What version of GCC is usually used to build Panda3D (presuming that GCC is used at all)? I've hit a problem when attempting to create a distributable build of a game, and some quick Googling suggests that it may be related to a bug in GCC 5...

Specifically, while packp3d seems to work, pdeploy (the version found in "built_thaumaturge/stage", I believe), exits with the following error:
Code: Select all
Couldn't load /home/thaumaturge/.panda3d/coreapi/linux_amd64/p3d_plugin.so: /home/thaumaturge/.panda3d/coreapi/linux_amd64/p3d_plugin.so: undefined symbol: __cpu_model
Unable to launch core API in /home/thaumaturge/.panda3d/coreapi/linux_amd64/p3d_plugin.so
Unable to load Panda3D plugin.
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby rdb » Wed Oct 18, 2017 8:30 am

Yeah, this is an issue in gcc. I think on the master branch this was eventually worked around in this commit:
https://github.com/panda3d/panda3d/comm ... dc5adef324

However there is another copy of stb_image.h in direct/src/plugin which may also need to be patched.

Rather than applying the whole patch, the easiest way for you to fix it might be to look for the stbi__sse2_available function in both copies of stb_image.h, and removing its contents so that it only returns 1, eg.
Code: Select all
static int stbi__sse2_available() {
  return 1;
}


and also changing this line:
Code: Select all
#if defined(__GNUC__) && (defined(STBI__X86_TARGET) || defined(STBI__X64_TARGET)) && !defined(__SSE2__) && !defined(STBI_NO_SIMD)


...to instead read this:
Code: Select all
#if defined(__GNUC__) && defined(STBI__X86_TARGET) && !defined(__SSE2__) && !defined(STBI_NO_SIMD)


As for compiler versions, for building Panda 1.10 we require at least clang 3 or gcc 4.8, though any version after that should work.
rdb
 
Posts: 11173
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Wed Oct 18, 2017 2:23 pm

Does the issue exist in gcc 4.8, do you know? If not, perhaps it would be easier to simply downgrade to that...
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby rdb » Thu Oct 19, 2017 7:51 am

I have no idea. I think it exists in GCC 5 and maybe also in 4.9, but I don't know about 4.8.

Switching from gcc to clang would also work.
rdb
 
Posts: 11173
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: Compiling 1.9.4, rtdist: __path__ empty

Postby Thaumaturge » Thu Oct 19, 2017 1:23 pm

Ah, clang looks like a good option, I think. I've installed it and set the relevant environment variables, I believe, and intend to report back once I've rebuilt the engine and tried to create a distributable with it.
MWAHAHAHAHA!!!

I have a website now! It holds information and a dev-blog for my nascent video-game, A Door to the Mists.
User avatar
Thaumaturge
 
Posts: 1602
Joined: Sat Jun 07, 2008 6:34 pm
Location: Cape Town, South Africa


Return to Compiling or Editing the Panda Source

Who is online

Users browsing this forum: No registered users and 1 guest