Hi this is mostly for rdb regarding an attempt to migrate Panda to Python 3, but if anyone has something useful to add feel free obviously.
I managed to get all the installation stuff set up correctly using the latest CVS and replacing the python 2 dist with python 3.3 in the “win-python” folder. I tried 3.2 first but the python installer refused to do anything but offer to alter my existing installation, so I went with 3.3. If this is a problem I’ll just swap them around before I start any serious work on this.
I tried building with “makepanda\makepanda.bat --everything --installer” just to see how far it would get. The first attempt crashed with Maya2010 being the culprit. Adding " --no-maya2010" got me through the Panda build part, but predictably it failed when it got to the python code generation stage:
[ 92%] Generating 'pandac' tree
Traceback (most recent call last):
File "direct\src\ffi\jGenPyCode.py", line 74, in <module>
from direct.ffi import DoGenPyCode
ImportError: cannot import name DoGenPyCode
What this unhelpful error msg is actually trying to say is that python 3 enforces absolute imports when using “import” directly, so for example in DirectNotifyGlobal.py (which is actually where the first error occurs, not jGenPyCode.py):
import DirectNotify # fails.
# Must be absolute:
import direct.directnotify.DirectNotify as DirectNotify # works.
I went through and fixed a few by hand just to see how far I’d get and I eventually began to run into some other errors related to version as expected. I guess this would be my starting point then? But I’m a bit confused as to what code is auto generated and what code I would be working with exactly. I’m presuming that altering the direct tree directly would be pointless since the build process would just regenerate it? I also notice that there is a src folder in the direct tree which duplicates the code which is what the compiler actually invokes. I didn’t really look into this that much, but maybe you could point me in the right direction here. Thanks.
There is little to no auto-generated Python code. The underlying C++ code is in libpandaexpress.dll, libpanda.dll, and more of these libraries, which are loaded by panda3d.py (for new-style imports) or pandac.PandaModules (for old-style imports).
So, I’d say you’re on the right path. built/direct is really just a copy of direct/src.
Loading the libpanda library crashed for me last time I tried (although libpandaexpress and using its classes worked fine), which means that you’ll probably run into this when you get to the pandac.PandaModules or panda3d.core imports. I have not had the time to track this down yet. However, you should be able to get quite far without this already.
I was able to make quite a bit of progress in my first run of actually trying to update some code. I just stepped in where the compiler failed “direct/ffi/jGenPyCode.py” and ran it from the console alternately using python 3.3 and 2.7, fixing issues as I encountered them to see how far I could get. My first impression is that it is not going to be as difficult as I originally thought, at least on the python side of things. A fairly sizable file like PythonUtil.py (4500 lines) only took about 30 minutes or so before it was working with both python versions (at least its initialization functionality; no syntax, import errors etc.). Also the callable issue is no longer a concern as they’ve re-added it to the language since version 3.2.
There were some things like the re-structuring of the xml and html modules in 3 that required hacks like:
if sys.version_info[0] < 3: from HTMLParser import HTMLParser # Py2
else: from html.parser import HTMLParser # Py3
but most of it should be fairly clean one line substitutions that work in both versions. The first real problem I encountered was the first time imp.load_dynamic is invoked in “extension_native_helpers.py” to get the libpandaexpress.dll. In 2.7 this obviously works fine but in 3.3 it crashes with:
File "C:\panda3d\direct\extensions_native\extension_native_helpers.py", line 81, in <module>
Dtool_PreloadDLL("libpandaexpress")
File "C:\panda3d\direct\extensions_native\extension_native_helpers.py", line 79, in Dtool_PreloadDLL
imp.load_dynamic(module, pathname)
MemoryError
You mentioned you were able to get at least libpandexpress working in your attempts so I’m assuming this isn’t the crash you were referring to? I watched the memory hit as it ran before it crashed and it didn’t even come close to maxing out my memory so I’m not sure what’s happening here. I checked the 3.3 ref and the “load_dynamic” function is no longer listed in the imp section and a lot of the other functionality is deprecated redirecting the user to importlib, but I couldn’t see anything there that resembled load_dynamic. Either way I should be able to continue working through other branches of the direct tree and start making the syntax changes and such whatever the problem turns out to be here.
I haven’t encountered the MemoryError myself. I compiled with Python 3.3 myself, but I admit I haven’t tried pandac.PandaModules, only panda3d.core and a direct import. Copying and renaming libpandaexpress.dll to libpandaexpress.pyd should make it importable directly, can you try this followed by “from libpandaexpress import *” to see if this yields the same error?
Using the .pyd extension gives the same errors, I tried various combinations:
from libpandaexpress import *
import libpandaexpress
from libpandaexpress import Buffer
with the same results. I tried the same thing with other libpanda*.pyd files, and even smaller libraries like libpandaskel.pyd produced the same thing. I’m guessing this is a problem on my end then since you haven’t encountered it; 2.7 has no problem with these tests so its obviously in my 3.3 build somewhere. I’ll try messing around with some empty scripts later tonight to see if I can get any imports to work at all. This is the system error report it dumps into the Win temp file if it’s of any use:
Ah, you compiled with Visual Studio 2010? We don’t currently support it, so the crash may be related to that. Do Panda tools such as pview even run correctly?
Turns out I had an old install of 2010 still kicking around so I thought that would be the problem. I uninstalled 2010 and scrubbed all traces of it from my system and then re-installed 2008 only to encounter the same error again with the report still saying it was built with Visual Studio 2010. I was pretty thorough in erasing all traces of Visual Studio (both versions) from my system before re-installing 2008 (even got rid of all the .NET installations), but it’s still saying it’s building from 2010 somehow, any thoughts on this?
The only thing that I can think of is that it might have something to do with MS no longer distributing 2008 publicly (all links just take you to 2012). I had to search around on some forums to find the an old link buried on an MS server to even get my hands on 2008, but I honestly can’t see how this would make a difference. pview and other panda tools run fine, btw.
Python versions prior to 3.3 use VS2008, but Python 3.3 is compiled with VS2010. So actually you might run into problems with VS2008. I use VS2010 for more than two years now to compile Panda3D, and had no problems so far. There is just a tiny modification needed so makepanda will find the Win7 SDK.
I will try and compile with Python 3.3 one of the next days.
By the way: are you using 2to3 for the Panda3D Python code? Updating by hand is - in my opinion - way too error prone.
Hmm, perhaps. I compiled using VS2008, but I use a different set of thirdparty packages, and I used a 64-bit build. Maybe I should share the thirdparty packages that I used, so that you can try making a 64-bits build like I did?
Just to see what would happen I tried a build with 3.2 but it fails fairly quickly with:
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(211) : error C3861: 'PyUnicode_AsUTF8': identifier not found
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(223) : error C3861: 'PyUnicode_AsUTF8': identifier not found
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(253) : error C3861: 'PyUnicode_AsUTF8': identifier not found
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(265) : error C3861: 'PyUnicode_AsUTF8': identifier not found
Storing dependency cache.
Elapsed Time: 1 min 5 sec
The following command returned a non-zero value: cl /wd4996 /wd4275 /wd4267 /wd4101 /wd4273 /Fobuilt/tmp/p3interrogatedb_composite2.obj /nologo /c /arch:SSE2 /Idtool/src/interrogatedb /Ibuilt/tmp /Ibuilt/include /Ithirdparty/win-python/include /DMAKEPANDA= /MD /Zi /O2 /Ob2 /Oi /Ot /fp:fast /DFORCE_INLINING /Fdbuilt/tmp/p3interrogatedb_composite2.pdb /DBUILDING_DTOOLCONFIG /EHa /Zm300 /DWIN32_VC /DWIN32 /W3 dtool/src/interrogatedb/p3interrogatedb_composite2.cxx
Build terminated.
The 3.3 ref indicates that PyUnicode_AsUTF8 is new in that version so I guess a build with anything less than 3.3 wouldn’t be possible at this point. I’ll wait to see if enn0x has any success building with 3.3 before I try anything more, but I can continue working through the direct tree making changes.
@enn0x: rdb has indicated that creating a direct tree that is compatible with both versions of Python would be preferable so using 2to3 straight up won’t be possible for all the changes, though I am using its output as a reference.
@rdb: Unfortunately I can’t try a 64-bit build right now because I’m still using a fairly old XP box, hopefully that will be changing in a couple of months though.
Hmm, yes, I used some 3.3-specific functions in there. I’ll see what I can do about eliminating them. 3.2 has a serious lack functions to easily encode a string for command-line output, though.
I can confirm the MemoryError when importing any panda3d lib (compiled with Python 3.3/32bit, VS2010, current head, on win7/64bit). When importing the extension lib more than 0.5 gig of memory get allocated, and there is probably an uncaught OutOfMemory exception). I try to dig deeper.
Hi rdb, I have a script ready that updates the panda python code base to be compatible with both python versions. I’ve tested the results against my 2.7 installation and have managed to get the most basic features working. However before I try any significant testing I wanted to confirm the plan with you.
The general consensus of the python community is that maintaining a codebase compatible with both versions is the most problematic / buggy solution. Maybe the user could select a version of python to use during the panda installation and the script could then update the code base if necessary. It wouldn’t need to worry about maintaining compatibility in this case which might be a more stable solution. I don’t know panda well enough to know if this is feasible but I thought I’d throw it out there. I have no preference either way, but just want to confirm with you because the first approach will take a lot more testing, though I’m confident it can be done.
Which leads into my next question: are there anything like unit tests available for panda? Or if not what would be the best approach to thoroughly testing functionality? There’s a lot of panda that I haven’t even gone near in my own personal use so I wouldn’t be the best person for final testing. If you want a copy let me know, you could maybe try it on your 3.3 (64bit) build. The script has an option to leave out the compatibility hacks and just update solely to python 3 so you would at least have a python 3 code base to test your build on.
Thanks a lot for your work! I’m not sure what the best approach is here, hmm. I hadn’t considered using a script yet. I’m a bit worried that the script will go out of date quickly as the Panda source is modified, though. Does the script make a lot of specific changes?
We don’t currently have a framework set up for unit testing, but I think that it would be a good idea to have one. I have no experience in setting up unit tests for a project like Panda though, so perhaps I would not be the right person to set that up.
It doesn’t actually make any panda specific changes if this is what you mean. lib2to3 does most of the work, I simply tap into its interface and add a second layer of changes for cases specific to compatibility issues between python 2 and 3 (since lib2to3 isn’t intended for this.) The panda codebase can be maintained and modified in python 2 with the script being used to generate python 3 snapshots on the fly. Once the script itself is ironed out and working, changes to panda source shouldn’t introduce any new issues (unless someone uses a particularly archaic language feature.)
It logs every change so any errors introduced can be found quickly. That’s basically how I’ve been testing it: working to failure, finding the change that made the failure, and then making a custom fix for that case to make it work in both versions of python. When the script is rerun it takes care of all those cases wherever it finds them. Library name changes would be an example:
# this
from StringIO import StringIO
# becomes:
if PY_VER < 3: from StringIO import StringIO
else: from io import StringIO
It makes the resulting code a bit messier, but doesn’t introduce any errors by itself. I’ll just continue testing the results against my 2.7 panda using as much functionality as I can figure out. I think some kind of unit testing would be essential in the end though… python has a large unit testing library so I’ll start looking into that when/if it becomes relevant. Here’s some stats output from a basic run over the direct, pandac, and examples folders, it shows how many changes it actually makes:
Hmm, that sounds good. Could you send the script over? I can add it to the Panda source, and anyone who wants to go out of his way to use Python 3 can run that script after building and installing Panda3D.
Perhaps I can incorporate it into makepanda, so that when building against Python 3 it would automatically convert the tree in built/direct.
If you have any suggestions for setting up unit testing in the Panda3D codebase, I’d love to hear your ideas.
If either of these approaches are used then it wouldn’t be necessary for the resulting code that comes out of the script to be backward compatible to python 2 would it? It should be enough to have it work only with python 3 since the user would be building it to that version anyways… I only ask because this would make the script a lot simpler than if it has to produce backward compatible results. There would be no more need for things like the “import” hack that I showed in my last post.
I can send you a copy but I haven’t been able to do any testing of the results on a python 3 build of panda given that memory error problem with the previous build attempt, so I’m not sure how useful it would be to you at this point. Maybe you could test it with your 64 bit build?
I can look into unit testing and see what I can come up with since it’s a natural follow up to this project; it just might be a bit longer term. A good way to get to know panda inside-out I guess.
Right, that’s true. There’s not much point in keeping compatibility then; when someone builds Panda3D for Python 3, it will only have to work with Python 3.
I don’t have a working build at the moment, since even my 64-bit build crashes due to a different issue. I’m currently in the middle of releasing 1.8.1, and next week I plan on participating in PyWeek, so I’m afraid I won’t have much time to look into it soon.
If you have any questions then please don’t hesitate to ask.
Ok, this approach should make things a lot simpler. I’ll get back to you in a couple of weeks then to see if there has been any change on the status of a Py 3 build. There’s really not much point in sending you the script until then, I guess, since I’m going to be taking a bunch of the compatibility stuff out of it anyways. There’s no rush on my end obviously but I’ll do the testing whenever a build is ready.
I’m going to try see what I can do about developing that panda unit testing framework, though I’m essentially starting from scratch and I’m going to be splitting time with my own panda project, so we’ll see how it goes; I guess I’ll start another thread whenever I have something.
I presume you’re going to be using something based on panda for the PyWeek challenge?