Migration to Python 3

[cslos77]

Hi this is mostly for rdb regarding an attempt to migrate Panda to Python 3, but if anyone has something useful to add feel free obviously.

I managed to get all the installation stuff set up correctly using the latest CVS and replacing the python 2 dist with python 3.3 in the "win-python" folder. I tried 3.2 first but the python installer refused to do anything but offer to alter my existing installation, so I went with 3.3. If this is a problem I'll just swap them around before I start any serious work on this.

I tried building with "makepanda\makepanda.bat --everything --installer" just to see how far it would get. The first attempt crashed with Maya2010 being the culprit. Adding " --no-maya2010" got me through the Panda build part, but predictably it failed when it got to the python code generation stage:

Code: Select all

[ 92%] Generating 'pandac' tree
Traceback (most recent call last):
  File "direct\src\ffi\jGenPyCode.py", line 74, in <module>
    from direct.ffi import DoGenPyCode
ImportError: cannot import name DoGenPyCode


What this unhelpful error msg is actually trying to say is that python 3 enforces absolute imports when using "import" directly, so for example in DirectNotifyGlobal.py (which is actually where the first error occurs, not jGenPyCode.py):

Code: Select all

import DirectNotify  # fails.

# Must be absolute:
import direct.directnotify.DirectNotify as DirectNotify  # works.


I went through and fixed a few by hand just to see how far I'd get and I eventually began to run into some other errors related to version as expected. I guess this would be my starting point then? But I'm a bit confused as to what code is auto generated and what code I would be working with exactly. I'm presuming that altering the direct tree directly would be pointless since the build process would just regenerate it? I also notice that there is a src folder in the direct tree which duplicates the code which is what the compiler actually invokes. I didn't really look into this that much, but maybe you could point me in the right direction here. Thanks.

[rdb]

There is little to no auto-generated Python code. The underlying C++ code is in libpandaexpress.dll, libpanda.dll, and more of these libraries, which are loaded by panda3d.py (for new-style imports) or pandac.PandaModules (for old-style imports).

So, I'd say you're on the right path. built/direct is really just a copy of direct/src.

Loading the libpanda library crashed for me last time I tried (although libpandaexpress and using its classes worked fine), which means that you'll probably run into this when you get to the pandac.PandaModules or panda3d.core imports. I have not had the time to track this down yet. However, you should be able to get quite far without this already.

[cslos77]

I was able to make quite a bit of progress in my first run of actually trying to update some code. I just stepped in where the compiler failed "direct/ffi/jGenPyCode.py" and ran it from the console alternately using python 3.3 and 2.7, fixing issues as I encountered them to see how far I could get. My first impression is that it is not going to be as difficult as I originally thought, at least on the python side of things. A fairly sizable file like PythonUtil.py (4500 lines) only took about 30 minutes or so before it was working with both python versions (at least its initialization functionality; no syntax, import errors etc.). Also the callable issue is no longer a concern as they've re-added it to the language since version 3.2.

There were some things like the re-structuring of the xml and html modules in 3 that required hacks like:

Code: Select all

if sys.version_info[0] < 3: from HTMLParser import HTMLParser # Py2
else: from html.parser import HTMLParser  # Py3


but most of it should be fairly clean one line substitutions that work in both versions. The first real problem I encountered was the first time imp.load_dynamic is invoked in "extension_native_helpers.py" to get the libpandaexpress.dll. In 2.7 this obviously works fine but in 3.3 it crashes with:

Code: Select all

File "C:\panda3d\direct\extensions_native\extension_native_helpers.py", line 81, in <module>
    Dtool_PreloadDLL("libpandaexpress")
  File "C:\panda3d\direct\extensions_native\extension_native_helpers.py", line 79, in Dtool_PreloadDLL
    imp.load_dynamic(module, pathname)
MemoryError


You mentioned you were able to get at least libpandexpress working in your attempts so I'm assuming this isn't the crash you were referring to? I watched the memory hit as it ran before it crashed and it didn't even come close to maxing out my memory so I'm not sure what's happening here. I checked the 3.3 ref and the "load_dynamic" function is no longer listed in the imp section and a lot of the other functionality is deprecated redirecting the user to importlib, but I couldn't see anything there that resembled load_dynamic. Either way I should be able to continue working through other branches of the direct tree and start making the syntax changes and such whatever the problem turns out to be here.

[rdb]

Good to hear you're making progress.

I haven't encountered the MemoryError myself. I compiled with Python 3.3 myself, but I admit I haven't tried pandac.PandaModules, only panda3d.core and a direct import. Copying and renaming libpandaexpress.dll to libpandaexpress.pyd should make it importable directly, can you try this followed by "from libpandaexpress import *" to see if this yields the same error?

[cslos77]

Using the .pyd extension gives the same errors, I tried various combinations:

Code: Select all

from libpandaexpress import *
import libpandaexpress
from libpandaexpress import Buffer


with the same results. I tried the same thing with other libpanda*.pyd files, and even smaller libraries like libpandaskel.pyd produced the same thing. I'm guessing this is a problem on my end then since you haven't encountered it; 2.7 has no problem with these tests so its obviously in my 3.3 build somewhere. I'll try messing around with some empty scripts later tonight to see if I can get any imports to work at all. This is the system error report it dumps into the Win temp file if it's of any use:

Code: Select all

<DATABASE>
<EXE
NAME="python.exe"
FILTER="GRABMI_FILTER_PRIVACY">




<MATCHING_FILE
NAME="msvcr100.dll"
SIZE="768848"
CHECKSUM="0xDA0DAA00"
BIN_FILE_VERSION="10.0.30319.460"
BIN_PRODUCT_VERSION="10.0.30319.460"
PRODUCT_VERSION="10.00.30319.460"
FILE_DESCRIPTION="Microsoft«
C
Runtime
Library"
COMPANY_NAME="Microsoft
Corporation"
PRODUCT_NAME="Microsoft«
Visual
Studio«
2010"
FILE_VERSION="10.00.30319.460"
ORIGINAL_FILENAME="msvcr100_clr0400.dll"
INTERNAL_NAME="msvcr100_clr0400.dll"
LEGAL_COPYRIGHT="⌐
Microsoft
Corporation.

All
rights
reserved."
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x40004"
VERFILETYPE="0x2"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0xC94C9"
LINKER_VERSION="0xA0000"
UPTO_BIN_FILE_VERSION="10.0.30319.460"
UPTO_BIN_PRODUCT_VERSION="10.0.30319.460"
LINK_DATE="04/22/2011
07:59:50"
UPTO_LINK_DATE="04/22/2011
07:59:50"
VER_LANGUAGE="English
(United
States)
[0x409]"
/>




<MATCHING_FILE
NAME="py.exe"
SIZE="93696"
CHECKSUM="0xC6281BDD"
BIN_FILE_VERSION="3.3.150.1013"
BIN_PRODUCT_VERSION="3.3.150.1013"
PRODUCT_VERSION="3.3.0"
FILE_DESCRIPTION="Python
Launcher
for
Windows
(Console)"
COMPANY_NAME="Python
Software
Foundation"
PRODUCT_NAME="Python
Launcher
for
Windows"
FILE_VERSION="3.3.0"
ORIGINAL_FILENAME="py"
INTERNAL_NAME="py"
LEGAL_COPYRIGHT="Copyright
(C)
2011-2012
Python
Software
Foundation"
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x4"
VERFILETYPE="0x1"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x220F4"
LINKER_VERSION="0x0"
UPTO_BIN_FILE_VERSION="3.3.150.1013"
UPTO_BIN_PRODUCT_VERSION="3.3.150.1013"
LINK_DATE="09/29/2012
08:54:51"
UPTO_LINK_DATE="09/29/2012
08:54:51"
VER_LANGUAGE="English
(United
Kingdom)
[0x809]"
/>




<MATCHING_FILE
NAME="python.exe"
SIZE="26624"
CHECKSUM="0x40F3D4E"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0xA685"
LINKER_VERSION="0x0"
LINK_DATE="09/29/2012
08:56:02"
UPTO_LINK_DATE="09/29/2012
08:56:02"
/>




<MATCHING_FILE
NAME="python33.dll"
SIZE="2641408"
CHECKSUM="0x7EC52D86"
BIN_FILE_VERSION="3.3.150.1013"
BIN_PRODUCT_VERSION="3.3.150.1013"
PRODUCT_VERSION="3.3.0"
FILE_DESCRIPTION="Python
Core"
COMPANY_NAME="Python
Software
Foundation"
PRODUCT_NAME="Python"
FILE_VERSION="3.3.0"
ORIGINAL_FILENAME="python33.dll"
INTERNAL_NAME="Python
DLL"
LEGAL_COPYRIGHT="Copyright
⌐
2001-2012
Python
Software
Foundation.
Copyright
⌐
2000
BeOpen.com.
Copyright
⌐
1995-2001
CNRI.
Copyright
⌐
1991-1995
SMC."
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x40004"
VERFILETYPE="0x1"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x287AD3"
LINKER_VERSION="0x0"
UPTO_BIN_FILE_VERSION="3.3.150.1013"
UPTO_BIN_PRODUCT_VERSION="3.3.150.1013"
LINK_DATE="09/29/2012
08:55:52"
UPTO_LINK_DATE="09/29/2012
08:55:52"
VER_LANGUAGE="Language
Neutral
[0x0]"
/>




<MATCHING_FILE
NAME="pythonw.exe"
SIZE="27136"
CHECKSUM="0x8E956FF"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x11720"
LINKER_VERSION="0x0"
LINK_DATE="09/29/2012
08:56:12"
UPTO_LINK_DATE="09/29/2012
08:56:12"
/>




<MATCHING_FILE
NAME="pyw.exe"
SIZE="94208"
CHECKSUM="0xD4E3226F"
BIN_FILE_VERSION="3.3.123.1013"
BIN_PRODUCT_VERSION="3.3.123.1013"
PRODUCT_VERSION="3.3.0"
FILE_DESCRIPTION="Python
Launcher
for
Windows
(Console)"
COMPANY_NAME="Python
Software
Foundation"
PRODUCT_NAME="Python
Launcher
for
Windows"
FILE_VERSION="3.3.0"
ORIGINAL_FILENAME="py"
INTERNAL_NAME="py"
LEGAL_COPYRIGHT="Copyright
(C)
2011-2012
Python
Software
Foundation"
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x4"
VERFILETYPE="0x1"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x1A9BA"
LINKER_VERSION="0x0"
UPTO_BIN_FILE_VERSION="3.3.123.1013"
UPTO_BIN_PRODUCT_VERSION="3.3.123.1013"
LINK_DATE="09/29/2012
08:54:48"
UPTO_LINK_DATE="09/29/2012
08:54:48"
VER_LANGUAGE="English
(United
Kingdom)
[0x809]"
/>




<MATCHING_FILE
NAME="w9xpopen.exe"
SIZE="43008"
CHECKSUM="0x9A6D4C01"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x19DC4"
LINKER_VERSION="0x0"
LINK_DATE="09/29/2012
08:54:44"
UPTO_LINK_DATE="09/29/2012
08:54:44"
/>




<MATCHING_FILE
NAME="DLLs\python3.dll"
SIZE="73216"
CHECKSUM="0xED22442B"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x0"
LINKER_VERSION="0x0"
LINK_DATE="09/29/2012
08:54:44"
UPTO_LINK_DATE="09/29/2012
08:54:44"
/>




<MATCHING_FILE
NAME="DLLs\sqlite3.dll"
SIZE="466944"
CHECKSUM="0x3DFFD454"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x74688"
LINKER_VERSION="0x0"
LINK_DATE="09/29/2012
08:54:55"
UPTO_LINK_DATE="09/29/2012
08:54:55"
/>




<MATCHING_FILE
NAME="DLLs      cl85.dll"
SIZE="925696"
CHECKSUM="0xAD7D4390"
BIN_FILE_VERSION="8.5.2.11"
BIN_PRODUCT_VERSION="8.5.2.11"
PRODUCT_VERSION="8.5.11"
FILE_DESCRIPTION="Tcl
DLL"
COMPANY_NAME="ActiveState
Corporation"
PRODUCT_NAME="Tcl
8.5
for
Windows"
FILE_VERSION="8.5.11"
ORIGINAL_FILENAME="tcl85.dll"
LEGAL_COPYRIGHT="Copyright
⌐
2001
by
ActiveState
Corporation,
et
al"
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x4"
VERFILETYPE="0x2"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0xE5092"
LINKER_VERSION="0x0"
UPTO_BIN_FILE_VERSION="8.5.2.11"
UPTO_BIN_PRODUCT_VERSION="8.5.2.11"
LINK_DATE="05/31/2012
18:54:30"
UPTO_LINK_DATE="05/31/2012
18:54:30"
VER_LANGUAGE="English
(United
States)
[0x409]"
/>




<MATCHING_FILE
NAME="DLLs      clpip85.dll"
SIZE="7680"
CHECKSUM="0xEE0F701A"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0xB102"
LINKER_VERSION="0x0"
LINK_DATE="05/31/2012
18:54:30"
UPTO_LINK_DATE="05/31/2012
18:54:30"
/>




<MATCHING_FILE
NAME="DLLs      k85.dll"
SIZE="1346560"
CHECKSUM="0x162BB769"
BIN_FILE_VERSION="8.5.2.11"
BIN_PRODUCT_VERSION="8.5.2.11"
PRODUCT_VERSION="8.5.11"
FILE_DESCRIPTION="Tk
DLL"
COMPANY_NAME="ActiveState
Corporation"
PRODUCT_NAME="Tk
8.5
for
Windows"
FILE_VERSION="8.5.11"
ORIGINAL_FILENAME="tk85.dll"
LEGAL_COPYRIGHT="Copyright
⌐
2001
by
ActiveState
Corporation,
et
al"
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x4"
VERFILETYPE="0x2"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x14A85E"
LINKER_VERSION="0x0"
UPTO_BIN_FILE_VERSION="8.5.2.11"
UPTO_BIN_PRODUCT_VERSION="8.5.2.11"
LINK_DATE="05/31/2012
18:54:52"
UPTO_LINK_DATE="05/31/2012
18:54:52"
VER_LANGUAGE="English
(United
States)
[0x409]"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-10.0-amd64.exe"
SIZE="222208"
CHECKSUM="0x141F146C"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x381F2"
LINKER_VERSION="0x0"
LINK_DATE="05/13/2012
16:06:49"
UPTO_LINK_DATE="05/13/2012
16:06:49"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-10.0.exe"
SIZE="190976"
CHECKSUM="0x46A86445"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x39967"
LINKER_VERSION="0x0"
LINK_DATE="04/04/2012
02:16:07"
UPTO_LINK_DATE="04/04/2012
02:16:07"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-6.0.exe"
SIZE="61440"
CHECKSUM="0xE589B8AD"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x0"
LINKER_VERSION="0x0"
LINK_DATE="05/31/2008
04:52:45"
UPTO_LINK_DATE="05/31/2008
04:52:45"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-7.1.exe"
SIZE="65536"
CHECKSUM="0xA2833DFD"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x0"
LINKER_VERSION="0x0"
LINK_DATE="05/31/2008
04:53:42"
UPTO_LINK_DATE="05/31/2008
04:53:42"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-8.0.exe"
SIZE="61440"
CHECKSUM="0x8527B654"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x1701B"
LINKER_VERSION="0x0"
LINK_DATE="10/04/2006
15:16:27"
UPTO_LINK_DATE="10/04/2006
15:16:27"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-9.0-amd64.exe"
SIZE="224256"
CHECKSUM="0x2C21FF43"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x3C5B1"
LINKER_VERSION="0x0"
LINK_DATE="01/29/2009
13:03:06"
UPTO_LINK_DATE="01/29/2009
13:03:06"
/>




<MATCHING_FILE
NAME="Lib\distutils\command\wininst-9.0.exe"
SIZE="196096"
CHECKSUM="0x4C294F6E"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x3C5CF"
LINKER_VERSION="0x0"
LINK_DATE="01/29/2009
13:02:55"
UPTO_LINK_DATE="01/29/2009
13:02:55"
/>




<MATCHING_FILE
NAME="tcl\dde1.3        cldde13.dll"
SIZE="18432"
CHECKSUM="0x9B1CE45B"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x4C5E"
LINKER_VERSION="0x0"
LINK_DATE="05/31/2012
18:54:31"
UPTO_LINK_DATE="05/31/2012
18:54:31"
/>




<MATCHING_FILE
eg1.2   clreg12.dll"
SIZE="17408"
CHECKSUM="0xD32D68B6"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0xCFEA"
LINKER_VERSION="0x0"
LINK_DATE="05/31/2012
18:54:30"
UPTO_LINK_DATE="05/31/2012
18:54:30"
/>




<MATCHING_FILE
NAME="tcl       ix8.4.3 ix84.dll"
SIZE="262144"
CHECKSUM="0x5E6172C"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x42FD5"
LINKER_VERSION="0x0"
LINK_DATE="05/31/2012
19:08:51"
UPTO_LINK_DATE="05/31/2012
19:08:51"
/>
</EXE>
<EXE
NAME="python33.dll"
FILTER="GRABMI_FILTER_THISFILEONLY">




<MATCHING_FILE
NAME="python33.dll"
SIZE="2641408"
CHECKSUM="0x7EC52D86"
BIN_FILE_VERSION="3.3.150.1013"
BIN_PRODUCT_VERSION="3.3.150.1013"
PRODUCT_VERSION="3.3.0"
FILE_DESCRIPTION="Python
Core"
COMPANY_NAME="Python
Software
Foundation"
PRODUCT_NAME="Python"
FILE_VERSION="3.3.0"
ORIGINAL_FILENAME="python33.dll"
INTERNAL_NAME="Python
DLL"
LEGAL_COPYRIGHT="Copyright
⌐
2001-2012
Python
Software
Foundation.
Copyright
⌐
2000
BeOpen.com.
Copyright
⌐
1995-2001
CNRI.
Copyright
⌐
1991-1995
SMC."
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x40004"
VERFILETYPE="0x1"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0x287AD3"
LINKER_VERSION="0x0"
UPTO_BIN_FILE_VERSION="3.3.150.1013"
UPTO_BIN_PRODUCT_VERSION="3.3.150.1013"
LINK_DATE="09/29/2012
08:55:52"
UPTO_LINK_DATE="09/29/2012
08:55:52"
VER_LANGUAGE="Language
Neutral
[0x0]"
/>
</EXE>
<EXE
NAME="kernel32.dll"
FILTER="GRABMI_FILTER_THISFILEONLY">




<MATCHING_FILE
NAME="kernel32.dll"
SIZE="990208"
CHECKSUM="0xCC2C4544"
BIN_FILE_VERSION="5.1.2600.6293"
BIN_PRODUCT_VERSION="5.1.2600.6293"
PRODUCT_VERSION="5.1.2600.6293"
FILE_DESCRIPTION="Windows
NT
BASE
API
Client
DLL"
COMPANY_NAME="Microsoft
Corporation"
PRODUCT_NAME="Microsoft«
Windows«
Operating
System"
FILE_VERSION="5.1.2600.6293
(xpsp_sp3_gdr.121001-1622)"
ORIGINAL_FILENAME="kernel32"
INTERNAL_NAME="kernel32"
LEGAL_COPYRIGHT="⌐
Microsoft
Corporation.
All
rights
reserved."
VERFILEDATEHI="0x0"
VERFILEDATELO="0x0"
VERFILEOS="0x40004"
VERFILETYPE="0x2"
MODULE_TYPE="WIN32"
PE_CHECKSUM="0xFBCBC"
LINKER_VERSION="0x50001"
UPTO_BIN_FILE_VERSION="5.1.2600.6293"
UPTO_BIN_PRODUCT_VERSION="5.1.2600.6293"
LINK_DATE="10/03/2012
04:58:13"
UPTO_LINK_DATE="10/03/2012
04:58:13"
VER_LANGUAGE="English
(United
States)
[0x409]"
/>
</EXE>
</DATABASE>


[rdb]

Ah, you compiled with Visual Studio 2010? We don't currently support it, so the crash may be related to that. Do Panda tools such as pview even run correctly?

[cslos77]

Turns out I had an old install of 2010 still kicking around so I thought that would be the problem. I uninstalled 2010 and scrubbed all traces of it from my system and then re-installed 2008 only to encounter the same error again with the report still saying it was built with Visual Studio 2010. I was pretty thorough in erasing all traces of Visual Studio (both versions) from my system before re-installing 2008 (even got rid of all the .NET installations), but it's still saying it's building from 2010 somehow, any thoughts on this?

The only thing that I can think of is that it might have something to do with MS no longer distributing 2008 publicly (all links just take you to 2012). I had to search around on some forums to find the an old link buried on an MS server to even get my hands on 2008, but I honestly can't see how this would make a difference. pview and other panda tools run fine, btw.

[enn0x]

Python versions prior to 3.3 use VS2008, but Python 3.3 is compiled with VS2010. So actually you might run into problems with VS2008. I use VS2010 for more than two years now to compile Panda3D, and had no problems so far. There is just a tiny modification needed so makepanda will find the Win7 SDK.

I will try and compile with Python 3.3 one of the next days.

By the way: are you using 2to3 for the Panda3D Python code? Updating by hand is - in my opinion - way too error prone.

[rdb]

Hmm, perhaps. I compiled using VS2008, but I use a different set of thirdparty packages, and I used a 64-bit build. Maybe I should share the thirdparty packages that I used, so that you can try making a 64-bits build like I did?

[cslos77]

Just to see what would happen I tried a build with 3.2 but it fails fairly quickly with:

Code: Select all

c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(211) : error C3861: 'PyUnicode_AsUTF8': identifier not found
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(223) : error C3861: 'PyUnicode_AsUTF8': identifier not found
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(253) : error C3861: 'PyUnicode_AsUTF8': identifier not found
c:\panda3d\dtool\src\interrogatedb\py_panda.cxx(265) : error C3861: 'PyUnicode_AsUTF8': identifier not found
Storing dependency cache.
Elapsed Time: 1 min 5 sec
The following command returned a non-zero value: cl /wd4996 /wd4275 /wd4267 /wd4101 /wd4273 /Fobuilt/tmp/p3interrogatedb_composite2.obj /nologo /c /arch:SSE2 /Idtool/src/interrogatedb /Ibuilt/tmp /Ibuilt/include /Ithirdparty/win-python/include /DMAKEPANDA= /MD /Zi /O2 /Ob2 /Oi /Ot /fp:fast /DFORCE_INLINING /Fdbuilt/tmp/p3interrogatedb_composite2.pdb /DBUILDING_DTOOLCONFIG /EHa /Zm300 /DWIN32_VC /DWIN32 /W3 dtool/src/interrogatedb/p3interrogatedb_composite2.cxx
Build terminated.


The 3.3 ref indicates that PyUnicode_AsUTF8 is new in that version so I guess a build with anything less than 3.3 wouldn't be possible at this point. I'll wait to see if enn0x has any success building with 3.3 before I try anything more, but I can continue working through the direct tree making changes.

@enn0x: rdb has indicated that creating a direct tree that is compatible with both versions of Python would be preferable so using 2to3 straight up won't be possible for all the changes, though I am using its output as a reference.

@rdb: Unfortunately I can't try a 64-bit build right now because I'm still using a fairly old XP box, hopefully that will be changing in a couple of months though.

[rdb]

Hmm, yes, I used some 3.3-specific functions in there. I'll see what I can do about eliminating them. 3.2 has a serious lack functions to easily encode a string for command-line output, though.

[enn0x]

I can confirm the MemoryError when importing any panda3d lib (compiled with Python 3.3/32bit, VS2010, current head, on win7/64bit). When importing the extension lib more than 0.5 gig of memory get allocated, and there is probably an uncaught OutOfMemory exception). I try to dig deeper.

[cslos77]

Hi rdb, I have a script ready that updates the panda python code base to be compatible with both python versions. I've tested the results against my 2.7 installation and have managed to get the most basic features working. However before I try any significant testing I wanted to confirm the plan with you.

The general consensus of the python community is that maintaining a codebase compatible with both versions is the most problematic / buggy solution. Maybe the user could select a version of python to use during the panda installation and the script could then update the code base if necessary. It wouldn't need to worry about maintaining compatibility in this case which might be a more stable solution. I don't know panda well enough to know if this is feasible but I thought I'd throw it out there. I have no preference either way, but just want to confirm with you because the first approach will take a lot more testing, though I'm confident it can be done.

Which leads into my next question: are there anything like unit tests available for panda? Or if not what would be the best approach to thoroughly testing functionality? There's a lot of panda that I haven't even gone near in my own personal use so I wouldn't be the best person for final testing. If you want a copy let me know, you could maybe try it on your 3.3 (64bit) build. The script has an option to leave out the compatibility hacks and just update solely to python 3 so you would at least have a python 3 code base to test your build on.

[rdb]

Thanks a lot for your work! I'm not sure what the best approach is here, hmm. I hadn't considered using a script yet. I'm a bit worried that the script will go out of date quickly as the Panda source is modified, though. Does the script make a lot of specific changes?

We don't currently have a framework set up for unit testing, but I think that it would be a good idea to have one. I have no experience in setting up unit tests for a project like Panda though, so perhaps I would not be the right person to set that up.

[cslos77]

I hadn't considered using a script yet. I'm a bit worried that the script will go out of date quickly as the Panda source is modified, though. Does the script make a lot of specific changes?

It doesn't actually make any panda specific changes if this is what you mean. lib2to3 does most of the work, I simply tap into its interface and add a second layer of changes for cases specific to compatibility issues between python 2 and 3 (since lib2to3 isn't intended for this.) The panda codebase can be maintained and modified in python 2 with the script being used to generate python 3 snapshots on the fly. Once the script itself is ironed out and working, changes to panda source shouldn't introduce any new issues (unless someone uses a particularly archaic language feature.)

It logs every change so any errors introduced can be found quickly. That's basically how I've been testing it: working to failure, finding the change that made the failure, and then making a custom fix for that case to make it work in both versions of python. When the script is rerun it takes care of all those cases wherever it finds them. Library name changes would be an example:

Code: Select all

# this
from StringIO import StringIO

# becomes:
if PY_VER < 3: from StringIO import StringIO
else: from io import StringIO


It makes the resulting code a bit messier, but doesn't introduce any errors by itself. I'll just continue testing the results against my 2.7 panda using as much functionality as I can figure out. I think some kind of unit testing would be essential in the end though... python has a large unit testing library so I'll start looking into that when/if it becomes relevant. Here's some stats output from a basic run over the direct, pandac, and examples folders, it shows how many changes it actually makes:

Code: Select all

panda_to_py3: 163.796875 seconds

dirs:         104
files:        460
lines read:   130033
lines fixed:  3258
lines added:  417


[rdb]

Hmm, that sounds good. Could you send the script over? I can add it to the Panda source, and anyone who wants to go out of his way to use Python 3 can run that script after building and installing Panda3D.

Perhaps I can incorporate it into makepanda, so that when building against Python 3 it would automatically convert the tree in built/direct.

If you have any suggestions for setting up unit testing in the Panda3D codebase, I'd love to hear your ideas.

[cslos77]

I can add it to the Panda source, and anyone who wants to go out of his way to use Python 3 can run that script after building and installing Panda3D.

Perhaps I can incorporate it into makepanda, so that when building against Python 3 it would automatically convert the tree in built/direct.

If either of these approaches are used then it wouldn't be necessary for the resulting code that comes out of the script to be backward compatible to python 2 would it? It should be enough to have it work only with python 3 since the user would be building it to that version anyways... I only ask because this would make the script a lot simpler than if it has to produce backward compatible results. There would be no more need for things like the "import" hack that I showed in my last post.

I can send you a copy but I haven't been able to do any testing of the results on a python 3 build of panda given that memory error problem with the previous build attempt, so I'm not sure how useful it would be to you at this point. Maybe you could test it with your 64 bit build?

I can look into unit testing and see what I can come up with since it's a natural follow up to this project; it just might be a bit longer term. A good way to get to know panda inside-out I guess.

[rdb]

Right, that's true. There's not much point in keeping compatibility then; when someone builds Panda3D for Python 3, it will only have to work with Python 3.

I don't have a working build at the moment, since even my 64-bit build crashes due to a different issue. I'm currently in the middle of releasing 1.8.1, and next week I plan on participating in PyWeek, so I'm afraid I won't have much time to look into it soon.

If you have any questions then please don't hesitate to ask.

[cslos77]

Ok, this approach should make things a lot simpler. I'll get back to you in a couple of weeks then to see if there has been any change on the status of a Py 3 build. There's really not much point in sending you the script until then, I guess, since I'm going to be taking a bunch of the compatibility stuff out of it anyways. There's no rush on my end obviously but I'll do the testing whenever a build is ready.

I'm going to try see what I can do about developing that panda unit testing framework, though I'm essentially starting from scratch and I'm going to be splitting time with my own panda project, so we'll see how it goes; I guess I'll start another thread whenever I have something.

I presume you're going to be using something based on panda for the PyWeek challenge?

[rdb]

Thanks!

Yes, we will be using Panda3D.

[cslos77]

Hi rdb, just checking in to see if there has been any change in the status of a python3 build for panada. I'll have a script ready to test whenever there is anything working. No rush on my end but I thought I'd see how things are going.

[rdb]

I've been able to work around the segfault. It only occurs when I import libpandaexpress before libpanda, and not the other way around. Which baffles me, as libpandaexpress is a dependency of libpanda. Since I don't have a debugging environment set up on the laptop that I do the Python 3 work on, I have no means to see what's going on. But here's some code that works for me:

Code: Select all

import panda3d
mgr = panda3d.__manager__
mgr.libimport('libpanda')
mgr.libimport('libpandaexpress')
print("Imported")

from libpandaexpress import *
from libpanda import *

Presumably, simply swapping the libraries around in the definition for the 'core' module in panda3d.py will also work, but I won't do that until I've found out why this happens. At some point in the future, I plan on doing a complete overhaul of the way we compile the Python bindings into the libraries, which may solve some of the problems we're encountering, but it may not make much sense to wait for that unless I find time to get around to it soon.

I'm still getting another segfault, but it happens at shutdown, so I suppose it's less important.

The next step for me would probably be to integrate the py2to3-based tool into makepanda. Can you send the script so that I can try it for myself?

[cslos77]

Here's the script, sorry for the length but I don't have any host to use right now:

Code: Select all

# ============
# panda_to_py3
# ============

__version__ = "0.3"

import sys
import os
import shutil
from datetime import datetime
from lib2to3.refactor import *
from lib2to3.main import *

# Options.
LOGGING = True
PRINTING = True
BACK_UP = True
if "--no-log" in sys.argv: LOGGING = False
if "--no-backup" in sys.argv: BACK_UP = False
if "--no-print" in sys.argv: PRINTING = False


class Custom_Fixes:

    def panda_fix_builtin_ref(line):
        """Fix refs to __builtin__ that py2to3 misses."""
        fixed_line = line.replace("__builtin__", "builtins")
        return fixed_line

    # Map str patterns to custom fix methods.
    fix_map = {"__builtin__":panda_fix_builtin_ref,}


class Panda3D_Refactoring_Tool(StdoutRefactoringTool):

    def __init__(self, src_dir):
        """Custom Panda3d py2to3 refactoring tool."""
        fixers = sorted(refactor.get_fixers_from_package("lib2to3.fixes"))
        StdoutRefactoringTool.__init__(self, fixers, [],
                                       [], True, None,
                                       input_base_dir=src_dir)
                                       
    def refactor_panda_file(self, src_file):
        """Perform py2to3 conversion on panda file "src_file"."""
        global file_count, line_count, lines_fixed
        _file_lines_fixed = lines_fixed
        _file_lines_added = lines_added
       
        # Extract lines from src_file for stats and testing.
        with open(src_file) as file:
            lines = file.readlines()
        line_count += len(lines)
        file_count += 1
       
        # Refactor (test for indents by line.)
        for line in lines:
            if line == "\n": continue
            if line.startswith(" ") or line.startswith("\t"):
                # Some files begin indented; use workaround.
                self.handle_parse_error(src_file, "in")
                self.refactor_file(src_file, write=True)
                self.handle_parse_error(src_file, "out")
            else:
                self.refactor_file(src_file, write=True)
            break

        # Print file refactor info.
        if PRINTING:
            file_name = os.path.split(src_file)[-1]
            f_lines_fixed = lines_fixed - _file_lines_fixed
            f_lines_added = lines_added - _file_lines_added
            if f_lines_fixed > 1: f_str = "lines fixed"
            else: f_str = "line fixed"
            if f_lines_added > 1: a_str = "lines added"
            else: a_str = "line added"
            if f_lines_added:
                f_line_str = "({} {}, {} {})".format(f_lines_fixed, f_str,
                                                     f_lines_added, a_str)
            else:
                f_line_str = "({} lines fixed)".format(f_lines_fixed)
            print("  {:<30}{}".format(file_name, f_line_str))
       
    def handle_parse_error(self, file_path, mode):
        """Allow py2to3 to handle files in direct/extensions folder that
        begin with indents by putting a temporary "class Temp:" statement
        at the top of the file to fool the parser. Remove it after parse."""
        with open(file_path, "r") as file:
            lines = file.readlines()
            if mode == "in":
                lines.insert(0, "class Temp:")  # Temp class statement.
            elif mode == "out":
                lines.pop(0)  # Remove temp class statement in "out" mode.
                lines.insert(0, "\n")
        with open(file_path, "w") as file:
            file.writelines(lines)
                                       
    def print_output(self, old, new, filename, equal):
        """Override method in StdoutRefactoringTool so that we can
        set up custom log output as well as perform a second layer
        of custom fixes to cover things that py2to3 misses."""
        global lines_fixed, lines_added, custom_fixes
       
        if LOGGING:
            # Start new set of log lines for each file.
            dec_str = "".zfill(len(filename)).replace("0", "-")
            log_lines.extend(["".join(["\n", dec_str]),
                              filename.replace(".\\", ""),
                              "".join([dec_str, "\n"])])
        # Handle custom fixes.
        new_lines = new.split("\n")
        fix_list = list(Custom_Fixes.fix_map.keys())
        _new_lines = []
        for line in new_lines:
            for fix in fix_list:
                if fix in line:
                    panda_fix = Custom_Fixes.fix_map[fix]
                    line = panda_fix(line)
                    custom_fixes += 1
                    break
            _new_lines.append(line)
        new = "\n".join(_new_lines)
           
        # Get "diff_lines" (from py2to3.main) and set line tracking vars.
        diff_lines = diff_texts(old, new, filename)
        line_no = 0
        fixed_line_no = 0
        fixed_lines_offset = 0
       
        # "diff_lines" lists subtractions and additions in series
        # ie: (-,-,-,-) then (+,+,+,+). Increment 'stack' for every "-"
        # line and decrement it for every "+" line.
        stack = 0
        _prev_stack = 0  # helps track line removals.
       
        # Process diff lines to find and apply required panda fixes
        # Generate a log if LOGGING is True.
        for line in diff_lines:
            if line.startswith("+++") or line.startswith("---"):
                continue
               
            # Use sentinel lines to reset line counting vars.
            if line.startswith("@@"):
                line_list = line.split(" ")
                line_no = int(line_list[1].split(",")[0].replace("-", ""))
                fixed_line_no = line_no + fixed_lines_offset
                continue
               
            # Log removals.
            if line.startswith("-") and str(line).strip() != "-":
                stack += 1
                lines_fixed += 1
                if LOGGING: # Update log_lines.
                    log_str = "{:>6}: {}".format(line_no, line)
                    log_lines.append(log_str)
                line_no += 1
                continue
               
            # Log additions.
            elif line.startswith("+") and str(line).strip() != "+":
                _lines_added = 0
                if stack > 0:
                    fixed_line = line
                else:
                    # A new line when stack is at zero means this is a line
                    # that was simply added by py2to3, usually an import.
                    _lines_added += 1
                    line_no += 1
                    fixed_line_no += 1
                    fixed_line = line
                   
                # Update stats and log.
                if stack: stack -= 1
                if LOGGING:
                    log_str = "{:>6}: {}".format(fixed_line_no, fixed_line)
                    if stack == 0: log_str = "{}\n".format(log_str)
                    log_lines.append(log_str)
                fixed_line_no += 1
               
                # Handle rare case where py2to3 removes a line .
                if _prev_stack:
                    if LOGGING:
                        log_lines[-1] = "{}\n".format(log_lines[-1])
                   
                fixed_lines_offset += _lines_added
                lines_added += _lines_added
                continue
           
            # Line stats.
            line_no += 1
            fixed_line_no += 1
            _prev_stack = stack
           
        # Rewrite files.
        self.apply_fixes_to_file(filename, new)
       
    def apply_fixes_to_file(self, file_path, new):
        """Perform actual update of python file."""
        new_lines = []
        lines = new.splitlines(keepends=True)
        for i, line in enumerate(lines):
                new_lines.append(line)
        # Write new file.
        with open(file_path, "w") as file:
            file.writelines(new_lines)
   
    def write_file(self, new_text, filename, old_text, encoding=None):
        """Overriden from StdoutRT to prevent overwrite of our changes."""
        self.wrote = True

# Utility objects.
class Timer:
    def __enter__(self):
        self.start_dt = datetime.now()
        return self
    def __exit__(self, *e_info):
        if e_info[0] != None: print(e_info)
        end_dt = datetime.now()
        delta = end_dt - self.start_dt
        self.time = "{}.{}".format(delta.seconds, round(delta.microseconds, 3))

class Change_Logger:
    def __init__(self, dir):
        """Generate a "changes.txt" file for this dir if LOGGING == True."""
        self.dir = dir
    def __enter__(self):
        self._ref_dir_count = dir_count
        self._ref_file_count = file_count
        self._ref_line_count = line_count
        self._ref_lines_fixed = lines_fixed
        self._ref_lines_added = lines_added
        return self
    def __exit__(self, *e_info):
        if not LOGGING: return
        d_dir_count = dir_count - self._ref_dir_count
        d_file_count = file_count - self._ref_file_count
        d_line_count = line_count - self._ref_line_count
        d_lines_fixed = lines_fixed - self._ref_lines_fixed
        d_lines_added = lines_added - self._ref_lines_added
        with open(os.path.join(self.dir, "changes.txt"), "w") as log:
            date_str = datetime.now().strftime("%a %b %d, %Y - %H:%M")
            log.write("================\n")
            log.write("Panda to Python3 - {}\n".format(self.dir))
            log.write("================\n\n")
            log.write("date:         {}\n\n".format(date_str))
            log.write("dirs:         {}\n".format(d_dir_count))
            log.write("files:        {}\n".format(d_file_count))
            log.write("lines:        {}\n".format(d_line_count))
            log.write("lines fixed:  {}\n".format(d_lines_fixed))
            log.write("lines added:  {}\n\n\n".format(d_lines_added))
            log.write("'-' = old line\n'+' = py2to3 fix\n'*' = panda fix\n\n")
            for line in log_lines:
                line = "".join([line, "\n"])
                log.write(line)

# -----------------------------------
# Refactor panda.py files in SRC_DIRS
#------------------------------------

# Source directories for files to be refactored.
src_dir_list = ["direct", "pandac", "samples"]
SRC_DIRS = []
for s_dir in src_dir_list:
    src_dir = os.path.join(".", s_dir)
    SRC_DIRS.append(src_dir)
    if BACK_UP:
        # Back up src dirs.
        if PRINTING: print("Backing up: {}".format(s_dir))
        copy_dir = os.path.join(".", "_backup", s_dir)
        shutil.copytree(src_dir, copy_dir)   

# Refactoring algorithm.
dir_count, file_count, line_count = 0, 0, 0
lines_fixed, lines_added, custom_fixes = 0, 0, 0
with Timer() as timer:
    for src_dir in SRC_DIRS:
        for root, dirs, files in os.walk(src_dir):
            if PRINTING: print("".join(["\n", root.replace(".\\", "")]))
            with Change_Logger(root):
                prt = Panda3D_Refactoring_Tool(root)
                if LOGGING: log_lines = []
                for file in files:                    ## maybe allow __?
                    if file.endswith(".py") and not file.startswith("__"): 
                        src_file = os.path.join(root, file)
                        prt.refactor_panda_file(src_file)
                        pass
            dir_count += 1

# Finally refactor "panda3d.py" in main panda3d installation folder.
if PRINTING: print();print("<root dir>")
with Change_Logger("."):
    prt = Panda3D_Refactoring_Tool(".")
    if LOGGING: log_lines = []
    file_path = os.path.join(".", "panda3d.py")
    if BACK_UP:
        print("Backing up: panda3d.py")
        copy_path = os.path.join(".", "_backup")
        shutil.copy2(file_path, copy_path)  # backup panda3d.py.
    prt.refactor_panda_file(file_path)

if PRINTING:
    # Print totals.
    print();print()
    print("panda_to_py3: {} seconds\n".format(timer.time))
    print("dirs:         {}".format(dir_count))
    print("files:        {}".format(file_count))
    print("lines:        {}".format(line_count))
    print("lines fixed:  {}".format(lines_fixed))
    print("lines added:  {}".format(lines_added))
    print("custom fixes: {}".format(custom_fixes))



Just run it as a file in the main panda installation directory using python 3. It prints the results for each file and creates a change log for each directory so if there's any issues with the resulting code the exact change that caused it can be quickly found. It also backs up all the files and puts them in a dir called "_backup" before changing them. These features are really only for the development phase and can be turned off with various options for the release version.

Right now, I don't expect that it would produce working results because there are numerous things that the py2to3 script misses. There's a "Custom_Fixes" object for creating panda specific fixes for these cases, but I wasn't able to get very far in my testing before encountering the memory error so I was only able to create a fix for one of these (py2to3 doesn't catch references to the old __builitin__ module if they are in quotations). I would anticipate there being similar cases along the way.

I've set it up so it's fairly straightforward to add more custom fixes so either you can do this yourself if you encounter further problems or I can do it when there's a 32-bit build ready that works with python 3. Also, obviously re-write or remove any parts you want for the final release version. Hopefully this helps; let me know if you have any issues.