eigen

Return to Compiling or Editing the Panda Source

eigen

Postby jean-claude » Mon Dec 19, 2011 5:14 pm

I noticed that eigen can be used within 1.8, the questions are:
> which version: Eigen 3.0.4?
> where to drop eigen folder: ./panda3d/thirdparty/win-libs-vc9/eigen?
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Mon Dec 19, 2011 5:52 pm

This is the brand-spanking-new code I'm working on right now, so if you enable it in your build you do so at your own risk. However, as it happens, I think I'm done, so I can recommend you give it a try.

Version 3.0.4 is what I used to develop with. Unpack the eigen tarball into panda3d/thirdparty/win-libs-vc9/eigen/include, eliminating the parent directory name, so that there exists a file panda3d/thirdparty/win-libs-vc9/eigen/include/Eigen/Dense .

The bad news is that as a large template library, Eigen dramatically slows down Panda's compilation time. On my computers, I'm now getting more than 3 hours for a clean build, where it use to compile in about 45 minutes. (This reminds me again one reason why I dislike template libraries so much. Still, at least it's not as bad as Boost.)

The good news is that the resulting compiled binary is noticeably faster, especially for computing animations.

Edit: I should add that you will want to empty your model-cache and/or re-bamify any models in order to take full advantage of the new SSE2 enhancements provided by Eigen. (Your bam files need to have the proper 16-byte alignment baked in at bam generation time.)

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby jean-claude » Mon Dec 19, 2011 6:11 pm

good news!
I'm going to give it right now a try and will let you know.
jean-claude

BTW. since I'm quite regularly compiling Panda with Intel Parallel Optimized Compiler, I think that Intel SSE4 + vectorization and //ization will provide an additional boost too.
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby jean-claude » Mon Dec 19, 2011 6:56 pm

FYI. Some problems while building 1.8 with eigen

Compiling on Windows with :
makepanda --verbose --everything --no-python --use-bullet --thread=4 --no-ode --optimize 4

(1) Issue with Bullet
1>c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include\vector(717) : error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
1> c:\Users\jc\Desktop\PANDA_BUILD_BOT\Source MAJ\panda3d\built\include\pvector.h(39) : see reference to class template instantiation 'std::vector<_Ty,_Ax>' being compiled
1> with
1> [
1> _Ty=LMatrix4,
1> _Ax=pallocator_array<LMatrix4>
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.h(68) : see reference to class template instantiation 'pvector<Type>' being compiled
1> with
1> [
1> Type=LMatrix4
1> ]
1>c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include\vector(717) : error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\built\include\pvector.h(39) : see reference to class template instantiation 'std::vector<_Ty,_Ax>' being compiled
1> with
1> [
1> _Ty=BulletDebugNode::Line,
1> _Ax=pallocator_array<BulletDebugNode::Line>
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\built\include\bulletDebugNode.h(89) : see reference to class template instantiation 'pvector<Type>' being compiled
1> with
1> [
1> Type=BulletDebugNode::Line
1> ]
1>c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include\vector(717) : error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\built\include\pvector.h(39) : see reference to class template instantiation 'std::vector<_Ty,_Ax>' being compiled
1> with
1> [
1> _Ty=BulletDebugNode::Triangle,
1> _Ax=pallocator_array<BulletDebugNode::Triangle>
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\built\include\bulletDebugNode.h(90) : see reference to class template instantiation 'pvector<Type>' being compiled
1> with
1> [
1> Type=BulletDebugNode::Triangle
1> ]
1>c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include\vector(717) : error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\built\include\pvector.h(39) : see reference to class template instantiation 'std::vector<_Ty,_Ax>' being compiled
1> with
1> [
1> _Ty=BulletDebugNode::Line,
1> _Ax=pallocator_array<BulletDebugNode::Line>
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\bullet\bulletDebugNode.h(89) : see reference to class template instantiation 'pvector<Type>' being compiled
1> with
1> [
1> Type=BulletDebugNode::Line
1> ]
1>c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include\vector(717) : error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\built\include\pvector.h(39) : see reference to class template instantiation 'std::vector<_Ty,_Ax>' being compiled
1> with
1> [
1> _Ty=BulletDebugNode::Triangle,
1> _Ax=pallocator_array<BulletDebugNode::Triangle>
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\bullet\bulletDebugNode.h(90) : see reference to class template instantiation 'pvector<Type>' being compiled
1> with
1> [
1> Type=BulletDebugNode::Triangle
1> ]
1>The following command returned a non-zero value: cl /wd4996 /wd4275 /wd4267 /wd4101 /wd4273 /DPANDA_WIN7 /DWINVER=0x601 /Fobuilt/tmp/p3physics_composite1.obj /nologo /c /Ipanda/src/physics /Ibuilt/tmp /Ibuilt/include /Ithirdparty/win-libs-vc9/eigen/include /DMAKEPANDA= /MD /Zi /Ox /Ob2 /Oi /Ot /arch:SSE2 /fp:fast /DFORCE_INLINING /DNDEBUG /GL /Oy /Zp16 /Fdbuilt/tmp/p3physics_composite1.pdb /DBUILDING_PANDAPHYSICS /EHa /Zm300 /DWIN32_VC /DWIN32 /W3 panda/src/physics/p3physics_composite1.cxx
1>Storing dependency cache.
1>Elapsed Time: 28 min 27 sec
1>Build process aborting.
1>Build terminated.


(2) If I remove bullet from the build (--no-bullet), then some alignment problems.. (alignment is obviously key for SSEx optimization!)
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\particlesystem\colorInterpolationManager.cxx(327) : error C2719: 'color_b': formal parameter with __declspec(align('16')) won't be aligned
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\particlesystem\colorInterpolationManager.cxx(344) : error C2719: 'color_a': formal parameter with __declspec(align('16')) won't be aligned
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\particlesystem\colorInterpolationManager.cxx(344) : error C2719: 'color_b': formal parameter with __declspec(align('16')) won't be aligned
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\particlesystem\colorInterpolationManager.cxx(362) : error C2719: 'color_a': formal parameter with __declspec(align('16')) won't be aligned
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\particlesystem\colorInterpolationManager.cxx(362) : error C2719: 'color_b': formal parameter with __declspec(align('16')) won't be aligned


(3) Still pursuing the build.
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Mon Dec 19, 2011 7:13 pm

Your compiler is indeed faster than mine. You've passed my poor Windows computer by, it hasn't gotten to that part of the tree yet.

(1) The solution here is to replace pvector<foo> with epvector<foo>, but only for those vectors which are generating errors. This is a special Eigen-provided pvector specialization that avoids this Windows library bug. I'll try to make guesses as to which pvectors need to be replaced from the error messages you report.

(2) This happens when someone mistakenly passes a lvec object by value instead of by reference, e.g. foo(LColorf color) instead foo(const LColorf &color). The solution is to correct the prototype to pass these in by reference correctly. I've just committed changes to particlesystem to do this.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby jean-claude » Tue Dec 20, 2011 2:09 am

Next: changes to be made in .\panda\src\physics\baseIntegrator.I
Code: Select all
INLINE const epvector< LMatrix4 > &BaseIntegrator::
get_precomputed_linear_matrices() const {
  return _precomputed_linear_matrices;
}

//////////////////////////////////////////////////////////
// Function : get_precomputed_angular_matrices
//   Access : protected
//////////////////////////////////////////////////////////
INLINE const epvector< LMatrix4 > &BaseIntegrator::
get_precomputed_angular_matrices() const {

to cope with :
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.I(20) : error C2556: 'const pvector<Type> &BaseIntegrator::get_precomputed_linear_matrices(void) const' : overloaded function differs only by return type from 'const BaseIntegrator::MatrixVector &BaseIntegrator::get_precomputed_linear_matrices(void) const'
1> with
1> [
1> Type=LMatrix4
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.h(55) : see declaration of 'BaseIntegrator::get_precomputed_linear_matrices'
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.I(20) : error C2371: 'BaseIntegrator::get_precomputed_linear_matrices' : redefinition; different basic types
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.h(55) : see declaration of 'BaseIntegrator::get_precomputed_linear_matrices'
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.I(29) : error C2556: 'const pvector<Type> &BaseIntegrator::get_precomputed_angular_matrices(void) const' : overloaded function differs only by return type from 'const BaseIntegrator::MatrixVector &BaseIntegrator::get_precomputed_angular_matrices(void) const'
1> with
1> [
1> Type=LMatrix4
1> ]
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.h(56) : see declaration of 'BaseIntegrator::get_precomputed_angular_matrices'
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.I(29) : error C2371: 'BaseIntegrator::get_precomputed_angular_matrices' : redefinition; different basic types
1> c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\baseIntegrator.h(56) : see declaration of 'BaseIntegrator::get_precomputed_angular_matrices'
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\panda\src\physics\linearEulerIntegrator.cxx(64) : error C2264: 'BaseIntegrator::get_precomputed_linear_matrices' : error in function definition or declaration; function not called
1>The following command returned a non-zero value: cl /wd4996 /wd4275 /wd4267 /wd4101 /wd4273 /DPANDA_WIN7 /DWINVER=0x601 /Fobuilt/tmp/p3physics_composite1.obj /nologo /c /Ipanda/src/physics /Ibuilt/tmp /Ibuilt/include /Ithirdparty/win-libs-vc9/eigen/include /DMAKEPANDA= /MD /Zi /Ox /Ob2 /Oi /Ot /arch:SSE2 /fp:fast /DFORCE_INLINING /DNDEBUG /GL /Oy /Zp16 /Fdbuilt/tmp/p3physics_composite1.pdb /DBUILDING_PANDAPHYSICS /EHa /Zm300 /DWIN32_VC /DWIN32 /W3 panda/src/physics/p3physics_composite1.cxx
1>Storing dependency cache.
1>Elapsed Time: 1 min 36 sec
1>Build process aborting.
Last edited by jean-claude on Tue Dec 20, 2011 3:58 am, edited 1 time in total.
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby jean-claude » Tue Dec 20, 2011 3:42 am

in daematerials.cxx and daematerials.h

make change to:
convert_blend(FCDEffectStandard::TransparencyMode mode, LColor &transparent, double transparency) {


same in ftltheader.h and fltheader.cxx
int get_closest_color(LColor &color) const;
int get_closest_rgb(LRGBColor &color) const;


in maxeggloader.cxx (have a closer look at this one)
int GetCVert(LColor &col);


need to have a look too to .\panda3d\pandatool\src\vrmlegg\indexedFaceSet.h
class VrmlPolygon {
public:
EggPolygon _attrib;
epvector<VrmlVertex> _verts;
}; ???

... and so on

At this point let me switch to --no-pandatool and try to pursue...

... almost done, but for whatever reason pzip is giving problems while compressing models (first time I see this...)
1>**Number of tasks left [T2] Compressing21 built/models/misc/lilsmiley.egg.pzout of
1> 324 starting tasks
1>built/bin/pzip built/models/misc/lilsmiley.egg
1>Finished generating code
1>The following command returned a non-zero value: built/bin/pzip built/models/misc/lilsmiley.egg
1>Storing dependency cache.
1>Elapsed Time: 18 min 15 sec
1>Build process aborting.
1>Build terminated.
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby teedee » Tue Dec 20, 2011 5:06 am

I am interested to test this out too. :)
drwr wrote:(2) This happens when someone mistakenly passes a lvec object by value instead of by reference, e.g. foo(LColorf color) instead foo(const LColorf &color). The solution is to correct the prototype to pass these in by reference correctly.

I think odeBody has the same issue with the set_quaternion function. After making the suggested change the build continues. Will see where it ends up in the morning.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby jean-claude » Tue Dec 20, 2011 6:13 am

pzip is generating crashes at the end of the build
this seems to be related to compress_stream.cxx in libpandaexpress.dll

here is the call stack:

ntdll.dll!77bf18d7()
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
msvcp90.dll!_Mtxlock(_RTL_CRITICAL_SECTION * _Mtx=0x003ed8b0) Line 45 C
msvcp90.dll!std::_Mutex::_Lock() Line 24 + 0x7 bytes C++
msvcp90.dll!std::basic_istream<unsigned short,std::char_traits<unsigned short> >::_Sentry_base::_Sentry_base(std::basic_istream<unsigned short,std::char_traits<unsigned short> > & _Istr={...}) Line 68 C++
msvcp90.dll!std::basic_istream<char,std::char_traits<char> >::sentry::sentry(std::basic_istream<char,std::char_traits<char> > & _Istr={...}, bool _Noskip=true) Line 86 C++
msvcp90.dll!std::basic_istream<char,std::char_traits<char> >::_Read_s(char * _Str=0x0084ebf0, unsigned int _Str_size=0xffffffff, int _Count=0x00001000) Line 693 + 0x13 bytes C++
msvcp90.dll!std::basic_istream<char,std::char_traits<char> >::read(char * _Str=0x0084ebf0, int _Count=0x00001000) Line 712 C++
> libpandaexpress.dll!compress_stream(std::basic_istream<char,std::char_traits<char> > & source={...}, std::basic_ostream<char,std::char_traits<char> > & dest={...}, int compression_level=0x00000006) Line 170 C++
pzip.exe!main(int argc=0x00000002, char * * argv=0x022d3128) Line 185 C++
pzip.exe!__tmainCRTStartup() Line 586 + 0x17 bytes C
kernel32.dll!761b339a()
ntdll.dll!77bc9ed2()
ntdll.dll!77bc9ea5()


bool
compress_stream(istream &source, ostream &dest, int compression_level) {
OCompressStream compress;
compress.open(&dest, false, compression_level);

static const size_t buffer_size = 4096;
char buffer[buffer_size];

source.read(buffer, buffer_size);
size_t count = source.gcount();
while (count != 0) {
compress.write(buffer, count);
source.read(buffer, buffer_size);
count = source.gcount();
}
compress.close();

return (!source.fail() || source.eof()) && (!compress.fail());
}
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby jean-claude » Tue Dec 20, 2011 6:22 am

Make sure too that
.\panda3d\thirdparty\win-libs-vc9\eigen\include\Eigen directory is moved to build/include
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Tue Dec 20, 2011 10:37 am

I've committed the suggested changes to the repository, thanks for that.

I'll be able to investigate the runtime crash in a couple of hours. My apologies for these unexpected issues; I hadn't actually tried to *run* this thing on Windows yet (I've been doing most of my development on OSX and Linux).

Edit: the pzip crash appears to be intermittent. I saw it happen once, but now I can't get it to happen again. Still investigating. . .

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby zhao » Tue Dec 20, 2011 6:52 pm

Go David go! :P
zhao
 
Posts: 225
Joined: Tue Nov 10, 2009 5:32 pm

Postby jean-claude » Tue Dec 20, 2011 7:12 pm

Almost ok, but still getting pzip.exe and flt2egg.exe crashing at the very end of the build

build launched with
makepanda --verbose --everything --no-python --use-bullet --thread=4 --no-ode --optimize 4 --use-max2011 --use-pandatool


Code: Select all
1>**Number of tasks left 31 out of [T1] Compressing  497built/models/misc/fade_sphere.egg.pz
1> starting tasksbuilt/bin/flt2egg -ps keep -o built/models/misc/fade_sphere.egg dmodels/src/misc/fade_sphere.flt
1>Reading dmodels/src/misc/camera.fltReading dmodels/src/misc/fade.flt
1>Writing built/models/misc/camera.egg
1>Writing built/models/misc/fade.egg
1>built/bin/pzip built/models/misc/camera.egg
1>built/models/misc/camera.egg.pz
1>The following command returned a non-zero value: built/bin/flt2egg -ps keep -o built/models/misc/fade_sphere.egg dmodels/src/misc/fade_sphere.flt
1>Storing dependency cache.
1>The following command returned a non-zero value: built/bin/pzip built/models/misc/camera.egg
1>The following command returned a non-zero value: built/bin/flt2egg -ps keep -o built/models/misc/fade.egg dmodels/src/misc/fade.flt
1>Elapsed Time: 1 hours 9 min
1>Build process aborting.
1>Build terminated.
1>Build log was saved at "file://c:\Users\jc\Desktop\PANDA_BUILD_BOT\Source MAJ\panda3d\built\BuildLog.htm"
1>makepanda - 0 error(s), 235 warning(s)
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby teedee » Tue Dec 20, 2011 7:24 pm

I get the same thing. If I keep running the build repeatedly it eventually makes it to then end, but the resulting build crashes.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Tue Dec 20, 2011 8:28 pm

I think I found it. It was a problem with our underlying ptmalloc2 scheme, which didn't like being asked to provide 16-byte alignment for some reason. I fixed it by not using that scheme anymore; it turns out not be the performance advantage it used to be anyway.

Let me know if you're still seeing these crashes. My own Windows build seems to be stable.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby jean-claude » Tue Dec 20, 2011 8:51 pm

too bad, something new is popping up :

1> No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\dtool\src\dtoolutil\pandaSystem.cxx(385) : error C2440: '<function-style-cast>' : cannot convert from 'TypeHandle' to 'PandaSystem::SystemTags'
1> No constructor could take the source type, or constructor overload resolution was ambiguous
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\dtool\src\dtoolutil\pandaSystem.cxx(385) : error C2228: left of '.second' must have class/struct/union
1>c:\users\jc\desktop\panda_build_bot\source maj\panda3d\dtool\src\dtoolutil\pandaSystem.cxx(401) : error C2440: '<function-style-cast>' : cannot convert from 'TypeHandle' to 'PandaSystem::SystemTags'
1> No constructor could take the source type, or constructor overload resolution was ambiguous
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Tue Dec 20, 2011 9:32 pm

Whoops, sorry, there are lots of different build options, and I missed a case. :(

Please try it again.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby jean-claude » Tue Dec 20, 2011 9:42 pm

ok, restarting!

btw. where is LINMATH_ALIGN defined?


edit: DONE! Thank you once more David !
1>-------------------------------------------------------------------
1>Build successfully finished, elapsed time: 54 min 49 sec
1>Build log was saved at "file://c:\Users\jc\Desktop\PANDA_BUILD_BOT\Source MAJ\panda3d\built\BuildLog.htm"
1>makepanda - 0 error(s), 218 warning(s)
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Tue Dec 20, 2011 11:17 pm

where is LINMATH_ALIGN defined?

In makepanda.py. It's hardcoded on by default.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Wed Dec 21, 2011 12:12 am

I was able to build successfully, but got a bit of a snag here:
Code: Select all
from panda3d.core import *
from direct.showbase.ShowBase import ShowBase

class Game(ShowBase):
    def __init__(self):
        """Get the game ready to play."""
        ShowBase.__init__(self)
        self.node = render
        self.model = self.loader.loadModel('smiley')
        self.model.reparentTo(self.node)
       
        shaderdata = Mat4(0, 0, 0, 0,
                          0, 0, 0, 0,
                          0, 0, 0, 0,
                          0, 0, 0, 0)
        self.model.setShaderInput('shaderdata', shaderdata)

game = Game()
game.run()

Code: Select all
Assertion failed: sizeof(mat(0, 0)) * mat.size() == pta.size() * sizeof(pta[0]) at line 338 of c:\work\panda3d\built\include\shader.I
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    game = Game()
  File "test.py", line 16, in __init__
    self.model.setShaderInput('shaderdata', shaderdata)
AssertionError: sizeof(mat(0, 0)) * mat.size() == pta.size() * sizeof(pta[0]) at line 338 of c:\work\panda3d\built\include\shader.I
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Wed Dec 21, 2011 10:11 am

Whoops. Clearly when I said "I think I'm done" back in post 2 of this thread, what I really meant was "I've committed everything and haven't tested a damn thing."

Please give it another try now. ;)

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby jean-claude » Wed Dec 21, 2011 12:46 pm

Hey David,
Forgive us to have put you under pressure... but you know Panda's groupies are like spoiled children waiting anxiously for Xmas morning!
:lol: :lol:
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Wed Dec 21, 2011 1:32 pm

As far as I believe now, everything is correct and working. Please let me know how your own experiments go. :)

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Wed Dec 21, 2011 3:39 pm

First thing I notice is Panda is using up a lot more system memory. Running my game the frame rate was actually a bit lower, but then I remembered about clearing out the BAM cache. Unfortunately, it is now crashing with an out of memory message (presumably while generating BAM files):
Code: Select all
Couldn't allocate memory page of size 131072: Not enough storage is available to
 process this command.


This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

That is for the game client. The server does run and takes up about 1.2GB of RAM as opposed to 0.6GB previously.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Wed Dec 21, 2011 4:14 pm

Hmm, that's strange. I would have expected a bit more memory usage, due to the 16-byte memory alignment requirements, but not double. Maybe the code itself is now much bigger due to the template explosion problem? This bears investigation.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby drwr » Wed Dec 21, 2011 6:50 pm

My early experiments seem to indicate about 20-30% more memory required by ordinary Panda operations. I'm trying a compile now with the original alternative malloc scheme to see how much of this bloat is due to alignment requirements and how much is due to the change in malloc schemes.

Is that 30% additional requirement enough to push you over the edge on the client? How close were you running previously? How much does memory grow before your client dies?

Edit: another thought--are you perhaps comparing against the memory usage in 1.7.2? It's also possible that the threading change with 1.8.0 is another major contributor to more memory consumption.

Would you be satisfied to use a Panda3D compiled with Win64, to allow it take advantage of more than 2GB of RAM?

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Thu Dec 22, 2011 12:52 am

I am comparing against my previous build from cvs, both of which I have disabled threading in.

I may not have waited long enough for my original memory measurement, the server seems to be using 0.9GB on the previous build now, so the increase to 1.2GB would be consistent with the 30% you are seeing. Actually now that I think about it that is a ton of memory for what is actually going on in the server so perhaps something for me to investigate there.

It seems to use a lot more memory when it has to cache EGG to BAM files and not release that memory. For example on the previous build (before eigen), first run of the client after clearing the BAM cache it is using almost 1.5GB, but after closing it and running again it is using only 0.8GB. It is likely that this is the real reason why it is hitting the 32 bit memory limit, and it's just that this extra 30% on top of that has pushed it over the limit. Looks like the client at about 1.8GB when it crashes. If it was going to be taking up 1.5GB in the previous build, then that extra 30% would put it at about 2GB.

I can't rely on a 64 bit build since end users may be running a 32 bit OS.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby jean-claude » Thu Dec 22, 2011 11:50 am

Hi teedee,
As a quick & dirty turn around you can try
editbin /LARGEADDRESSAWARE yourapp.exe
This should provide a true 3gb addressable space on x32 machines

EDIT: recommandation: do it merely as a post build command
User avatar
jean-claude
 
Posts: 384
Joined: Sun Jan 23, 2011 1:41 pm
Location: Paris - France

Postby drwr » Thu Dec 22, 2011 4:49 pm

It appears after some experimentation that this 30% memory bloat is due to the need to impose an external 16-byte alignment on top of the system malloc library. If we go back to an alternative malloc library that can provide 16-byte alignment natively, this waste goes away. I'll run a few more tests to confirm this, then commit this change.

jean-claude's advice is good, though. Also, you might consider distributing your game to end-users with your egg files already prebuilt to bams, to avoid this potential problem in general (the egg loader was never intended to be particularly memory-efficient; it was primarily designed for the developer, not the end user).

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Thu Dec 22, 2011 5:50 pm

Yes the plan would be to eventually distribute as BAM files. I just didn't want to maintain a 64-bit build for development in addition to the 32-bit for distribution. The large address tweak sounds like it would be a good solution for development though.

About the EGG loader, I just found it strange that the memory it uses sticks around even after the game is loaded up. Shouldn't that memory it uses to convert to BAM be able to be released once it has finished loading?

I suppose I could have some pre-process that runs and loads up the EGG files to populate the BAM cache before running the game itself. I had assumed that is what was happening under the hood anyways, but maybe I've misunderstood.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Next

Return to Compiling or Editing the Panda Source

Who is online

Users browsing this forum: No registered users and 0 guests