Just wanted to give a little update.
Either by my own accidental doing or by recent changes in Panda my server app is now using less than 100mb of memory instead of 900mb. That is huge.
Using the 3GB patch on python lets me keep my 32-bit build for development since the memory eaten by the EGG loader will not push the client over the memory limit. Thanks for that suggestion jean-claude!
With eigen enabled I am getting a frame rate increase somewhere in the range of 5-10%, basically for free. 
Did you ever figure out where the memory bloat was coming from? I seem to get about the same memory usage regardless of enabling eigen or not in the build.