I have a big problem, my game leaks memory, and when i say leaks that is an under statement, it like a ship with with a hole sinking … where the hole is bigger then the ship (quantum mechanics)…
Well i climb to about 1gb in about 30 minutes of playing. We have tested this on different computers, some computers climb faster others slower.
At first it seamed like that was based on frame rate, faster computers climbed faster … but that is not the case any more one of our fastest computers climbed slower to about 300mb(still leaks) in 1 hour of playing.
The mem leak makes the panda run very slow, it drops to about 5fps when i hit my memory limit the computer almost freezes.
Independent test confirmed that leaks stops or is very small when you go into ‘radar mode’ in that mode all the ships rendered as billboards…
So i must assume the leak is tided to the geometry some how. All of my geometry is static (no animating vertexes) just ships with turrets (turrets are separate static nodes)
My question is how do i find it? How do i stop it? And what are you suggestions?
I’d be tempted to use python module “gc” to turn on the garbage collector’s debugging and statistics mechanisms. I bet you can find out something like “there are 1,000,000 allocated objects of class Interval” or something like that.
I found this code some time ago (don’t know anymore where I found it, sorry):
import sys
import types
def get_refcounts():
d = {}
sys.modules
# collect all classes
for m in sys.modules.values():
for sym in dir(m):
o = getattr (m, sym)
if type(o) is types.ClassType:
d[o] = sys.getrefcount (o)
# sort by refcount
pairs = map (lambda x: (x[1],x[0]), d.items())
pairs.sort()
pairs.reverse()
return pairs
def print_top_100():
for n, c in get_refcounts()[:100]:
print '%10d %s' % (n, c.__name__)
if __name__ == '__main__':
top_100()
Should return the 100 classes which the most objects allocated. From there on you can get single references and their referents with GC.
Sounds like there is some cyclic references or misuse of try/except. Even with Python it is possible to shoot your own foot.
thanks guys! i implemented your thing and it did help me a bit. Well i know that my memory does not leak but the number of geoms rises. Is there a way to list/search all the geoms? it starts at 900 but soon climbs to 2000 and keeps climbing.
Well, you could try turning on directtools, and use the scene graph browser to see if those 2000 geoms are part of the scene - maybe you can find a branch of the tree that’s much bigger than it should be. You could also check pstats output to see if it’s rendering or culling those geoms. If they don’t show up in the scene, then it’s possible that you have a circular reference count thing going on.
There are actually tools built into Panda to identify leaked objects. It’s a bit arcane to use, but it’s workable.
First, put:
track-memory-usage 1
in your Config.prc file. Note that this will dramatically slow down your performance.
Now you can do:
mup = MemoryUsagePointers()
MemoryUsage.getPointers(mup)
for i in range(mup.getNumPointers()):
obj = mup.getPythonPointer(i)
print obj.__class__
And you can walk through the entire list of allocated Panda objects. It will probably be a long list.
There are other tricks you can do, but this is the most powerful feature. It might give you some insight into what objects are leaking. Once you know what they are, you might be able to figure out why they’re leaking (almost certainly because of a circular reference count somewhere).
ok it leaks! It does not leaks Geoms … no no, nor node paths nor any python objects at all! Yes the number of ints,strings,lists … and classes hovers about the same value. What leaks is geometry data. When i display many triangles 400,000 in my case (yes i get 20fps we are working on lods) it leaks a lot 100k/s when i display just couple of billboards (200 tri) it leaks about 10-6k/s. I think its got to be some scene graph traversal of only the visible objects. Question is how do I debug that?
Interesting. I am watching memory a lot, because of the C++ extension stuff I do. And so far I never found memory leaks in Panda3D (only in my own C++ extensions grumble).
Can you narrow this down a bit, and perhaps provide a small demo? Best would bust a just single geom, and whatever code you do to manipulate it every frame.
Note that there is an internal Panda cache of recently-rendered Geom objects. The default cache size is quite large. If you are creating and destroying many of these, it might be a while before the cache fills up; and until then, it will look like a memory leak.
Set:
geom-cache-size 0
in your Config.prc to disable this cache for the purposes of memory leak detection.
But it leaks into virtual memory at witch point the program comes to a grinding halt… that is over about 800MB where the games run fine with just 100MB. Because the computer start to thrash when i git the memory limit i have reason to believe that it uses all of that memory; that and the fact that game gets slower and slower as more memory is used.
EDIT:
David, you are right now i see it freeing memory now! This is great. But there is still stuff some leaks but at least it does not leak 8-40k per second it stays at normal level now. Can we have more info on how Geom Cache works?
Memory leak now is in the space system jumping (and now it might be some thing i do or don’t like clean objects from the last system)
The geom cache keeps hold of the n last rendered Geoms, in case they might be rendered again any time soon. This is an important optimization, because when Panda renders a new Geom for the first time, it has to “munge” it appropriately for the render engine; for instance, it has to convert OpenGL color specification to DirectX format if necessary, or vice-versa.
The default value for n (the default value for geom-cache-size) is 5000. This is a fairly generous default value, though for most apps, it’s better to be generous than conservative.
If your app is creating a lot of large dynamic Geoms, and throwing them away every frame, it will create 5000 of them before any of their memory gets recycled, if you leave geom-cache-size at its default value. For your app, then, you should choose a smaller value for geom-cache-size. But you probably want it larger than 0 for performance reasons.
But are you really creating and throwing away large dynamic Geoms every frame? Maybe it would be better to create a Geom once, and modify it every frame thereafter, instead of creating brand new geometry from scratch.
Is there a way to control which geom go into the cache? like Geom.UHDynamic/UHStatic?
First i draw lots of lines for orders/positions. Then i created my own particle system that draws lots of particles based on one plate texture. I will look into how to use modify instead of recreate.
I know it’s an old thread but I have EXACTLY the same problem as you are telling. I’m not at home right now so I can’t test the “geom-cache-size 0” that was suggested but since the thread never tell “I fix it” I was wondering if you ever fix that bug.
Because I also based my code on the architecture of your game Iron Angel (class for everything, list of weapon, etc), maybe we have the same bug!
I check a couple of things and discover that it’s highly (if not completly) relatad to my weapons. Like each time I fire a weapon, I get 0.5 meg added to the memory (when I look at the process in windows). If I do not fire weapon and the enemy too, the memery is stable.
I will wait for your answer and depending of it, I could paste some code.
Thanks
Jaff
P.S: It’s a very annoying bug because during the last 2 or 3 months, neither me or my tester detect this because we were just playing for a couple of minute and then quit but since my game is almost complete, we get this bug everytime.
“architecture of your game AFF” - you mean iron angels.
We never really tested iron angels - it was done in such haste (1 week) that i don’t think we even removed them properly. My problem was filling up the geom cache with dynamic but labeled as static data the geom cache fixed the problem momentary then the real problem was fixed with the pyro projectile system.