setOneShot and RTMCopyRam?

Return to Scripting Issues

setOneShot and RTMCopyRam?

Postby drwr » Fri Sep 30, 2011 1:51 pm

This message is directed at Craig, who asked about this question on my recent blog post:

I've been so far unable to reproduce the reported problem. setOneShot() appears to be working for me in conjunction with RTMCopyRam. Here is some sample code that demonstrates it working--load it up and press "b" to create a buffer and start things going. It should render a scene offscreen and, when the result gets back to RAM, it should save it to tex.png and also display it onscreen. Does this code work for you, or does it fail?

Code: Select all
from direct.directbase.DirectStart import *
from panda3d.core import *

def saveTexture(tex, buffer, task):
    if not tex.hasRamImage():
        return task.cont

    print "Got texture"
    tex.write('tex.png')
    cm = CardMaker('card')
    cm.setFrame(-0.5, 0.5, -0.5, 0.5)
    card = aspect2d.attachNewNode(cm.generate())
    card.setTexture(tex)

    base.graphicsEngine.removeWindow(buffer)

    return task.done

def makeBuffer():
    tex = Texture('tex')
    buffer = base.win.makeTextureBuffer('buffer', 256, 256, tex, True)
    buffer.setClearColor((0, 0, 0, 1))
    print "Made %s" % (buffer.getType())

    bufscene = NodePath('bufscene')
    bufcam = bufscene.attachNewNode(Camera('bufcam'))
    dr = buffer.makeDisplayRegion()
    dr.setCamera(bufcam)

    m = loader.loadModel('smiley.egg')
    m.reparentTo(bufscene)
    m.setPos(0, 20, 0)

    buffer.setOneShot(True)

    taskMgr.add(saveTexture, 'saveTexture', extraArgs = [tex, buffer], appendTask = True)
   
   
base.accept('b', makeBuffer)
run()


David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby zhao » Fri Sep 30, 2011 7:57 pm

David, I have a tangential problem. Opening an offscreen buffer using base.graphicsEngine.makeOutput is crashing under both gl and dx9 when I enable the new threading model using 'threading-model Cull/Draw'. This is on a GTX 260/winxp 32 bitmachine.


Code: Select all
from panda3d.core import loadPrcFileData
#loadPrcFileData('', 'load-display pandadx9' )
loadPrcFileData('', 'load-display pandagl' )
loadPrcFileData('', 'framebuffer-multisample 0')
loadPrcFileData('', 'multisamples 0')
import direct.directbase.DirectStart
from panda3d.core import NodePath, FrameBufferProperties, GraphicsPipe, GraphicsOutput, Texture, WindowProperties

winprops = WindowProperties()
winprops.setSize( 512, 512 )
fbprops = FrameBufferProperties()
fbprops.setColorBits(1)
fbprops.setAlphaBits(1)
objBuffer = base.graphicsEngine.makeOutput(base.pipe, 'hello', -10, fbprops, winprops, GraphicsPipe.BFRefuseWindow, base.win.getGsg(), base.win)

run()
zhao
 
Posts: 225
Joined: Tue Nov 10, 2009 5:32 pm

Postby teedee » Fri Sep 30, 2011 9:30 pm

I believe I have the same problem with the buffers.
Is this the sort of error you get?
Code: Select all
Traceback (most recent call last):
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 1719, in __igLoop

    self.graphicsEngine.renderFrame()
AssertionError: pipeline_stage == 0 at line 667 of c:\work\panda3d\panda\src\display\displayRegion.cxx
:task(error): Exception occurred in PythonTask igLoop
Traceback (most recent call last):
  File "client.py", line 564, in <module>
    game.run()
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 2644, in run
    self.taskMgr.run()
  File "C:\work\sp1\Panda3D\direct\task\Task.py", line 502, in run
    self.step()
  File "C:\work\sp1\Panda3D\direct\task\Task.py", line 460, in step
    self.mgr.poll()
  File "C:\work\sp1\Panda3D\direct\showbase\ShowBase.py", line 1719, in __igLoop

    self.graphicsEngine.renderFrame()
AssertionError: pipeline_stage == 0 at line 667 of c:\work\panda3d\panda\src\display\displayRegion.cxx


Another curious side-effect is that by setting a window icon, Panda becomes unable to open a window. For example:
Code: Select all
loadPrcFileData('', 'icon-filename icon.ico')

Specifically, Python will sit idle with no CPU usage when it hits line 611 of ShowBase.py, the graphicsEngine.makeOutput call.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby zhao » Fri Sep 30, 2011 10:34 pm

Yep. Same error message, in opengl mode, it gives a slightly different error message


:display:gsg:glgsg(error): at 87 of c:\buildslave\dev_sdk_win32\build\panda3d\pa
nda\src\glstuff\glGraphicsBuffer_src.cxx : invalid operation
zhao
 
Posts: 225
Joined: Tue Nov 10, 2009 5:32 pm

Postby drwr » Sat Oct 01, 2011 12:48 pm

OK, it's true that it doesn't work in multithreaded mode, and I'm working on that right now.

But I was hoping to answer whether it works in the original, single-threaded mode. Craig's report seems to suggest that it doesn't work even in that case. Can anyone else confirm this?

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Sat Oct 01, 2011 1:46 pm

In my case it works just fine in the single-thread mode, albeit slower than threads disabled which is to be expected.
Edit: By "it" I meant my game, but I also just tried the example you posted, David, and it works as well.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby zhao » Sat Oct 01, 2011 3:46 pm

It works fine for me as well in the single thread mode.

It does throw a warning about pnmimage and transparencies though.
zhao
 
Posts: 225
Joined: Tue Nov 10, 2009 5:32 pm

Postby Craig » Sun Oct 02, 2011 11:25 pm

Running your code from the first post on my older mac, it runs until I hit b, then if freezes for a while and crashes with a bus error:


DirectStart: Starting the game.
Known pipe types:
osxGraphicsPipe
(all display modules loaded.)
Made GLGraphicsBuffer
/var/folders/M4/M4oCK5IpHAucjm-kozc7uk+++TI/Cleanup At Startup/untitled text-339307867.848.command: line 3: 252 Bus error /usr/local/bin/python /private/var/folders/M4/M4oCK5IpHAucjm-kozc7uk+++TI/Cleanup\ At\ Startup/untitled\ text-339307867.730
logout

[Process completed]



Anyway, thats not the computer I usually use (horrible graphics support), and your code works fine on the machine which I was producing my issue on (a newer mac).


This modified version which reuses the same buffer only works the first time, then all repeated attempts never finish. This is the issue I was having I think.

Code: Select all
from direct.directbase.DirectStart import *
from panda3d.core import *

def saveTexture(tex, buffer, task):
    if not tex.hasRamImage():
        return task.cont

    print "Got texture"
    tex.write('tex.png')
    cm = CardMaker('card')
    cm.setFrame(-0.5, 0.5, -0.5, 0.5)
    card = aspect2d.attachNewNode(cm.generate())
    card.setTexture(tex)
   
    buffer.clearRenderTextures()

    return task.done

buffer = base.win.makeTextureBuffer('buffer', 256, 256, Texture(), True)

def makeBuffer():
    tex = Texture('tex')
   
   
    mode=GraphicsOutput.RTMCopyRam
    buffer.addRenderTexture(tex,mode)
   
    buffer.setClearColor((0, 0, 0, 1))
    print "Made %s" % (buffer.getType())

    bufscene = NodePath('bufscene')
    bufcam = bufscene.attachNewNode(Camera('bufcam'))
    dr = buffer.makeDisplayRegion()
    dr.setCamera(bufcam)

    m = loader.loadModel('smiley.egg')
    m.reparentTo(bufscene)
    m.setPos(0, 20, 0)

    buffer.setOneShot(True)

    taskMgr.add(saveTexture, 'saveTexture', extraArgs = [tex, buffer], appendTask = True)
   
   
base.accept('b', makeBuffer)
run()


That should work right?
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Postby drwr » Mon Oct 03, 2011 11:32 am

Ah, hmm. This is actually beyond the original intention of setOneShot(), which was to permanently disable the buffer after the first time.

Actually, the original intention of setOneShot() was to *delete* the buffer after one frame, but that turned out to be problematic, so we eventually softened it into *disabling* the buffer after one frame, and required the user to explicitly delete it later. But since we have made that change, it makes sense to allow the buffer to be re-used again later if desired. I'll see about making the necessary changes to support that.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby drwr » Wed Oct 05, 2011 9:19 am

FYI, I've committed my recent fixes for these issues, and the buildbot server has picked them up. In particular: (1) the above code, and offscreen buffers in general, should now work correctly in threaded mode, and (2) setOneShot() is now reusable.

Please let me know as you find additional issues. :)

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Thu Oct 06, 2011 5:13 am

I tried it out on the "Teapot on TV" sample and got about a +50% FPS boost, pretty good for a sample that has almost nothing going on!
My game still freezes before it gets to render a frame, I'll try to make a small example which fails.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby teedee » Wed Oct 26, 2011 1:37 am

I finally got a chance to debug this, here is some code which causes the threading to fail:

Code: Select all
from panda3d.core import *
loadPrcFileData('', 'threading-model Cull/Draw')
from direct.directbase.DirectStart import *

base.wireframeOn()
res_x, res_y = 16, 9
# writers
vdata = GeomVertexData('vdata', GeomVertexFormat.getV3t2(), Geom.UHDynamic)
vtxwriter = GeomVertexWriter(vdata, 'vertex')
texwriter = GeomVertexWriter(vdata, 'texcoord')
# verts
for y in range(res_y):
    for x in range(res_x):
        vtxwriter.addData3f(x, -y, 0)
        vtxwriter.addData3f(x+1, -y, 0)
        vtxwriter.addData3f(x+1, -y-1, 0)
        vtxwriter.addData3f(x, -y-1, 0)
        for i in range(4):
            texwriter.addData2f(0, 0)
# polys
prim = GeomTriangles(Geom.UHStatic)
for i in range(res_x * res_y):
    prim.addVertices(i*4+2, i*4+1, i*4)
    prim.addVertices(i*4+3, i*4+2, i*4)
prim.closePrimitive()
# model
geom = Geom(vdata)
geom.addPrimitive(prim)
node = GeomNode('geom')
node.addGeom(geom)
model = render.attachNewNode(node)

run()
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Wed Oct 26, 2011 9:59 am

Ah, I can run the code and reproduce the failure. Deadlock! Curse you, subtle threading issues! Thanks for the sample code.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby drwr » Wed Oct 26, 2011 10:07 am

Ah, I see what's going on. Not precisely a bug in Panda--the problem is that your GeomVertexWriter objects are still in scope at the time you call run(), so you are still holding the write locks on the vertices, and when the renderer tries to grab the write locks--deadlock.

The simple solution is to add:
Code: Select all
del vtxwriter
del texwriter

before your call to run().

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Wed Oct 26, 2011 12:53 pm

Ah yes, that has got it going!
I like how the animation goes into the Cull thread, that really helps even things out.

I am still getting some sort of deadlock in one of the levels in the game that is not happening in the other level I tested so I guess I can sort out what is causing that by the difference in the levels. Is there an easier way to find out what is causing the lock?

I thought you might be interested in the performance numbers:
Non-threaded build: 40-45 fps
Threaded build WITHOUT threading-model set: 30ish fps
Threaded build WITH threading-model set: 55-65 fps

Supposing I wanted to get the best speed on a single core or multiple cores, would I need two builds of Panda (threaded and non-threaded)?
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Wed Oct 26, 2011 1:15 pm

Is there an easier way to find out what is causing the lock?

Well, you can break into the running Panda with a debugger and see where in the code each thread is stopped; that's what I did in this case. But that may not tell you much unless you already have a good sense of what the code is supposed to be doing. (I saw that the main thread was waiting for the cull thread to complete, which is normal, and that the cull thread was waiting to access a GeomVertexData, which isn't normal. That gave me the clue to suspect that the main thread was holding the lock on the GeomVertexData.)

However, if you build a version of Panda with DEBUG_THREADS defined, then it will compile a special version of the threading library that monitors each lock and unlock and attempts to check for deadlock. Not all deadlock conditions can be reported, but many can (this example would have been), and if it detects the deadlock it will tell you what locks were being held by which threads, which may also give you insight. Of course the DEBUG_THREADS version of Panda runs more slowly.

Supposing I wanted to get the best speed on a single core or multiple cores, would I need two builds of Panda (threaded and non-threaded)?

Yes. The fastest possible speed on a single core is always with a single-threaded build. Having threading support available necessarily adds additional overhead to all low-level operations across the board, even if you're not using threads at the moment.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Wed Oct 26, 2011 4:18 pm

Thanks, that was helpful. I managed to find a couple more problems related to the vertex data issue. Now it runs for a fair bit longer, but I still get a deadlock.
I tried the thread debug build, this is what I've got:
Code: Select all
:thread(error):

****************************************************************
*****                 Deadlock detected!                   *****
****************************************************************

:thread(error): Thread Cull attempted to lock Mutex  03D9BDE0 which is held by MainThread Main
:thread(error): MainThread Main is blocked waiting on CyclerMutex PandaNode::CData which is held by Thread Cull
:thread(error): Deadlock!
Assertion failed: Deadlock at line 205 of c:\work\panda3d\panda\src\pipeline\mutexDebug.cxx
Assertion failed: node->_prev != NULL && node->_prev->_next == node && node->_next->_prev == node at line 92 of c:\work\panda3d\built\include\linkedListNode.I
Assertion failed: Thread Cull attempted to release Mutex  03D9BDE0 which it does not own at line 369 of c:\work\panda3d\panda\src\pipeline\mutexDebug.cxx
Traceback (most recent call last):
  File "C:\work\panda3d\built\direct\showbase\ShowBase.py", line 1656, in __resetPrevTransform
    PandaNode.resetAllPrevTransform()
AssertionError: Deadlock at line 205 of c:\work\panda3d\panda\src\pipeline\mutexDebug.cxx
:task(error): Exception occurred in PythonTask resetPrevTransform
Traceback (most recent call last):
  File "client.py", line 564, in <module>
    game.run()
  File "C:\work\panda3d\built\direct\showbase\ShowBase.py", line 2648, in run
    self.taskMgr.run()
  File "C:\work\panda3d\built\direct\task\Task.py", line 502, in run
    self.step()
  File "C:\work\panda3d\built\direct\task\Task.py", line 460, in step
    self.mgr.poll()
  File "C:\work\panda3d\built\direct\showbase\ShowBase.py", line 1656, in __resetPrevTransform
    PandaNode.resetAllPrevTransform()
AssertionError: Deadlock at line 205 of c:\work\panda3d\panda\src\pipeline\mutexDebug.cxx

CData, is it collision?
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Wed Oct 26, 2011 5:15 pm

CData is short for CycleData, which is the internal name for the data that is copied between pipeline stages across all Panda objects. In this case, it's PandaNode::CData, which means the deadlock is on something in PandaNode: something in the main thread is waiting for a PandaNode lock (held by Cull) at the same time as it's also holding this unnamed Mutex 03D9BDE0, and in the meantime Cull tried to grab this unnamed Mutex.

That's deadlock all right, but it's not very informative since we're not sure what the unnamed mutex was. The stack trace gives us a bit of a clue, that it happened during resetAllPrevTransform(), but I'm still not sure what the contention is. Bleah.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Sun Nov 06, 2011 6:08 pm

After some digging, it seems this deadlock is related to something in my positional audio update code, and possibly FMOD. The more sounds I have the more quickly it will deadlock.
My guess would be as a result of many calls to sound.status(), sound.set3dAttributes(), or sound.setVolume(), but who knows.
I'll try to isolate the sound code and see if it still has a problem on its own. That should make a fairly good demo of the problem.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Sun Nov 06, 2011 7:11 pm

Ah, OK. I'd be willing to bet that Panda's FMod audio layer (and its OpenAL audio layer, for that matter) doesn't properly protect itself against multithreaded access.

Both of these were originally implemented by people who likely weren't thinking in terms of multithreaded code.

I'll put it on the list to investigate. In the meantime, one easy way to assure yourself that it is, in fact, strictly related to the audio subsystem is to run with "audio-library-name null" in your Config.prc file.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Sun Nov 06, 2011 7:23 pm

Well simply skipping my audio update task seems to prevent it from deadlocking, or at least reduces the chance of deadlock by so much that I wasn't able to make it happen. I should have some sample code soon.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby teedee » Sun Nov 06, 2011 11:21 pm

I managed to separate out the audio portion of my code but it is not locking, probably because whatever other thing wanted the same data is now gone. It is for sure the audio system, setting the library to null there were no problems at all whereas otherwise it would deadlock in a few seconds.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Mon Nov 07, 2011 12:50 am

Hmm, I don't suppose you had things in different threads that were might have been starting the same sound object at once? How about starting different sound objects, but from different threads? Or loading sound objects? Or setting 3-D positional sound?

How many different threads do you have accessing sound at all?

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Mon Nov 07, 2011 12:38 pm

I don't have any custom threads, so it would just be the App/Cull/Draw pipeline and FMOD's own thread that it creates. All of my audio stuff is happening in the main thread.

Actually I realized that the game does not seem to deadlock until I start doing stuff with an audiomanager other than the default one. Could it be that holding onto a reference to the audiomanager is a problem?

I can have another go at dismantling my code bit by bit again if that helps, it will just take a while.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Mon Nov 07, 2011 4:57 pm

Hmm, the only thing FMod does in its child threads (that relates to Panda) is to make callbacks into the vfs to read file data, and that should be thread-safe. Hmm. Does it crash if you:

(1) Try forcing all sound files to load fully in RAM by setting "fmod-audio-preload-threshold -1"?

(2) Switch to OpenAL?

(3) Use a single-threaded rendering model (by disabling graphics-threading-model)?

How about if you set "notify-level-audio debug" to get more debugging information? This should tell you when it is streaming data from sound files, which might give more insight into what is going on just before a crash.

I can't think of any reason that having multiple AudioManagers could cause a problem, but I certainly can't rule it out.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Mon Nov 07, 2011 6:13 pm

"fmod-audio-preload-threshold -1" is already set, as I had performance problems previously with streaming too many sounds.
I don't have OpenAL in my current build but I will give it a try when I get a chance to do a new build. I remember having other issues with OpenAL previously in any configuration so it might fail for a different reason.
Using both a non-threaded build and single-threaded work just fine.

The last printout before deadlock seems to always be
:audio(debug): FmodAudioManager::audio_3d_set_listener_attributes()
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Mon Nov 07, 2011 6:24 pm

That's pretty damning, isn't it? I bet the problem is somewhere in FmodAudioManager::audio_3d_set_listener_attributes(). ;)

Looking at that function now, it is only calling a method on the global FMod _system object, but that could be trouble if it gets called from separate threads, or even if it is only called from the main thread but there is something else FMod-related going on in another thread simultaneously.

I think all of these calls into the global _system object need to be protected with a mutex inside the p3fmod layer. I'll try to make this change in the next few days, and we can see if it helps your crash.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Mon Nov 07, 2011 7:37 pm

Yeah I think the audio_3d_set_listener_attributes is a bit deceptive because it seems there are a lot of functions that don't do a debug printout.
If I remove all my python calls to audio3dSetListenerAttributes there is still deadlock.
I'm actually going through right now and slowly removing any audio calls from my code to try and find out exactly what the cause is.
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Postby drwr » Mon Nov 07, 2011 8:32 pm

It didn't take as long as I'd thought; I've just committed the changes that ought to protect all of the FMod stuff against multithreaded access. Give it a try and let me know if it does any better now.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby teedee » Mon Nov 07, 2011 11:10 pm

Sorry to say it did not resolve the deadlock, I will continue to try to trim down my code until I have a smaller sample.
Also there was a build error introduced in regard to the recent pfm changes. Line 152 in pfmTrans.cxx is using the old interface still "NodePath mesh = file.generate_vis_mesh(true);"
teedee
 
Posts: 861
Joined: Tue May 12, 2009 11:33 pm
Location: Kepler-22b

Next

Return to Scripting Issues

Who is online

Users browsing this forum: Bing [Bot] and 1 guest