Threading using Python

Hi,

I’m trying to build an object-based graphics development environment using Panda3D. A critical aspect of this is running code in a thread, while the main loop continues updating frames. However I have struck a snag doing this. I can get a test program working with the standard Python threading module, but according to the Panda3D documentation this will lead to highly unstable code. If I use the Panda3D threading module then the thread runs but the main thread stops rendering frames. This behaviour is the same whether I use my own threads, or use an Async Task Chain. Examples below.

My question is, is it safe for me to continue using the standard Python threading module? If not, is there any way to get Panda3D threads to run while frames continue to be rendered?

Here is the sample code for threading (Panda3D SDK 1.6.2):

from direct.directbase import DirectStart
from direct.showbase.DirectObject import DirectObject
from direct.interval.IntervalGlobal import *
from direct.actor.Actor import Actor
from pandac.PandaModules import Point3


#from direct.stdpy import threading
import threading

class myThread(threading.Thread):

    def run(self):
        print 'thread starting.'
        mydata = threading.local()
        mydata.i =panda.hprInterval(1, Point3(panda.getH(),panda.getP(),panda.getR()+360))
        mydata.i.start()
        # With a simple "import threading" this code runs in a separate thread and the main loop keeps on
        # running, hence the interval plays until it stops. But with "from direct.stdpy import threading"
        # this code stops the interval from playing and the program freezes.
        while mydata.i.isPlaying():
            pass
        print 'thread ending.'

base.disableMouse()
base.camera.setPos(0,-1000,0)
base.camera.lookAt(0,0,0)

panda = Actor("models/panda-model")
panda.setScale(0.1, 0.1, 0.1)
panda.setPos(0,0,0)
panda.reparentTo(render)

t = myThread()
t.start()

run()

here is the sample code for Async Tasks (Panda3D SDK 1.6.2):

from direct.directbase import DirectStart
from direct.showbase.DirectObject import DirectObject
from direct.interval.IntervalGlobal import *
from direct.actor.Actor import Actor
from pandac.PandaModules import Point3

def exampleTask(task):
    # I expect that since this task is running in a separate thread from the main loop it should
    # allow the interval to play, but it doesn't - it blocks the main loop, the interval stops playing,
    # and the program freezes.
    while i.isPlaying():
        pass
    print 'threaded task ending.'
    return task.done

base.disableMouse()
base.camera.setPos(0,-1000,0)
base.camera.lookAt(0,0,0)

panda1 = Actor("models/panda-model")
panda1.setScale(0.1, 0.1, 0.1)
panda1.setPos(0,0,0)
panda1.reparentTo(render)
        
i = panda1.hprInterval(4, Point3(panda1.getH(),panda1.getP(),panda1.getR()+360))
i.start()

taskMgr.setupTaskChain('threadedChain1', numThreads = 1)
taskMgr.doMethodLater(2, exampleTask, 'someTaskName', taskChain = 'threadedChain1')
print taskMgr


run()

Any help appreciated.

Panda threads are cooperative, so you have to call Thread.considerYield() inside the loop in your sub-thread, to allow the main thread to run.

But I don’t see anything that you’re doing that really requires running in a thread. Wouldn’t a normal, synchronous task be a better choice?

David

Thanks for your prompt reply David. You have answered my immediate question and the code now works as desired.

I am creating a software development environment and one of my objectives is to enable ‘sprites’ to have code that can run simultaneously with the code of other sprites. As far as I can see each sprite has to run its own code in a separate thread. The ability of Panda3D to run multiple sequences in parallel is great, and certainly gives the best performance, but since sprite code is written in Python it could do any arbitrary task, not necessarily only tasks that can be done with sequences. That’s the plan, although there might be problems with performance if too many separate threads are running. I will continue to develop and experiment as I go.

I am using the Windows pre-compiled SDK, and Thread.isTrueThreads() returns false. Would it be possible for me to get away without the need to have calls to Thread.considerYield() or Thread.forceYield() if I was able to use “true” threads?

Chris

Yes, if you compile Panda yourself, you can remove the ‘SIMPLE_THREADS’ setting from makepanda and then you will have true threads enabled. Note that doing so will result in about 10% across-the-board performance degradation, due to the need to protect all operations (including reference count management) from race conditions. In this mode, though, Thread.considerYield() is a no-op, and can be omitted if desired.

You might consider offering the option for your users to use synchronous tasks for their sprite code. It just means they will have to be more responsible in the design of their coding, and write it so that it will not do any arbitrary task, but will only do a little bit at a time. This will allow the diligent deevloper the option of avoiding the overhead of threads when possible.

David

Thanks David, that’s good advice and I will allow for this as I develop the project.

Normal python threads with Thread.setDaemon(True) works fine. Coroutine are very good too. Both don´t affect FrameRate.

No, it doesn’t. It might appear to work fine, but it will make your program highly unstable and subject to a messy crash, if you use it in conjunction with a Panda build that doesn’t include true threading support.

If you implement coroutines yourself, sure. But this is the same thing as using the task system.

Any additional usage of the CPU can affect frame rate.

David

Hey master

Ok … i don´t understand Panda´s internal architecture. I just share my code experiences. I really don´t know how normal python threads can make my program unstable (i´m not a developer i´m just a user).

Coroutines can be frozen and just generate values when you need, again i don´t know if i can do this with tasks or else, if task is a coroutine itself. I personally, for some “tasks”, prefer coroutines :wink:

See … i´m not telling Panda´s Thread, Task, Intervals, Parallel, etc is worst (no it´s brillant). Either i don´t wanna make apologizes to worst pratices … i just shared some code experiences

regards

I’m now encountering problems trying to start a thread from within a thread. The following code appears to crash Panda3D…

from direct.directbase import DirectStart
from direct.showbase.DirectObject import DirectObject

# if I start a thread within a thread, and the nested thread finishes first, Panda3D appears to crash...

def exampleTask2(task):
    print "nested thread"
    return task.done

def exampleTask1(task):
    print 'thread starting.'
    taskMgr.add(exampleTask2, 'someTaskName2', taskChain = 'threadedChain2')
    print 'thread ending.'
    return task.done

taskMgr.setupTaskChain('threadedChain1', numThreads = 1)
taskMgr.setupTaskChain('threadedChain2', numThreads = 1)

taskMgr.add(exampleTask1, 'someTaskName1', taskChain = 'threadedChain1')

run()

Depending on the timing of when the threads end other behaviour is exhibited. The error:

Assertion failed: _num_busy_threads == 0 at line 865 of c:\p\p3d\panda3d-1.6.2\panda\src\event\asyncTaskChain.cxx

can easily be obtained from more complex situations. I am still using Simple Threads with Panda3D-1.6.2 under Windows.

Will these problems go away with ‘true’ threads?

(I haven’t yet recompiled Panda with ‘true’ threading because the necessary downloads to recompile under Windows are such a huge mission for me on fairly slow Internet connections - hence I’m trying to get things working under Simple Threads first.)

Cheers
Chris

Hmm, I’m not able to reproduce the problem. It appears to run fine for me, but I’m running on a more current version of Panda. Have you tried it with 1.7.0 to see if the problem persists there?

David

No I haven’t tried 1.7.0. I have developed the following alternative way of starting threads, which seems to solve the problem for me. I think I may stick with this kind of model anyway, because it can be generalised into a message broker for passing messages between actors.

However, I do have another question: is it possible to reduce the stack size allocated to threads? I have tried using Thread.stack_size() but have had no success. I’m keen to run lots of threads, but ecah thread being only a small piece of code not really needing much stack space.

Here’s the code …

class threadStarter(threading.Thread):
    def run(self):
        self.currentFn(self.currentObject)

# set up threadManager
threadQue = deque([])
threadLock = threading.Lock()
useAsyncTaskManager = False # using the AsyncTaskManager leads to deadlock, so better not use it!

def enqueThread(threadFn, threadObject, threadName='thread'):
    threadLock.acquire()
    threadQue.append((threadFn, threadObject, threadName))
    threadLock.release()
            
def threadManager(task):
    threadLock.acquire()
    newTaskNumber = len(threadQue)
    if newTaskNumber > 0:
        if useAsyncTaskManager:
            taskMgr.setupTaskChain('threadedTaskChain', numThreads=threadedTaskChain.getNumTasks()+newTaskNumber)
            while len(threadQue) > 0:
                currentFn, currentObject, currentName = threadQue.popleft()
                taskMgr.add(currentFn, currentName, extraArgs=[currentObject], taskChain='threadedTaskChain')
            print "Active Threads=", threadedTaskChain.getNumTasks()
        else:
            while len(threadQue) > 0:
                currentFn, currentObject, currentName = threadQue.popleft()
                currentThread = threadStarter()
                currentThread.currentFn = currentFn
                currentThread.currentObject = currentObject
                currentThread.setName(currentName)
                currentThread.start()
                del currentThread
            print "Active Threads=", threading.activeCount()
    threadLock.release()
    return task.cont 

taskMgr.setupTaskChain('threadedTaskChain', numThreads = 1)
threadedTaskChain = AsyncTaskManager.getGlobalPtr().findTaskChain("threadedTaskChain")
taskMgr.add(threadManager, 'startThreads')

run()  # Panda handles the main loop

The thread stack size is controlled by the Config variable:

thread-stack-size 4194304

which specifies the desired size in bytes. The default is 4MB. The implementation may or may not respect this on all platforms (but it does on Windows).

David

I tried Thread.considerYield() and Thread.forceYield() while one thread was waiting for a sequence to play, but I was getting a huge performance hit - slow and jerky running of sequences. I then tried Thread.sleep(0.04) and everything speeded up and became smooth. I can’t explain it, but am going ahead using sleeps.

I’m having problems with animating geometry that I have created procedurally in a thread, animated with an interval, and then removed from the scene graph.

The error I get is

Assertion failed: _states->find(this) == _saved_entry at line 1951 of c:\p\p3d\panda3d-1.6.2\panda\src\pgraph\transformState.cxx
Assertion failed: _saved_entry == _states->end() at line 108 of c:\p\p3d\panda3d-1.6.2\panda\src\pgraph\transformState.cxx

Full source code is available at teach.hilderbuild.com/tiki-index.php?page=Cicada

I am wondering about the version of functions like reparentTo() and copyTo() that have a current_thread parameter. Should I be using those versions of the NodePath functions when I manipulate the scene graph from a thread?

Hmm, that error does sound like trouble. I wonder if you get the same error in 1.7.0?

No, no need. That parameter is just available as a potential optimization for low-level C++ code–if the C++ code already knows what the current_thread value is, it can save a tiny bit of time by passing it in, obviating the need to look it up again. But it is not necessary for correctness, and the performance win has no impact on Python code.

David

I’ve avoided trying 1.7.0 because it says not to be used for production. I am intending to release my software this year. How unstable is 1.7.0 really, and how far away is a more stable release?

Don’t believe it. 1.7.0 is stable, and has been in production for a couple of months now. :slight_smile:

David

I’ve switched to 1.7.0 and still getting the same problem.

Assertion failed: _states->find(this) == _saved_entry at line 1944 of c:\panda3d-1.7.0\panda\src\pgraph\transformState.cxx
Assertion failed: _saved_entry == _states->end() at line 108 of c:\panda3d-1.7.0\panda\src\pgraph\transformState.cxx

Interesting? Will continue to ponder this. It may be that I require some thread locking at critical places, but it does does seem to be a persistent pattern that if I create a piece of geometry, animate it with an interval, and then detatch it from the scene graph the error is raised as above. It doesn’t then go away, but persists with the error being raised again and again as long as run() is being reentered. Any further guidance would be appreciated.

Cheers

Can you provide a simple program that demonstrates the problem? Thanks!

David

sure, I’ll work on it - finding out what part of the 1,000 lines of code actually causes the problem could take me a while …

meanwhile I think I might have found a related forum topic (looks like my problem is nothing to do with my use of threads). [url]LerpFunc doesn't register with the IntervalManager?] If I am right on this I will post to that topic instead when I have tracked it down.