it’s the hardware, not pyserial, that causes the problem.
if sleeping 2 sec is not acceptable in a Panda3D app, can you delay the read/write operations by 60 frames (assuming framerate in Panda is 30fps) ? so that the app keeps running, but Serial would think the app slept for 2 sec. 