I don’t know why that is treated different with and without windowmanager but the solution is to parent nothing to the camera. Create a new node and parent everything which should be parented to the camera and the camera to that node and move this node instead of the camera. Solves this issue.
But I still don’t understand why.