It looks like it’s related to the camera which I use to pitch the view up and down (I use physics for other movement and just fix the camera to my character controller). If I disable that, everything works ok. So I’ll try and replicate that with some other code.
I’ll probably change the way I’ve implemented the camera across the server and just implement it locally on the client which should avoid that issue, I had doubts about the way I’d done it anyway.