Augmented Reality with Bullet physics

In the meantime I managed to get it to work, pretty much as you just suggested.
If you have a chance to create a github repo I’d be happy to help you maintain it.