You can create stub models on the server. The are only some extruded outline of the models with a low poly count and no texture.
I think you can even use polygon collision with them.

Do you use a custom solution, c_move or PhysX for the collision?