You can also add audio and networking without Open Library - but an Open Library interface makes a lot of sense here: it allows to replace one library by another without changing your application. F.i. when you replace DirectPlay by Enet, your online game will still work without modification.

But this is different for physics libraries. Physics engines use different equation systems and thus behave different. You can not just replace one physics library by another. Therefore no one needs an Open Library interface for physics engines. The current API is fully sufficient for implementing physics libraries, as you need to adapt your app to the used physics library anyway.