it is possible - and we've done it (using RakVoice, which uses speex and Portaudio, which are the basis of Skype), and it's a really cool thing.

http://www.coniserver.net/ubbthreads/showflat.php/Cat/0/Number/682921/an/0/page/0#Post682921

It was a great thing to experience this kind of 3d telephony, it really changes your perspective. In fact we didn't care about game but setting this thing up.