What would you like to achieve with a captcha? Keeping bots from signing up? If so, what would the problem of that be?
Keep in mind that the bots have to be taught your custom protocol, so someone needs to have an incentive to write such a bot.

If you want to protect yourself against flooding, you should introduce rate limiting into your API endpoints, and define sensible limits (eg. a peer may only request the server list 5 times per minute). That's not a (D)DoS protection, but it can help you avoid heavy computations on the database.

Rate limiting you can do on something like redis, or some other in-memory store. Doesn't need to persistent, if the data is lost all rates are reset, but who cares. The advantage is that you don't need to do the full roundtrip to the full-blown database backend which has to drop down to the disk to ensure integrity.

Last but not least, here is a presentation about Cryptography called Everything you need to know about cryptography in 1 hour. Keep in mind thought that cryptography ins't the same as security.

Edit: Here is the video to the slides: http://blip.tv/fosslc/everything-you-need-to-know-about-cryptography-in-1-hour-3646795