I was able to solve it (I hope) by reducing a bunch of texture sizes. Whether the problem will reappear after extended play sessions is still a concern for me, but it's much more stable than it was before.

One thing I realized is that I could reduce the size of my normal maps without impacting the appearance as much.

I'm not sure what metric to use as far as how much texture size I can use, what's the limit, etc... it would be nice to have a specific number that we can make sure we stay under. Of course it would vary depending on video card, computer memory, nexus, etc...

I think this all falls under optimizing the game. I'm guessing that big name games have all sorts of tricks and methods of getting really big textures on screen seamlessly. Reduce, reuse, recycle ... that sort of thing.