I think I can explain this the best way through images. The grid shows how the camera.pan is. The red/yellow/green lines are my drawed boxes. Green mouse cursors are the begin and end points of my mouse selection. (note that boxes are 2d on the images but 3d in my game)

First one without your vec2d_rotate function:


Second is with your vec2d_rotate function:


Third, this is with your vec2d_rotate function and with the tweak I posted in my previous post (only 1 of the 2 vectors are correct):


Fourth, this is what I want: