With the help of Superku (who provided a very helpfull matrix and motivation), I finally got some kind of solid view frustum culling to work:

Not pretty and at the time of the screenshot (with 500 entities) I even had doubts that the culling would cause any peroformance gain at all. But when starting at the center looking to the outside, I have stable 60fps with culling and only 23fps without culling, so it actually seems to bring some performance gain, even quite a lot