some things i noticed:
- the mass matrix already gets set in newton_addentity().
- be careful with vector()/vectorf(). newton doesn't work with vars.
- i think if your "character" is a rolling sphere then it doesn't really make sense to rotate your steering force like that. or does setomega block all rotation?

it's hard to say... maybe someone else has ideas.