Well, inertial systems have a velocity too, not only an origin. So it's the change from one velocity to another which causes the dilation.
http://en.wikipedia.org/wiki/Twin_paradox