I don't know what causes it or how long you are away from fixing it, but steering drift is a HUGE problem. (Especially on Xbox)
Small rovers are just pointless at this point as it's impossible to move in a straight line. Large rover kinda get away with it but the problem still exists.
When it comes to wheels, I think by trying to make them as realistic as possible, you've actually made them as unrealistic as they could be. You need to go backwards with the wheels and make them as simple as they can be. At the moment, everything's too complicated and too many factors are trying to compete, like gravity/mass and friction, making the wheels unstable.
The wheels themselves don't need to calculate mass, that's done by the suspension. If the wheels were solid and couldn't pass through the ground like blocks don't, it might fix the issue.