no, the spacers are for taking the internal stress, which persists as you screw the axles tight. It should be doing its job if you feel that the spacers are stiff.
If there is no spacers, then the stress goes into the bearings, bc the bearings are very fine and the elasticity of metal, the bearings can deform and roll slower.