The tolerance of the crystals are specified to meet the requirements of the USB specification. The crystal will affect the measured bit jitter on the USB bus. Reducing the tolerance on the crystal will increase the jitter on the data appearing on the USB bus. This increase jitter will affect the signal quality and cause it to fail this compliance test.
Increasing the jitter on the bus will also cause some configurations to fail. Taking worst case cables, and 5 tiers of hubs on a system will most likely cause enumeration problems on devices which cannot meet the signal quality requirements.