What is the exact part numbet?. I have found the reference voltage to be stable. You can use the internal vref and bring it out for measurement or other uses.
Please refer the below link for more information:
If supply is not stable, and if this is being provided as reference, the ADC output won't be stable. So it is required to have stable power supply. For current fix, high capacitor can be connected in power supply lines which could remove fluctuations and make it stable. Let me know if this helps.