have you seen if you have signal in the rotary encoder? I ask because I had similar problem with other kind of rotary encoders with extended power supply (PS) and at the end the problem was the min PS was 5.5 (not 5V), I had 6 encoders with different behaviors, With a PS of 6V, running without problem.
The one rotary encoder I have is for 5V, but works down to about 3.1V, I have tested it and measured also the signals with an oscilloscope when they are connected to the pins.
I found the problem now and it works:
The Quad Decoder Component expect a logic low index pulse, my encoders have a logic 1 index pulse. By adding a NOT Gate to the index pulse it works fine now.
I found this by trying to not use the index pulse and then it counts correct.
If someone experience a similar problem with encoders: So please check the polarity of your signals always.