Sorry my bad. There was a loose connection, I forgot to flood my homemade pcb with solder. I tried everything but later found that once the wiring was fixed everything worked like a charm.
My apologies, i had checked the voltages on the pins but after tearing hair on it for a whole day, I checked it with some current (led & resistor) and found the problematic ground connection. Once that was fixed, even with the buffer there were no issues.