The filter is a lowpass filter 64 taps (about 5% overshoot)
on the picture we have :
signal3 : 300kHz DMA clock (input and ouput of the filter are DMA)
signal4 : The duration of an ISR program that changes the input value of the filter from -100 to 120 and back, to perform a signed square signal
signal5 : the sign bit of the filter output (bit 7 of HOLDAH register)
CH2 (red analogic curb) : the output of the filter with an offset of 128 ==> 8 bits DAC
We can see that at the end of the rising edge of CH2 (and before descent) the sign bit of the filter is swicthing to negative values generating a bug in the response of the filter.