I got it working, but had to jump through many issues, what fixed it was having a dummy READ/WRITE word that is discarded. That got is solid. I suspect the fact that the slave is running on the ILO only, is the real culprit. On the production design we will add a watch crystal and see if that does not improve things.
There were many other ideas to try, cycling the slave SPI on and off between transfers also helped.