The slave maintains two buffers: one where the writes from master go to (ie the memory address you mention) and one where the master reads from. So a fast reaction to setup the transfer buffers accordingly is required.
I can see how setting up the buffers fast is important. My question was whether a buffer address or position could be directly addressed by the master. It doesn't look like it. Perhaps by writing custom slave API functions?
buffer address or position could be directly addressed by the master No, you have to wait for a master address write, then set up (or fill) the master read buffer and let the master do the read.