I've seen that the BCM20737 chip was supporting RSA directly from the hardware.
What about the BCM20736 ? Is everything done on the software part and what are the performances ?
RSA is supported on the 20737S within Firmware. This feature is not available on the 20736S and performance benchmarks have not been recorded to measure the impact during encrypt/decrypt operations.
I think you will find this thread useful: How to operate RSA in BCM20737S & BCM20737S document,