We havent seen this issue before.
-Please attach USB traces using a USB hardware analyzer during the hub enumeration if possible.
-In the failure case, does VBUS appear if you assert Reset on the hub?
-Could you also try the same sequence using a HX2VL DVK if you have one?
Ah! I believe I have found the problem. I used a uni-directional TVS diode on the data lines to VBUS. The problem is that VBUS is also used to perform reset on the USB hub. With a self-powered USB hub, the data lines are pulled high, and in this case that means that the VBUS is pulled high (well, to an indeterminate 2.0 - 2.4V) when the USB cable is un-plugged. In this case, the Mac host is interpreting the vbus signal as an SRP data-line pulse, and it all goes down hill from there.
I see your recommended schematic has bidirectional TVS diodes.
I pulled the TVS diode off of the board and everything worked without issues.
That's an easy fix for the next rev.