I think you can remove the crystals. The BLE only uses ECO which is driven by the 24 MHz as you stated. If you do not need WDT with high precision and also counting time in sleep mode the 32 kHz can also be removed.
My question will be if you do not need BLE why not choosing another chip from Cypress or other company. I think this approach will be cheaper in long term HW creation.