Please refer to Page 24 in the datasheet for MODE_SEL settings.
1. Default setting is 11 which corresponds to internal ROM mode.
2. Pulling both up would cause the hub to boot from the internal ROM firmware which cannot be modified.
3. No, HX3 itself can be used to program the EEPROM before setting MODE_SEL pins to the I2C boot mode. The programming through HX3 is done using Blaster Plus utility. Please refer to the KBA for more details on the process.